Amazon EKS Part 2: Operations, Upgrades, Security, Observability and Cost

Creating an EKS cluster takes about fifteen minutes and one Terraform apply. Running that cluster for three years, through nine Kubernetes versions, a hundred AMI patches, three audits and a regulator's visit, with zero customer-visible downtime: that is the job the JD is describing. Post 27 gave you the architecture, the VPC CNI and the IAM model. This post is day two onward: who owns what, how an upgrade really goes, how nodes live and die, which security controls a bank's auditors will ask you to prove, how you see what the cluster is doing, what it costs, and how you argue for the platform's next step. After reading it you will be able to walk an interviewer through a bank-grade EKS upgrade from change ticket to rollback plan, name the AWS service behind every line of the security checklist, and answer "the pods can't get IPs" without a pause.

The day-2 operating model: who owns what

Day 0 = design, Day 1 = build, Day 2 = everything after the first workload lands: patching, upgrades, incidents, access requests, audits, cost reviews. The most useful thing you can say about day 2 in an interview is who owns which layer, because a bank's EKS estate is never run by one team, and the JD's "collaborate with Cloud Engineering and Architecture teams" only makes sense once you see the layers.

The line between cloud engineering and the platform team is where most day-2 friction lives: you need a secondary CIDR for pods and they own IPAM; you need a Transit Gateway route to the mainframe gateway and they own the TGW; you need an account for a new environment and they own account vending. The last section of this post is about working across that line without it turning into a ticket queue.

Environments are accounts, not namespaces

The AWS account is the hardest isolation boundary AWS offers: IAM, service quotas, billing and blast radius all stop at the account edge. So environments map to accounts: sandbox (platform experiments, rebuilt weekly), dev, test (often called UAT in a bank), prod, each under an Organizational Unit with its own SCPs, plus a shared-services account holding ECR, the Argo CD management cluster, the observability workspaces and the backup buckets. Inside each account the next decision is cluster topology:

• Cluster per team or application: maximum isolation and the easiest story for an auditor, but upgrade work, control plane cost, add-on drift and quota consumption multiply by the number of clusters.
• Multi-tenant clusters: dozens of teams share one cluster per environment per region, separated by namespaces, quotas, network policies and admission policies. One upgrade per environment and dense bin-packing, but a bad upgrade has a huge blast radius and a noisy tenant affects everyone.

Banks usually land in the middle: a few multi-tenant clusters per environment and region, plus dedicated clusters where regulation or risk demands it (card data in PCI scope, a trading platform with its own change windows, a vendor product that insists on cluster-admin). The interview answer is the reasoning, not the number: blast radius, upgrade cadence, compliance boundaries, cost.

Naming, tagging and quotas

Pick a naming standard once and enforce it in the Terraform module so nobody can deviate:

eks-<business-unit>-<env>-<region-short>-<nn>
eks-payments-prod-cac1-01
eks-payments-prod-cac1-02     # blue/green partner, see Upgrades below
eks-shared-dev-cac1-01

Tags do the same job for money and ownership. Every cluster, node group, launch template, volume and load balancer carries Environment, CostCenter, Owner, Application, DataClassification, BackupPolicy and ManagedBy=terraform. Tags only show up in Cost Explorer after you activate them as cost allocation tags (aws ce update-cost-allocation-tags-status --cost-allocation-tags-status TagKey=CostCenter,Status=Active), and EKS adds its own aws:eks:cluster-name tag to the EC2 instances it launches so you can split the EC2 bill per cluster.

Service quotas = per-account, per-region limits AWS imposes, and the classic "worked in dev, failed in prod at 2 a.m." cause. The ones that bite EKS:

• Network interfaces per region (the VPC CNI attaches an ENI per node per IP block; prefix delegation reduces the count).
• Elastic IPs per region (default 5; NAT gateways and internet-facing NLBs consume them).
• Running On-Demand vCPUs and, separately, Spot vCPUs; a Karpenter scale-out fails with VcpuLimitExceeded when you hit them.
• Application and Network Load Balancers per region (default 50 each); share ALBs with the load balancer controller's group.name annotation.
• Security groups per ENI and rules per security group, once you use security groups for pods.
• IPv4 CIDR blocks per VPC and routes per route table, once the Transit Gateway gets busy.

$ aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A \
    --query 'Quota.{name:QuotaName,value:Value}'
{
    "name": "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances",
    "value": 1152.0
}
$ aws service-quotas list-service-quotas --service-code elasticloadbalancing \
    --query 'Quotas[?contains(QuotaName, `Application Load Balancers`)].[QuotaName,Value]' --output text
Application Load Balancers per Region	50.0

Put a CloudWatch alarm on quota usage (Service Quotas publishes AWS/Usage metrics) and raise limits a quarter before you need them; vCPU increases can take days.

Blueprints and the cluster factory

EKS Blueprints for Terraform = AWS-maintained, opinionated Terraform patterns: the community terraform-aws-modules/eks/aws module for the cluster and node groups, and aws-ia/eks-blueprints-addons/aws for the add-ons everyone needs (load balancer controller, Karpenter, External Secrets, metrics-server, Argo CD, the CloudWatch agent). The platform team wraps these in one internal module with the bank's defaults baked in (private endpoint, KMS key, logging on, Bottlerocket, the standard tags) and runs it from one pipeline, so a new cluster is a short tfvars file and a pull request. That is the cluster factory: clusters are stamped, identical and disposable, which is what makes the blue/green upgrade strategy below realistic. Post 29 builds it.

Upgrades: the three-tier dance, in detail

Interviewers check upgrade facts one at a time, so get them exact. EKS ships a new Kubernetes minor a few months after upstream. Each version gets roughly 14 months of standard support, then 12 months of extended support at six times the control plane price (US$0.60 instead of US$0.10 per cluster-hour at the time of writing), and at the end of extended support AWS upgrades the control plane for you whether you are ready or not. You move one minor version at a time (1.31 → 1.32 → 1.33, never 1.31 → 1.33) and there is no downgrade. An upgrade has three tiers in a fixed order: control plane, then add-ons, then nodes. The kubelet version skew policy (a kubelet may trail the API server by up to three minors) is what makes "control plane first, nodes later" safe; kube-proxy and the CNI have narrower compatibility windows, which is why add-ons come second.

Pre-flight checks

EKS upgrade insights = checks the control plane runs continuously against your cluster and exposes through the API: deprecated API usage seen in the audit log over the last 30 days, kubelet and kube-proxy version skew, add-on compatibility and cluster health. Start every upgrade here:

$ aws eks list-insights --cluster-name eks-payments-dev-cac1-01 \
    --query 'insights[].{name:name,status:insightStatus.status,reason:insightStatus.reason}' --output table
------------------------------------------------------------------------------------------------------
|                                            ListInsights                                            |
+--------------------------------------------+----------+----------------------------------------------+
|  Deprecated APIs removed in Kubernetes v1.33|  PASSING | No deprecated API usage detected within the  |
|                                            |          | last 30 days.                                |
|  Kubelet version skew                      |  WARNING | Kubelet versions in cluster are older than   |
|                                            |          | the control plane version.                   |
|  kube-proxy version skew                   |  PASSING | kube-proxy versions are compatible.          |
|  EKS add-on version compatibility          |  ERROR   | Add-on vpc-cni v1.18.3 is not compatible     |
|                                            |          | with Kubernetes v1.33.                       |
|  Cluster health issues                     |  PASSING | No health issues detected.                   |
+--------------------------------------------+----------+----------------------------------------------+

An ERROR means the control plane update call will be rejected until you fix it. The deprecated-APIs insight only sees what the audit log saw, so back it up with a client-side scan of what is stored in the cluster and in Helm releases: kubent (kube-no-trouble) or pluto, checked against the upstream deprecated API migration guide for the target version.

$ kubent --target-version 1.33.0
2:14PM INF >>> Kube No Trouble `kubent` <<<
2:14PM INF Target K8s version is 1.33.0
2:14PM INF Retrieved 412 resources from collector name=Cluster
2:14PM INF Retrieved 37 resources from collector name="Helm v3"
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.32 <<<
------------------------------------------------------------------------------------------
KIND         NAMESPACE     NAME           API_VERSION                            REPLACE_WITH (SINCE)
FlowSchema   <undefined>   legacy-batch   flowcontrol.apiserver.k8s.io/v1beta3   flowcontrol.apiserver.k8s.io/v1 (1.29.0)

The rest of the pre-flight list, which a strong candidate recites without prompting:

• PodDisruptionBudgets: a PDB with minAvailable equal to the replica count can never be satisfied during a drain, so the node rollout stalls. Fix the impossible ones before the change window, not during it.
• Cluster Autoscaler or Karpenter version: Cluster Autoscaler is versioned per Kubernetes minor and Karpenter has a compatibility matrix; upgrade the autoscaler alongside the control plane or scale-out breaks the moment the API changes.
• Third-party controllers: the AWS Load Balancer Controller, External Secrets Operator, Argo CD, cert-manager, Kyverno, the CSI drivers. Each publishes a supported Kubernetes range.
• Add-on compatibility: aws eks describe-addon-versions --kubernetes-version 1.33 gives the exact versions of vpc-cni, coredns, kube-proxy and the EBS CSI driver that support the target.
• Free IPs: the control plane upgrade needs a few free IP addresses in the cluster subnets for new API server ENIs; a subnet with two IPs left fails the update.

Tier 1: the control plane

The control plane update is one API call. EKS builds new API server instances at the target version behind the same endpoint, drains the old ones and rolls the control plane back on its own if health checks fail. Your kubectl keeps working; expect a few seconds of API errors while connections move, which is why controllers with retries are fine and a script without retries is not.

$ aws eks update-cluster-version --name eks-payments-dev-cac1-01 --kubernetes-version 1.33
{
    "update": {
        "id": "b5f0ba18-9a87-4450-b1e7-0c1d2e3f4a5b",
        "status": "InProgress",
        "type": "VersionUpdate",
        "params": [
            { "type": "Version", "value": "1.33" },
            { "type": "PlatformVersion", "value": "eks.6" }
        ],
        "createdAt": "2026-03-10T02:01:12.318000-05:00",
        "errors": []
    }
}
$ aws eks describe-update --name eks-payments-dev-cac1-01 --update-id b5f0ba18-9a87-4450-b1e7-0c1d2e3f4a5b \
    --query 'update.status'
"Successful"
$ kubectl version --output=json | jq -r .serverVersion.gitVersion
v1.33.3-eks-3f4a5b6

Expect 10 to 20 minutes. Nothing on the nodes changes and no pods restart; the control plane now reports 1.33 while every kubelet still reports 1.32, and that is fine.

Tier 2: managed add-ons

Managed add-ons = the cluster components AWS packages and version-tracks for you: vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver, eks-pod-identity-agent, metrics-server, amazon-cloudwatch-observability and more. Ask EKS which versions match the new control plane, then update each. AWS guidance is to move the VPC CNI one minor version at a time and keep kube-proxy at the control plane's minor; a sensible order is vpc-cni, kube-proxy, coredns, then the CSI drivers and the rest.

$ aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.33 \
    --query 'addons[0].addonVersions[?compatibilities[0].defaultVersion].addonVersion' --output text
v1.20.3-eksbuild.1
$ aws eks update-addon --cluster-name eks-payments-dev-cac1-01 --addon-name vpc-cni \
    --addon-version v1.20.3-eksbuild.1 --resolve-conflicts PRESERVE
{
    "update": {
        "id": "7c2e9a10-4d3b-4f6e-9a1b-8e7d6c5b4a39",
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            { "type": "AddonVersion", "value": "v1.20.3-eksbuild.1" },
            { "type": "ResolveConflicts", "value": "PRESERVE" }
        ]
    }
}
$ aws eks list-addons --cluster-name eks-payments-dev-cac1-01 --output text
ADDONS	aws-ebs-csi-driver
ADDONS	coredns
ADDONS	eks-pod-identity-agent
ADDONS	kube-proxy
ADDONS	metrics-server
ADDONS	vpc-cni

--resolve-conflicts PRESERVE keeps any field you customised (a CoreDNS replica count, a CNI variable such as ENABLE_PREFIX_DELEGATION) instead of resetting it to the add-on default; in Terraform the same knob is resolve_conflicts_on_update. Add-ons you installed yourself with Helm (Karpenter, the load balancer controller, ESO, Kyverno) get their chart bumps through Argo CD in the same change.

Tier 3: nodes

Nodes are where the risk is, because this is where pods move. There are three node types on EKS, and each upgrades differently.

Managed node groups do a rolling replacement. You give the node group the new Kubernetes version (or a new AMI release version, or a new launch template version) and EKS raises the Auto Scaling group's desired count, waits for new nodes to become Ready, then cordons and drains old nodes in batches of updateConfig.maxUnavailable (or maxUnavailablePercentage) and terminates them. If a PDB blocks a drain for about 15 minutes the update fails with PodEvictionFailure; --force overrides PDBs, which you do not do in prod without a conversation.

$ aws eks update-nodegroup-config --cluster-name eks-payments-dev-cac1-01 --nodegroup-name general-a \
    --update-config maxUnavailablePercentage=25
$ aws eks update-nodegroup-version --cluster-name eks-payments-dev-cac1-01 --nodegroup-name general-a \
    --kubernetes-version 1.33
{
    "update": {
        "id": "e1d2c3b4-a596-4877-8f69-5a4b3c2d1e0f",
        "status": "InProgress",
        "type": "VersionUpdate",
        "params": [
            { "type": "Version", "value": "1.33" },
            { "type": "ReleaseVersion", "value": "1.33.3-20260305" }
        ]
    }
}
$ kubectl get nodes -L eks.amazonaws.com/nodegroup
NAME                                           STATUS                     ROLES    AGE   VERSION               NODEGROUP
ip-10-42-11-23.ca-central-1.compute.internal   Ready,SchedulingDisabled   <none>   41d   v1.32.5-eks-5d4a308   general-a
ip-10-42-12-88.ca-central-1.compute.internal   Ready                      <none>   3m    v1.33.3-eks-3f4a5b6   general-a
ip-10-42-13-07.ca-central-1.compute.internal   Ready                      <none>   41d   v1.32.5-eks-5d4a308   general-a

Karpenter nodes are replaced through drift: Karpenter notices that a node no longer matches its NodePool and EC2NodeClass (the cluster version changed, or you pinned a new AMI alias) and replaces it, respecting PDBs and the disruption budgets you set. "Upgrade the nodes" becomes a Git commit that bumps the AMI alias, and the rollout pace is a policy, not a script:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: bottlerocket-hardened
      expireAfter: 720h          # force replacement after 30 days regardless
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "c", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"                       # never disrupt more than 10% at once
      - nodes: "0"                         # and nothing during business hours
        schedule: "0 13 * * 1-5"           # 08:00 Toronto, in UTC
        duration: 10h
        reasons: ["Drifted", "Underutilized"]
  limits:
    cpu: "400"

Fargate pods each run on their own micro-VM with their own kubelet, so they pick up the new version only when the pod is recreated: after the control plane moves, roll the Fargate deployments with kubectl rollout restart deployment -n <ns>.

A bank-grade upgrade runbook

Every regulated shop runs upgrades as a change with evidence. The shape that survives audits:

1. Rehearse in lower environments: sandbox first, then dev, then test, with at least a week of soak in test. The same Terraform and the same Argo CD app-of-apps drive every environment, so the rehearsal is real.
2. Change ticket: target version, the insights output, the kubent report, the add-on version matrix, the PDB review, the maintenance window, the rollback plan, the named approver, the communication to app teams.
3. Freeze: no app deployments through the window, enforced by pausing Argo CD auto-sync on tenant apps if necessary.
4. Execute tiers 1 to 3 from a pipeline, not a laptop, with each step's output attached to the ticket.
5. Validate: smoke tests per critical app, dashboards green, no NotReady nodes, no Pending pods, load balancer targets healthy, DNS latency normal.
6. Close with evidence, and record any surprise as a lesson for the next environment.

Rollback is the question that separates operators from readers. There is no control plane downgrade, so tier 1 cannot roll back in place; tiers 2 and 3 can (older add-on version, older AMI) as long as they stay within skew. The only true control plane rollback is blue/green clusters: the cluster factory stamps eks-payments-prod-cac1-02 at the new version, Argo CD syncs the same applications into it from the same Git repository, traffic shifts with a Route 53 DNS weight or the bank's GSLB, and the old cluster stays warm until the change closes. Rolling back is shifting the weight back. It costs a second control plane and duplicate nodes for a few days, and it is what a tier-one bank does for the clusters that matter.

Staying current is also a cost decision: a cluster that drifts into extended support pays roughly US$365 extra per month and accumulates two or three versions of technical debt that all have to be paid at once. Set the upgrade policy deliberately (aws eks update-cluster-config --name ... --upgrade-policy supportType=STANDARD) and put the upgrade calendar on the platform roadmap: one minor per quarter is the sustainable pace.

Node lifecycle: patching, draining, dying gracefully

Between Kubernetes upgrades, nodes still change every couple of weeks, because the AMI (Amazon Machine Image = the node's operating system image) gets kernel and CVE fixes. On OpenShift the Machine Config Operator handled this for you (Post 20). On EKS the mechanism depends on the OS you chose:

• AL2023 (Amazon Linux 2023) = the default EKS-optimised AMI now that Amazon Linux 2 is out of the picture. It uses nodeadm for bootstrap (a NodeConfig YAML in user data instead of a shell script), cgroup v2 and IMDSv2 by default. Patching means a new AMI release and a node replacement, either a managed node group version update or a Karpenter AMI alias bump.
• Bottlerocket = AWS's minimal, container-only OS: read-only root filesystem, no package manager, no SSH daemon, SELinux enforcing, configured through an API rather than files. Patching can be in-place: the Bottlerocket update operator (brupop) watches for new versions, cordons and drains one node at a time, applies the update to the inactive partition, reboots into it and uncordons, with automatic rollback if the new image fails to boot. Or you keep the replace-the-node model and bump the AMI alias. For a bank the tiny attack surface is the easy sell; the hard part is that there is no shell to poke around in, which turns out to be a feature.

Cadence: monthly AMI refresh in prod, faster for a critical CVE, with the same lower-environment rehearsal as a version upgrade but a lighter change record. Pin AMIs by version in the EC2NodeClass (alias: bottlerocket@1.44.0, not @latest) so a patch is a reviewed Git change and dev, test and prod are provably on the same image.

No SSH: Session Manager

Bank nodes have no SSH key pair and no port 22 in any security group. When you need a shell, AWS Systems Manager Session Manager opens one through the SSM agent that ships in the AL2023 and Bottlerocket AMIs; the node role carries AmazonSSMManagedInstanceCore, access is an IAM decision, and every session is logged (and can be recorded to S3 or CloudWatch Logs for the auditors).

$ aws ssm start-session --target i-0f1e2d3c4b5a69788
Starting session with SessionId: j.doe-0a1b2c3d4e5f6a7b8
sh-5.2$ sudo journalctl -u kubelet --since "10 min ago" --no-pager | tail -5
Mar 11 15:02:41 ip-10-42-11-23 kubelet[2143]: E0311 15:02:41.118 ... "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox ... failed to assign an IP address to container"
sh-5.2$ sudo crictl ps | head -3

On Bottlerocket the session lands in the control container; enter-admin-container then sudo sheltie gets you a root shell on the host for the rare cases where kubectl debug node/ is not enough.

Termination, drain and the two objects that protect you

Nodes die for planned reasons (upgrade, patch, Karpenter consolidation) and unplanned ones (hardware retirement, a Spot interruption with a two-minute warning). The platform's job is to turn every termination into a graceful drain: cordon the node, evict pods respecting PDBs, wait for the load balancer to deregister targets, then let EC2 take the instance.

• Managed node groups handle Spot interruptions and EC2 rebalance recommendations natively through Auto Scaling's capacity rebalancing and drain the node for you.
• Karpenter handles interruptions natively too: you give it an SQS queue (settings.interruptionQueue) fed by EventBridge rules for Spot interruption warnings, rebalance recommendations, scheduled maintenance and instance state changes, and it cordons, drains and pre-provisions a replacement.
• aws-node-termination-handler is the DaemonSet you still deploy for self-managed node groups, in IMDS mode or queue-processor mode. Naming it shows you know the history; saying you no longer need it on managed node groups shows you kept up.

Drains only stay graceful if the app teams gave you two objects, both from Post 9: a PodDisruptionBudget that allows at least one replica to move, and a topology spread constraint so all replicas are not on the node (or in the zone) being drained. The golden-path Helm chart should emit both by default.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ledger-api
  namespace: payments
spec:
  minAvailable: 2                 # replicas: 4 in the Deployment, so a drain can move two
  selector:
    matchLabels:
      app: ledger-api
---
# in the Deployment's pod template
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: ledger-api

Right-sizing and the three ways a node goes NotReady

Instance family choice is an operations decision, not a one-time architecture one: general purpose (m) for mixed tenants, compute-optimised (c) for CPU-bound services, memory-optimised (r) for JVM-heavy banking apps and caches, Graviton (m7g, c7g) when the images are multi-arch, for roughly 20% less. With Karpenter you list the families and generations you allow and let it pick per pending pod, and consolidation keeps replacing nodes with cheaper or fewer ones as load changes. With managed node groups you get one instance type list per group and right-size by hand from utilisation metrics.

When a node flips to NotReady on EKS, three causes cover most cases. First, the CNI: the aws-node pod on that node is crash-looping (IAM permission missing on the node role or the CNI's Pod Identity, or the ENI quota is exhausted), so the kubelet reports the network as not ready. Second, IP exhaustion in the subnet: the node stays Ready but every new pod on it sticks in ContainerCreating (the troubleshooting table below has the exact messages). Third, EBS attach limits: Nitro instances share a fixed number of attachment slots between ENIs and volumes, so a node dense with ENIs cannot attach more PVCs, and pods sit Pending with exceed max volume count. Each is visible in kubectl describe node Conditions and Events, then in the aws-node logs or the EBS CSI controller logs.

Security controls on EKS, and what the auditor asks for

On OpenShift most of these controls exist out of the box and you configure them (Post 22). On EKS you assemble them from AWS services and open-source components, and a bank will ask you to show evidence for each one. The table at the end of this section is the checklist; the prose walks it in the order an assessor would.

Secrets encryption at rest. EKS clusters created since early 2025 envelope-encrypt all Kubernetes API data at rest by default with an AWS-owned key; older clusters needed --encryption-config with a KMS key at creation. Either way a bank adds its own customer-managed KMS key (aws eks associate-encryption-config on an existing cluster, or encryption_config in Terraform) so it controls the key policy, rotation and the CloudTrail record of every Decrypt. Once associated, the key cannot be removed or swapped, so "we changed the KMS key later" is a tell.

Control plane logging. Five log types, off by default, each a stream in the /aws/eks/<cluster>/cluster log group: api, audit (who did what to which object, the one the bank cares about), authenticator (IAM-to-Kubernetes identity decisions, the one that debugs access entry problems), controllerManager and scheduler. Turn them all on, set retention, and ship audit and authenticator to the SIEM (a CloudWatch Logs subscription filter to Kinesis Data Firehose to Splunk, or to OpenSearch), because CloudWatch Logs is not where the security team lives.

$ aws eks update-cluster-config --name eks-payments-prod-cac1-01 \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
$ aws logs put-retention-policy --log-group-name /aws/eks/eks-payments-prod-cac1-01/cluster --retention-in-days 400
$ aws logs filter-log-events --log-group-name /aws/eks/eks-payments-prod-cac1-01/cluster \
    --log-stream-name-prefix kube-apiserver-audit --filter-pattern '{ $.objectRef.resource = "secrets" && $.verb = "get" }' \
    --query 'events[0].message' --output text | jq '{user: .user.username, ns: .objectRef.namespace, name: .objectRef.name, code: .responseStatus.code}'
{
  "user": "arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_PlatformAdmin_0123456789abcdef/j.doe",
  "ns": "payments",
  "name": "ledger-db",
  "code": 200
}

That query answers "who read the database secret last quarter". CloudTrail covers the other half: every call to the EKS API itself (CreateCluster, UpdateClusterVersion, CreateAccessEntry, DeleteNodegroup) is a management event in the organisation trail.

Least privilege for humans and pods. Humans come in through IAM Identity Center permission sets mapped to EKS access entries with the AWS-managed access policies (AmazonEKSClusterAdminPolicy for the platform team, AmazonEKSViewPolicy scoped to a namespace for a developer), reviewed quarterly like every other entitlement. Pods get AWS permissions through EKS Pod Identity (the eks-pod-identity-agent add-on plus an association between a service account and a role) or the older IRSA (IAM Roles for Service Accounts, via the cluster's OIDC provider), both from Post 27. Nobody, human or pod, holds a long-lived access key. The node role carries only what the node needs (ECR pull, CNI, SSM, CloudWatch agent), and IMDSv2 with a hop limit of 1 stops pods from borrowing the node's credentials through the metadata service:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: bottlerocket-hardened
spec:
  amiSelectorTerms:
    - alias: bottlerocket@1.44.0
  role: eks-payments-prod-node
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: eks-payments-prod-cac1-01 }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: eks-payments-prod-cac1-01 }
  metadataOptions:
    httpEndpoint: enabled
    httpTokens: required          # IMDSv2 only
    httpPutResponseHopLimit: 1    # pods cannot reach IMDS through the veth hop
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs: { volumeSize: 4Gi, volumeType: gp3, encrypted: true }
    - deviceName: /dev/xvdb
      ebs:
        volumeSize: 80Gi
        volumeType: gp3
        encrypted: true
        kmsKeyID: arn:aws:kms:ca-central-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

Endpoint and network perimeter. Private API endpoint only (endpointPublicAccess=false), reached over the Transit Gateway from the office VPN or a bastion account. Three security group layers: the cluster security group EKS creates for control-plane-to-node traffic, the node security group you manage, and security groups for pods (a SecurityGroupPolicy object binding a pod selector to specific SGs, so a database security group can allow one workload rather than every node). Inside the cluster, network policies give namespace isolation: the VPC CNI has a built-in engine (enableNetworkPolicy in the add-on configuration; it programs eBPF rules on the node), or you run Calico or Cilium for richer policy. A default-deny ingress policy per tenant namespace, with explicit allows, is the evidence the auditor wants.

Admission control. There is no SCC on EKS. SCC (Security Context Constraints) is OpenShift's own admission mechanism, evaluated per service account with a priority order; upstream Kubernetes uses Pod Security Admission (PSA), which enforces the three Pod Security Standards (privileged, baseline, restricted) per namespace through labels. PSA is coarser (no per-service-account grants, no custom profiles), so you pair it with Kyverno (YAML policies) or OPA Gatekeeper (Rego) for everything PSA cannot express: required labels, an image registry allowlist, digest pinning, no :latest, mandatory resource limits, no host ports.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.33
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: bank-ecr-and-digest-only
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              namespaces: ["kube-system", "karpenter"]
      validate:
        message: "Images must come from the bank's ECR registry and be pinned by digest."
        pattern:
          spec:
            containers:
              - image: "111122223333.dkr.ecr.ca-central-1.amazonaws.com/*@sha256:*"

$ kubectl -n payments apply -f deploy.yaml
Error from server: error when creating "deploy.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/payments/ledger-api was blocked due to the following policies

restrict-image-registries:
  autogen-bank-ecr-and-digest-only: 'validation error: Images must come from the bank''s ECR
    registry and be pinned by digest. rule autogen-bank-ecr-and-digest-only failed at path
    /spec/template/spec/containers/0/image/'

Image supply chain. ECR is the only registry the cluster can pull from (VPC endpoints for ecr.api, ecr.dkr and S3, no route to Docker Hub). Three ECR settings matter: immutable tags so v1.4.2 can never be overwritten; scanning (basic is free and OS-package only; enhanced hands the job to Amazon Inspector for continuous OS plus language-package scanning with Security Hub findings); and a pull-through cache so upstream images from ECR Public, Docker Hub or Quay are mirrored into ECR and scanned like everything else. "Was it scanned" is gated in the pipeline, not at the registry: CI scans, signs the image with cosign and attaches an attestation, and Kyverno's verifyImages rule refuses anything without a valid signature from the bank's key.

$ aws ecr put-image-tag-mutability --repository-name payments/ledger-api --image-tag-mutability IMMUTABLE
$ aws ecr put-registry-scanning-configuration --scan-type ENHANCED \
    --rules '[{"scanFrequency":"CONTINUOUS_SCAN","repositoryFilters":[{"filter":"*","filterType":"WILDCARD"}]}]'
$ aws ecr create-pull-through-cache-rule --ecr-repository-prefix docker-hub \
    --upstream-registry docker-hub --upstream-registry-url registry-1.docker.io \
    --credential-arn arn:aws:secretsmanager:ca-central-1:111122223333:secret:ecr-pullthroughcache/docker-hub-Ab12Cd
$ aws ecr describe-image-scan-findings --repository-name payments/ledger-api --image-id imageTag=v1.4.2 \
    --query 'imageScanFindings.findingSeverityCounts'
{
    "HIGH": 1,
    "MEDIUM": 4,
    "LOW": 11
}

Detection and compliance. GuardDuty EKS Protection has two halves: audit log monitoring, which reads the control plane audit stream directly (no CloudWatch logging needed) and raises findings such as Policy:Kubernetes/AnonymousAccessGranted or Execution:Kubernetes/ExecInKubeSystemPod; and runtime monitoring, an agent installed as a managed add-on that watches process, file and network activity on nodes (Execution:Runtime/NewBinaryExecuted). Security Hub aggregates those with AWS Config rule results, its Foundational Security Best Practices standard includes EKS controls, and the CIS Amazon EKS Benchmark is what kube-bench --benchmark eks-1.5.0 checks on the nodes. Run kube-bench as a CronJob and ship the report; that is the compliance-reporting automation the JD mentions.

Application secrets. Secrets belong in Secrets Manager or Parameter Store, not in Git and not typed into a Kubernetes Secret by hand. The External Secrets Operator syncs them into Kubernetes Secrets on a schedule so rotation reaches the pod; the Secrets Store CSI driver with the AWS provider mounts them as files without ever creating a Kubernetes Secret, which some security teams prefer. Both authenticate with Pod Identity or IRSA, so the only credential in the cluster is a role binding.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ca-central-1        # no auth block: the ESO pod's Pod Identity association is used
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ledger-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: aws-secrets-manager }
  target: { name: ledger-db }
  data:
    - secretKey: password
      remoteRef: { key: prod/payments/ledger-db, property: password }

Data on volumes. Turn on EBS encryption by default per region (aws ec2 enable-ebs-encryption-by-default) and still set encrypted: "true" and a KMS key in every StorageClass so the evidence is in Git. EFS file systems are created encrypted with a KMS key; a Config rule catches the exception.

Observability: assembling what OpenShift gives you for free

OpenShift ships Prometheus, Alertmanager, Thanos, Loki, Vector and console dashboards, wired and supported (Post 24). EKS ships an API server. Everything else is a decision, and the interviewer wants to hear that you made each one on purpose. There are two mainstream stacks and most banks run a blend.

CloudWatch Container Insights is the AWS-native path. The amazon-cloudwatch-observability managed add-on installs the CloudWatch agent (cluster, node, pod and container metrics, plus control plane metrics in the enhanced view) and Fluent Bit (container, node and data plane logs into four log groups under /aws/containerinsights/<cluster>/). Two commands and you have dashboards:

$ aws eks create-addon --cluster-name eks-payments-prod-cac1-01 --addon-name amazon-cloudwatch-observability \
    --pod-identity-associations serviceAccount=cloudwatch-agent,roleArn=arn:aws:iam::111122223333:role/eks-cloudwatch-agent
$ kubectl get pods -n amazon-cloudwatch
NAME                                                                READY   STATUS    RESTARTS   AGE
amazon-cloudwatch-observability-controller-manager-6c9d8f5b7-qx2lp  1/1     Running   0          3m
cloudwatch-agent-4x9kq                                              1/1     Running   0          3m
cloudwatch-agent-hs7tm                                              1/1     Running   0          3m
fluent-bit-2mnzr                                                    1/1     Running   0          3m
fluent-bit-p8vkd                                                    1/1     Running   0          3m
$ aws logs describe-log-groups --log-group-name-prefix /aws/containerinsights/eks-payments-prod-cac1-01 \
    --query 'logGroups[].[logGroupName,retentionInDays,storedBytes]' --output text
/aws/containerinsights/eks-payments-prod-cac1-01/application   30   184392011533
/aws/containerinsights/eks-payments-prod-cac1-01/dataplane     30   2210942201
/aws/containerinsights/eks-payments-prod-cac1-01/host          30   1809921355
/aws/containerinsights/eks-payments-prod-cac1-01/performance   30   40119320021

Amazon Managed Prometheus (AMP) plus Amazon Managed Grafana (AMG) is the path for teams that already speak PromQL and want the ecosystem's dashboards and alert rules. AMP is a Prometheus-compatible storage and query service; you push metrics to it with remote_write signed with SigV4, from a Prometheus you run in the cluster (kube-prometheus-stack), from an ADOT collector (AWS Distro for OpenTelemetry, the OTel collector with AWS exporters), or from an AMP managed scraper (aws amp create-scraper) with no agent to run at all. AMG is hosted Grafana behind IAM Identity Center, with AMP, CloudWatch, X-Ray and OpenSearch as data sources and no server to patch.

prometheus:
  prometheusSpec:
    externalLabels:
      cluster: eks-payments-prod-cac1-01
      environment: prod
    remoteWrite:
      - url: https://aps-workspaces.ca-central-1.amazonaws.com/workspaces/ws-0a1b2c3d-4e5f-6a7b-8c9d-0e1f2a3b4c5d/api/v1/remote_write
        sigv4:
          region: ca-central-1
        queueConfig:
          maxSamplesPerSend: 1000
          maxShards: 200
          capacity: 2500
    retention: 6h               # local buffer only; AMP is the store

Alerting: AMP has its own rules engine and Alertmanager, and the only receiver its Alertmanager supports is SNS, so SNS fans out to PagerDuty, Slack or the bank's ITSM. CloudWatch alarms cover the AWS-side signals (NAT gateway errors, load balancer 5xx, quota usage, log ingestion spikes) and page through the same topics. A starter set of alerts for an EKS platform, in Prometheus rule format:

groups:
  - name: eks-platform-starter
    rules:
      - alert: NodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 10m
        labels: { severity: critical }
        annotations: { summary: "Node {{ $labels.node }} NotReady for 10m" }
      - alert: VpcCniIpPoolNearlyExhausted
        expr: sum by (instance) (awscni_assigned_ip_addresses) / sum by (instance) (awscni_total_ip_addresses) > 0.9
        for: 15m
        labels: { severity: warning }
      - alert: PodsPendingTooLong
        expr: sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
        for: 15m
        labels: { severity: warning }
      - alert: CoreDnsLatencyHigh
        expr: histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m]))) > 0.1
        for: 10m
        labels: { severity: warning }
      - alert: ApiServerLatencyHigh
        expr: histogram_quantile(0.99, sum by (le, verb) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m]))) > 1
        for: 10m
        labels: { severity: critical }
      - alert: KarpenterNodeClaimFailures
        expr: increase(karpenter_nodeclaims_terminated_total{reason="launch_failed"}[30m]) > 3
        labels: { severity: warning }
      - alert: CertManagerCertExpiringSoon
        expr: certmanager_certificate_expiration_timestamp_seconds - time() < 7 * 24 * 3600
        labels: { severity: warning }

Add to that: PDB-blocked drains (a node cordoned for more than an hour), Argo CD applications out of sync or degraded, ESO sync failures, container restarts by namespace, and quota usage above 80%. Control plane metrics come two ways: scraped from the API server's /metrics endpoint (kubectl get --raw /metrics) and, since late 2024, as CloudWatch metrics in the AWS/EKS namespace, which is how you watch etcd database size and API server latency without running anything.

Logs are a routing decision. Fluent Bit is the DaemonSet whether it comes from the CloudWatch add-on or your own Helm release; its outputs decide where logs go: cloudwatch_logs for platform and application logs, opensearch if the bank runs OpenSearch, splunk or kinesis_firehose when the SIEM is Splunk. The number nobody budgets for is ingestion: CloudWatch Logs charges per GB ingested, and a chatty cluster's application log group easily reaches terabytes a month. Set retention on every log group (never "Never expire"), push debug-level logs to the Infrequent Access class or drop them at Fluent Bit, and sample the audit log before it reaches the SIEM.

Tracing: the ADOT collector accepts OpenTelemetry traces from apps and exports them to X-Ray or any OTLP backend; CloudWatch Application Signals layers service maps and SLOs on top. For an interview, know that OpenTelemetry is the vendor-neutral instrumentation, and that it is the app teams' instrumentation and your collector.

Storage operations: EBS, EFS, snapshots, Velero and DR

Storage basics were Post 7; on EKS the CSI drivers are managed add-ons and the operations questions are about defaults, resizing, backups and what happens when a region is gone.

EBS CSI (aws-ebs-csi-driver add-on, authenticated with Pod Identity) is the block storage for anything ReadWriteOnce. Older clusters still default to a gp2 StorageClass on the deprecated in-tree provisioner; the first thing a platform team does is make an encrypted gp3 class the default (cheaper per GB, baseline 3,000 IOPS regardless of size) with WaitForFirstConsumer so the volume is created in the zone where the pod actually lands:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  kmsKeyId: arn:aws:kms:ca-central-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
  tagSpecification_1: "BackupPolicy=daily"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshots
driver: ebs.csi.aws.com
deletionPolicy: Retain

$ kubectl patch storageclass gp2 -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
$ kubectl -n payments patch pvc ledger-db-data-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
$ kubectl -n payments get pvc ledger-db-data-0 -w
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
ledger-db-data-0   Bound    pvc-3c6f1d2e-8b7a-4f5e-9d0c-1a2b3c4d5e6f   100Gi      RWO            gp3-encrypted   210d
ledger-db-data-0   Bound    pvc-3c6f1d2e-8b7a-4f5e-9d0c-1a2b3c4d5e6f   200Gi      RWO            gp3-encrypted   210d

Resize is online (edit the PVC; the driver grows the EBS volume and the filesystem while the pod runs, one modification per volume every six hours). Snapshots need the external snapshot controller and CRDs installed separately from the add-on; a VolumeSnapshot object then becomes an EBS snapshot. For scheduled, policy-driven snapshots with vault lock and cross-account copy, AWS Backup selects EBS and EFS resources by tag (which is why the StorageClass tags every volume BackupPolicy=daily), and that is what auditors mean by "backup policy evidence".

EFS CSI is the ReadWriteMany option: a regional, multi-AZ NFS file system, with dynamic provisioning creating an access point (a per-PVC root directory with its own POSIX owner) per claim. Choose Elastic throughput unless a steady workload makes Provisioned cheaper, and remember EFS latency is milliseconds, not microseconds: shared content and uploads, not databases. EBS Multi-Attach is not RWX in any useful sense (io1/io2 only, same AZ, and the application must run a cluster-aware filesystem), so it is a niche answer, not the shared-storage answer.

Velero is the cluster-level backup tool: it serialises Kubernetes objects to S3 and either triggers CSI volume snapshots or copies file data with its built-in Kopia mover. Install it with a Pod Identity or IRSA role that can write the bucket and take snapshots, schedule it, and, the part most teams skip, restore from it regularly:

$ velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.12.0 \
    --bucket northbank-eks-velero-prod-cac1 --backup-location-config region=ca-central-1 \
    --snapshot-location-config region=ca-central-1 --use-volume-snapshots=true \
    --features=EnableCSI --service-account-name velero --no-secret
$ velero schedule create daily-all --schedule="0 2 * * *" --ttl 720h --exclude-namespaces kube-system,karpenter
Schedule "daily-all" created successfully.
$ velero backup create payments-pre-upgrade --include-namespaces payments --wait
Backup request "payments-pre-upgrade" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
...............
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe payments-pre-upgrade` and `velero backup logs payments-pre-upgrade`.
$ velero restore create --from-backup payments-pre-upgrade --namespace-mappings payments:payments-restore-test

DR strategy for EKS is "rebuild the cluster from code and restore the data", not "back up the cluster". The cluster is Terraform and Argo CD, so a new cluster in the DR region (ca-west-1 for a Canadian bank with data residency requirements) is a pipeline run. The data must already be there: ECR replication rules for images, S3 cross-region replication for the Velero bucket, AWS Backup copying EBS snapshots to a vault in the second region, and the databases (usually RDS or Aurora, outside the cluster) on their own replicas. Put numbers on it. RTO (recovery time objective = how long until service is back) and RPO (recovery point objective = how much data you may lose): a warm-standby design with a scaled-down DR cluster and continuous replication gives an RTO of tens of minutes and an RPO of minutes; rebuild-from-scratch gives an RTO of two to four hours and an RPO of the last Velero run, typically 24 hours. Business continuity assigns a tier per application; the platform publishes which tiers it can meet, tests twice a year and keeps the report. "We have backups" without a restore test is an audit finding.

Autoscaling: pods, nodes, and why Karpenter won

Three layers scale on EKS, and interviewers check you keep them apart. HPA (HorizontalPodAutoscaler, autoscaling/v2) scales replicas on CPU, memory or custom metrics; it needs metrics-server, now a managed add-on, and it is the app team's object except that you install the metric source. KEDA (Kubernetes Event-driven Autoscaling) extends HPA with event sources, so a consumer scales on SQS queue depth or Kafka lag and can go to zero between batches; the platform team installs KEDA and gives its operator an IAM role to read the queues, the app team writes a ScaledObject. VPA (VerticalPodAutoscaler) recommends or sets requests and limits from observed usage; run it in recommendation mode as a right-sizing input and never on the same metric as an HPA on the same workload.

Node scaling is where the platform decision lives. Cluster Autoscaler works through Auto Scaling groups: it finds pending pods, picks a node group whose template fits, and bumps that ASG's desired count. Reliable and boring, but each node group is one instance type (or a same-size family), scale-up takes a couple of minutes through the ASG, and it will not consolidate a half-empty fleet well. Karpenter talks to EC2 directly: for each batch of pending pods it computes the cheapest instance that fits from the whole list your NodePool allows (dozens of types, multiple families, Spot or On-Demand), launches it in tens of seconds through EC2 Fleet, and then keeps working: consolidation replaces underused nodes with smaller or fewer ones, drift replaces nodes whose spec changed, expireAfter caps node age for patch hygiene. Instance flexibility, speed, consolidation and lifecycle in one controller, kept polite by PDBs and disruption budgets, is why the industry moved. EKS Auto Mode is AWS running Karpenter, the load balancer controller and the storage drivers for you as part of the control plane, for a per-instance management fee, and it is where the platform roadmap is pointing.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: batch-spot
spec:
  template:
    metadata:
      labels: { workload-class: batch }
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: bottlerocket-hardened }
      taints:
        - key: workload-class
          value: batch
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]          # Spot first, On-Demand if Spot is unavailable
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small", "metal"]
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s                       # scale to zero the moment the last job finishes
  limits:
    cpu: "1000"

$ kubectl get nodeclaims
NAME               TYPE          CAPACITY   ZONE          NODE                                           READY   AGE
batch-spot-7hk2m   c6i.4xlarge   spot       ca-central-1a ip-10-42-21-140.ca-central-1.compute.internal  True    58s
batch-spot-x9pql   m6a.2xlarge   spot       ca-central-1b ip-10-42-22-17.ca-central-1.compute.internal   True    41s
general-2bnz8      m6i.2xlarge   on-demand  ca-central-1a ip-10-42-11-23.ca-central-1.compute.internal   True    12d

Scale-to-zero pools like this one are how a bank runs nightly batch: zero nodes until the CronJobs land, Spot capacity from Karpenter, and the pool empties itself at dawn. The interaction to remember: a PDB that cannot be satisfied blocks consolidation the same way it blocks an upgrade drain, so Karpenter's logs will say why a node it wants to remove is still there.

Cost management: what you actually pay for

An EKS bill has a shape, and the interviewer wants to hear that you know where the money hides:

The levers, in the order they usually pay off: tag-based visibility first, because you cannot cut what you cannot attribute; VPC endpoints for S3, ECR, STS, CloudWatch Logs and EC2 plus a pull-through cache, so NAT data processing collapses; a Compute Savings Plan for the baseline fleet (it covers EC2 and Fargate without locking you to an instance family the way a Reserved Instance does); Spot through Karpenter for every non-prod cluster and prod batch; right-sizing requests from VPA and Prometheus data, because requests, not usage, size the nodes; topology-aware routing so chatty services stay in zone; log retention and sampling; a scheduled cleanup of orphaned volumes and snapshots.

Showback = telling each team what their namespace costs without necessarily charging them; chargeback = actually billing the cost centre. AWS's own answer is split cost allocation data for EKS, which adds pod-level rows (with aws:eks:namespace, aws:eks:workload-name and cluster columns) to the Cost and Usage Report so Cost Explorer can group EC2 spend by namespace. Kubecost (commercial, with an AWS Marketplace integration that reads the real bill) and OpenCost (the CNCF project it grew from) give the same view in-cluster with a UI and an API, allocating node cost to namespaces by requests and usage, and both show the idle cost, the number that starts the right-sizing conversation.

The cost review ritual a platform team runs: monthly, thirty minutes, the platform lead and one person from FinOps or cloud engineering. Agenda: total EKS spend by cluster and by tag versus last month; top five namespaces by cost and by idle percentage; NAT and data-transfer lines; Savings Plan coverage and utilisation; Cost Anomaly Detection alerts; one action item per meeting with an owner. Publish the namespace showback to app teams the same day. Interviewers asking "how would you reduce EKS cost" are really asking whether you have a repeatable process; describe the ritual, then the levers.

EKS-specific troubleshooting: symptom, cause, command

The method is the one from Post 25 and Post 17: state the symptom, read Events before guessing, isolate the layer. EKS adds a layer most engineers have not debugged, the AWS side, so this table pairs each symptom with the AWS-side cause and the command that proves it.

$ kubectl -n payments describe pod ledger-api-7d9f8b6c5-x2k9p | sed -n '/^Events/,$p'
Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               4m12s               default-scheduler  Successfully assigned payments/ledger-api-7d9f8b6c5-x2k9p to ip-10-42-11-23.ca-central-1.compute.internal
  Warning  FailedCreatePodSandBox  4m11s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8f3c9a1e...": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  12s (x19 over 4m)   kubelet            (combined from similar events): Failed to create pod sandbox: ... failed to assign an IP address to container
$ aws ec2 describe-subnets --filters Name=tag:kubernetes.io/role/internal-elb,Values=1 \
    --query 'Subnets[].[SubnetId,AvailabilityZone,AvailableIpAddressCount]' --output text
subnet-0a1b2c3d4e5f6a7b8   ca-central-1a   3
subnet-1b2c3d4e5f6a7b8c9   ca-central-1b   0
subnet-2c3d4e5f6a7b8c9d0   ca-central-1d   7
$ aws eks update-addon --cluster-name eks-payments-prod-cac1-01 --addon-name vpc-cni --resolve-conflicts PRESERVE \
    --configuration-values '{"env":{"ENABLE_PREFIX_DELEGATION":"true","WARM_PREFIX_TARGET":"1"}}'

Prefix delegation makes the CNI allocate /28 prefixes instead of single IPs per ENI slot, which multiplies the pods per node without changing the subnet, but it needs contiguous free space; with a subnet at 0 free IPs the real fix is the secondary CIDR from Post 27, negotiated with cloud engineering. The pattern to say out loud: every row has a Kubernetes-side symptom and an AWS-side cause, and the fastest engineers check both in parallel, kubectl describe in one terminal and the relevant aws call in the other.

Governance and platform evolution

A platform that only reacts to tickets stays where it was built. The JD's "platform evolution" bullet asks whether you can move the platform forward deliberately, with other teams, and with a paper trail. Three tools do most of that work.

Architecture decision records (ADRs) = short, numbered documents in the platform repo, one per significant decision: context, options considered, decision, consequences. They are how a new engineer learns why prod runs Bottlerocket and why the payments cluster is separate, and they are the evidence an auditor accepts for "how are platform changes governed". Keep them to a page:

# ADR-014: Adopt EKS Pod Identity for new workloads; keep IRSA for existing ones

Status: Accepted (2026-02-18)
Context: 140 IRSA roles across 6 clusters; each new cluster needs OIDC provider and
         per-cluster trust policies; app teams copy trust policies incorrectly (12 tickets/quarter).
Options: (a) stay on IRSA; (b) Pod Identity for all, migrate existing; (c) Pod Identity for new,
         migrate opportunistically.
Decision: (c). New golden-path chart emits a Pod Identity association via Terraform module;
          IRSA remains supported until each workload's next major change.
Consequences: eks-pod-identity-agent add-on becomes mandatory on all clusters; roles need the
              pods.eks.amazonaws.com trust; Argo CD, ESO and Velero move first as reference examples.

A platform roadmap = the next four to six quarters of platform change, reviewed with architecture and cloud engineering and published to app teams. Typical 2025 to 2026 items, each a good interview example because each has a migration path and a reason:

• aws-auth ConfigMap → access entries: set authenticationMode to API_AND_CONFIG_MAP, create an access entry per mapping, verify with the authenticator log, then switch to API (one-way). Reason: IAM-native, auditable through CloudTrail, no more hand-edited ConfigMap outages.
• Cluster Autoscaler → Karpenter: run both, move workloads NodePool by NodePool with taints, shrink the managed node groups to the Karpenter controller itself. Reason: flexibility, speed, consolidation.
• IRSA → Pod Identity: the ADR above. Reason: fewer moving parts per cluster, simpler trust.
• Self-managed ingress and add-ons → EKS Auto Mode for new clusters: fewer components you patch, at a fee, once the security review clears it.
• AL2 → Bottlerocket or AL2023; one minor Kubernetes version per quarter; kube-bench and the CIS benchmark in the pipeline; Velero restore drills twice a year.

Collaboration with the two teams the JD names is a working rhythm, not a meeting. With cloud engineering: a shared backlog for network changes (secondary CIDRs, Transit Gateway routes, VPC endpoints, quota increases), an account-vending template that includes the EKS prerequisites (subnet tags, KMS key, log group, SSM settings) so a new environment account arrives ready, and a joint escalation path. With architecture: reference architectures you co-author (the standard EKS tenant, the DR pattern), a security review per roadmap item where you bring the ADR and the threat model, and trade-offs presented in a fixed shape: two or three options, cost, risk, operational load, your recommendation. Architects reward the engineer who brings the decision framed rather than the one who brings a demand.

Likely interview questions

Next up: Terraform for platform engineers, where the cluster factory from this post becomes real code: remote state and locking, modules for EKS and its add-ons, workspaces per environment, and pipelines that apply changes with approvals and drift detection.