Scenario Interviews: OpenShift and EKS Production Incidents

Scenario: "You started an OpenShift upgrade this morning. Six hours later oc get clusterversion still says Working towards 4.16.x, and the worker MachineConfigPool has been at 12 of 40 updated for three hours. What do you do?"

This normally means the control plane finished and the MCO is rolling workers one at a time with something blocking a drain. I'd confirm with oc get co (only machine-config should be Progressing) and oc get mcp worker, then oc describe mcp worker for the conditions and oc get nodes to find the node stuck SchedulingDisabled. The most common cause is a PodDisruptionBudget that cannot be satisfied: oc get pdb -A and look for ALLOWED DISRUPTIONS of 0, usually a single-replica app with minAvailable: 1. Other causes are a long terminationGracePeriodSeconds, a stuck finalizer, or an RWO volume that cannot detach. The MCD logs name the pod it is waiting for: oc logs -n openshift-machine-config-operator ds/machine-config-daemon filtered to the node. Mitigation is fixing the PDB or scaling the app with the owning team, not force-deleting production pods. Prevention: a pre-upgrade check that fails on any PDB with zero allowed disruptions, run in the change window before starting.

Scenario: "After the security team rotated a certificate, oc get co shows authentication Degraded and users cannot log in to the console. Users with existing tokens still work."

Existing tokens working tells me the API server is fine and the OAuth path is broken, so this is the cluster layer. I'd run oc get co authentication -o yaml and read the Degraded message; the authentication operator is unusually descriptive and names the route, ingress certificate or identity provider it cannot reach. Likely causes after a certificate change: the *.apps wildcard was replaced but the secret in openshift-ingress lacks the full chain or has a mismatched key; the OAuth server cannot validate the LDAP or OIDC provider because the CA ConfigMap in openshift-config still holds the old CA; or the API server named certificate was set without its intermediate. I'd check oc get pods -n openshift-authentication and their logs for x509 errors, and run openssl s_client against the OAuth route to see the served chain. Fix: replace the secret or ConfigMap with a correct chain and let the operator reconcile. Prevention: a certificate runbook with openssl verify and a post-change login test, plus an expiry alert from cluster monitoring.

Scenario: "A data-center power event took the whole cluster down. Power is back, VMs are up, but every node is NotReady and oc get co is a wall of Unknown. Where do you start?"

Control plane first, then workers. I'd check whether the API answers with oc get nodes; if not, oc debug node on a control-plane host or SSH via the bastion, then crictl ps for the static pods, etcd first, then kube-apiserver. Once the API answers, the two classic post-outage problems are pending CSRs and time skew. Kubelets whose certificates expired during the outage cannot rejoin until their CSRs are approved: oc get csr and, after confirming the requestors are my nodes, oc get csr -o name | xargs oc adm certificate approve, repeated because there are two rounds (client then serving). Time skew breaks TLS validation everywhere, so I'd verify chronyc tracking on each node and that the NTP servers are reachable through the firewall that also just rebooted. Then I'd watch oc get co settle and clear stuck pods. Prevention: a cold-start runbook tested in DR exercises, and monitoring for pending CSRs and clock drift so this is caught in minutes next time.

Scenario: "You get etcdHighFsyncDurations and etcdHighCommitDurations alerts, and developers say oc commands are taking ten seconds. What is happening and what do you do?"

etcd fsyncs every write to disk before acknowledging it, so slow disk means slow API for the whole cluster: a cluster-layer storage problem, not an application one. I'd confirm from inside etcd: oc rsh -n openshift-etcd etcd-<node>, then etcdctl endpoint status -w table and etcdctl endpoint health for DB size, leader changes and per-member latency. Mitigation depends on the cause. If one member's disk is degraded the leader keeps changing; I'd look at etcd_server_leader_changes_seen_total and that node's storage array or EBS volume. If the database is large, etcdctl defrag on each member, one at a time, non-leader first. If something is flooding the API, apiserver_request_total by user agent finds the controller in a tight loop. The permanent fix is storage that meets the requirement: dedicated SSD-class disks for control-plane nodes, validated with the fio benchmark Red Hat documents. Prevention: run that benchmark at node provisioning, keep etcd on separate disks from container storage, and alert on p99 fsync above 10 ms.

Scenario: "One of the three control-plane nodes has been lost, hardware is gone. Is the cluster okay? What do you do, and in what order?"

The cluster is up but fragile: etcd needs a majority, so two of three members keep quorum and the API works, but a second failure would freeze the cluster read-only. My first action is to reduce the risk window, not to rush. I'd confirm with oc get nodes, oc get co etcd -o yaml and, from a healthy member, etcdctl member list -w table showing the dead member unreachable. I'd take an etcd backup with cluster-backup.sh before touching anything. Then the documented "replace an unhealthy etcd member" procedure: etcdctl member remove <id>, delete its stale etcd secrets in openshift-etcd, and delete the corresponding Machine object so the Machine API provisions a replacement on IPI, or add a new host on UPI. The cluster-etcd-operator scales etcd back to three, which I'd watch with oc get etcd -o yaml and the member list. Prevention: automated daily etcd backups to off-cluster storage, control-plane nodes across failure domains, and a rehearsed member-replacement runbook so nobody improvises at 2 a.m.

Scenario: "The change advisory board asks you to prove that next week's OpenShift upgrade is safe. What evidence do you bring?"

A change record answering four questions: is the target supported, is the cluster healthy enough, what breaks, and how do we get back. Supported: the upgrade path from the Red Hat update graph and oc adm upgrade showing the recommended target on the chosen channel, plus the compatibility matrix for every OLM operator installed. Healthy: oc get co all Available and not Degraded, oc get mcp fully updated with no Degraded pools, oc get nodes all Ready, no pending CSRs, an etcd backup taken and verified, and Insights recommendations reviewed. What breaks: the deprecated API report from oc get apirequestcounts, plus results from the same upgrade on the non-production cluster with the same operators, including how long each pool took. Getting back: the honest answer that OpenShift does not support downgrading, so etcd restore is the last resort and the real safety net is the staging rehearsal and a paused worker pool so the control plane is verified before workers move. Then the runbook itself, with named owners, checkpoints and a stop condition.

Scenario: "You applied a MachineConfig to set the chrony servers on workers. Twenty minutes later the worker pool shows Degraded and one node is stuck. Walk me through it."

A MachineConfigPool goes Degraded when the machine-config-daemon cannot apply the rendered config on a node, and it stops there so the bad change does not spread. I'd start with oc get mcp worker and oc describe mcp worker, reading the NodeDegraded condition, then the exact error from the node annotation: oc get node <name> -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/reason}'. Common causes for a chrony change are a malformed Butane or Ignition file (bad base64 in the source data URL, wrong mode, missing overwrite: true when the file already exists) or "unexpected on-disk state," meaning someone hand-edited /etc/chrony.conf earlier and the MCD refuses to overwrite drift. I'd confirm in the MCD logs with oc logs -n openshift-machine-config-operator <mcd-pod>. Fix: correct the MachineConfig and re-apply, or delete it so the pool converges back; for drift, restore the file or force a reapply on that node. Prevention: butane --strict in CI, a small canary pool first, and no manual node edits.

Scenario: "A node has been SchedulingDisabled for two hours after an automated drain failed. Pods are still running on it. How do you handle it?"

The node is cordoned so nothing new lands there, which is why the platform still looks healthy; the drain is what failed. I'd confirm why from the drainer: if it was the MCO, oc describe mcp and the MCD logs; if a manual oc adm drain, its error names the pod it cannot evict. The usual culprit is a PodDisruptionBudget with zero allowed disruptions, visible with oc get pdb -A, or a pod with no controller that drain refuses to delete without --delete-emptydir-data and --force. My rule is to fix the cause with the owning team: scale the deployment to two replicas so the PDB is satisfiable, or agree to accept the disruption for a single-replica app in a window. Only then rerun oc adm drain <node> --ignore-daemonsets --delete-emptydir-data. If the maintenance is already done, oc adm uncordon restores capacity. Prevention: a policy that PDBs must allow at least one disruption, enforced by an admission policy or a scheduled report, so drains never block upgrades.

Scenario: "A worker's disk is full. Pods on it are being evicted with DiskPressure. What do you check and what is the fix?"

Disk pressure is the node layer, so I'd go to the node: oc describe node <name> for the DiskPressure condition, then oc debug node/<name> -- chroot /host df -h to see whether it is the root filesystem or /var/lib/containers. The two usual causes are logs and images. For logs, du -sh /var/log/pods/* finds the noisy pod, typically an app logging at debug level in a loop. For images, crictl images shows stale layers when garbage collection could not keep up, often after a busy release pulled many tags. Mitigation: crictl rmi --prune for unused images, and for logs ask the team to reduce the log level or restart the pod so the file rotates; do not delete files under a running container's log directory without understanding the effect on the collector. Prevention: a KubeletConfig CR setting containerLogMaxSize and containerLogMaxFiles, a tuned imageGCHighThresholdPercent, an alert on node filesystem usage at 80 percent, and a rule that apps log JSON to stdout rather than to files inside the container.

Scenario: "Security wants a kernel parameter set and SSH disabled on every node by Friday. It is Tuesday. How do you deliver it without breaking production?"

Both are node-level changes and both belong in MachineConfig, never in manual SSH sessions, because the MCO makes them declarative, automatic for new nodes, and auditable. The kernel parameter is a MachineConfig with spec.kernelArguments for the worker and master roles; SSH is a MachineConfig that masks the sshd unit and removes the core user's authorized keys. Every MachineConfig triggers a rolling reboot per pool, so the plan is about disruption, not YAML: apply to a small canary MachineConfigPool first, verify with oc debug node/ and cat /proc/cmdline, then roll infra and worker pools in a change window with maxUnavailable set so capacity stays safe, control plane last. I'd tell security the honest risk is losing break-glass access, so we agree the alternative: oc debug node/, which goes through the API and is audited, plus a documented path to re-enable SSH through MachineConfig. Audit evidence is the Compliance Operator scan showing the rule passing. If Friday is not safely achievable because of reboot windows, I say so on Tuesday.

Scenario: "A route returns 503 to users, but if you oc rsh into the pod and curl localhost it works. What is wrong?"

The pod test proves the workload layer healthy, so this is the network layer between router and pod. A 503 from the OpenShift router almost always means HAProxy has no healthy endpoints for the route. In order: oc get endpoints <svc>, empty if the service selector does not match the pod labels or the pods are not Ready; oc get route <name> -o yaml to compare the route's targetPort with the service port name and the container's real port; and oc get route -o jsonpath='{.status.ingress[*].conditions}' to confirm a router admitted it. The twist to mention: if the pod listens on 8080, the service says 80 and the route uses a named port that does not exist, endpoints exist but the router's health check fails, and oc logs -n openshift-ingress deploy/router-default shows the backend down. Fix the mismatch or the readiness probe and the 503 clears in seconds. Prevention: a golden-path Helm chart or template that wires port, service and route from one value so teams cannot mismatch them.

Scenario: "The route works from inside the cluster and from the bastion, but users on the corporate network get a connection timeout. Networking says nothing changed."

Route and pod are proven good, so the failure is between the corporate client and the router's load balancer; I'd split it into DNS, path and sharding. DNS: from a corporate machine, nslookup app.apps.cluster.bank.local; the *.apps wildcard may point at an old VIP or resolve differently on the corporate resolver than on the data-center one. Path: curl -v --resolve app.apps.cluster.bank.local:443:<vip> https://... to bypass DNS; a timeout there means a firewall or load-balancer rule, and I'd ask the network team for a flow check rather than guess. Sharding: oc get ingresscontroller -n openshift-ingress-operator shows whether a second router with its own domain and routeSelector exists; if the route carries a label a sharded router claims, it is served from that router's VIP, which the corporate firewall may not allow. endpointPublishingStrategy shows how the router is exposed. The fix is usually a DNS record or firewall rule owned by another team, so my job is precise evidence. Prevention: synthetic checks from a corporate vantage point, not only from inside the cluster.

Scenario: "Pods in the payments namespace cannot reach the on-prem Oracle database. The database team says nothing is blocked. Other namespaces are fine."

Egress from OVN-Kubernetes normally leaves with the node's IP, so if the Oracle firewall allows specific IPs the first suspect is EgressIP. oc get egressip shows in its status which node holds the address, and I'd confirm the namespace label matches the EgressIP's namespaceSelector. If the assigned node was drained or lost the k8s.ovn.org/egress-assignable label, the egress IP has no host and traffic falls back to node IPs the firewall rejects. Second suspect, policy inside the cluster: oc get networkpolicy -n payments for an egress rule limited to certain CIDRs, and oc get egressfirewall -n payments, the OVN object that restricts external destinations per namespace. Third, test from the pod with oc debug or oc rsh: nc -zv oracle.bank.local 1521 plus getent hosts for DNS. I'd bring the firewall team the exact source IP and timestamp so they can find the drop. Prevention: alert when an EgressIP has no assigned node, and keep EgressIP and EgressFirewall in the namespace's GitOps definition so they are reviewable.

Scenario: "Applications report intermittent DNS failures when calling external APIs. Maybe one in fifty requests fails with NXDOMAIN or a timeout. Internal service names are fine."

Intermittent failures for external names with internal names healthy point at the forwarding path from CoreDNS to the upstream resolvers, not at CoreDNS itself. I'd check oc get pods -n openshift-dns for restarts, oc logs -n openshift-dns ds/dns-default -c dns for SERVFAIL or upstream timeouts, and the metrics coredns_dns_request_duration_seconds and coredns_forward_responses_total by upstream. Two classic causes: one upstream in oc get dns.operator/default -o yaml is slow or flapping, so requests round-robined to it time out; and search-path amplification, where ndots:5 makes the pod try api.vendor.com.payments.svc.cluster.local and its siblings before the real name, multiplying upstream queries and hitting rate limits. I'd confirm the second from inside a pod with cat /etc/resolv.conf and a dig with and without the trailing dot. Fixes: remove or repair the bad upstream, use fully qualified names with a trailing dot or a lower ndots via dnsConfig, and enable CoreDNS caching TTLs. Prevention: alert on upstream forward error rate and add a DNS section to the golden path.

Scenario: "The PCI team wants their applications served by a dedicated router that only runs on their nodes and logs every request. How do you design it?"

This is router sharding, native to OpenShift through a second IngressController. I'd create one named pci in openshift-ingress-operator with its own domain such as pci.apps.cluster.bank.local, a routeSelector or namespaceSelector matching a label like zone: pci, a nodePlacement with a nodeSelector and tolerations for the PCI nodes, a strict tlsSecurityProfile, and logging.access forwarding HAProxy access logs to the log stack. The part people forget is the default router: it must exclude PCI routes with a matching NotIn expression, otherwise both routers admit the route and traffic leaks through the shared path. The corporate load balancer and DNS then point the PCI domain only at the new router's VIP. NetworkPolicy in the PCI namespaces allows ingress only from openshift-ingress pods carrying the pci deployment's label, not from any router. I'd deliver it as GitOps YAML with the PCI namespace template applying the zone label automatically. Evidence for auditors: the router's access logs in Splunk and a scan showing no PCI routes admitted by the default router.

Scenario: "A developer says the container runs fine with Docker on their laptop but goes CrashLoopBackOff on OpenShift. Logs say permission denied. What is happening?"

This is the most common onboarding incident, and the cause is the restricted-v2 SCC: OpenShift runs the container as a random non-root UID from the namespace's range, drops capabilities and forbids privilege escalation, while Docker on the laptop ran it as root. I'd confirm with oc logs --previous for the exact error and the openshift.io/scc annotation in oc get pod -o yaml. Three patterns cover almost every case: the process binds to port 80 or 443, which needs a capability the SCC drops, so bind to 8080; the process writes to a root-owned path, so make those directories group-writable in the Dockerfile with chgrp -R 0 /app && chmod -R g=u /app, because the random UID always belongs to group 0, or mount an emptyDir; or the image hardcodes USER root and a startup script that calls chown. The fix is the image, not the SCC. I'd point the team at a non-root base image such as nginx-unprivileged and the readiness checklist from Post 26. Prevention: an image linter in CI that flags root users and privileged ports before the first deploy.

Scenario: "The same developer now asks you to grant the privileged SCC to their service account because 'that makes it work.' What do you say?"

I'd say no, then do the work to find what they actually need. Privileged disables every isolation control: root, all capabilities, host namespaces, host paths. In a bank that is an audit finding and a lateral-movement path, with my name on the exception. The method: reproduce, read the real error from oc logs and the events, and run oc adm policy scc-subject-review -f deployment.yaml -z <sa> to see which SCC would admit the pod as written. Then match least privilege to the need: a fixed UID means fixing the image or, with justification, nonroot-v2; a specific root-owned UID means anyuid with a documented exception; one capability such as NET_BIND_SERVICE means a custom SCC adding only that, bound to the service account through a Role rather than by editing the SCC's users list. I'd record the exception with an owner, expiry and review date. Prevention: a published SCC decision tree in the onboarding docs so this conversation happens before go-live, and a quarterly security review of every non-default SCC binding.

Scenario: "After the internal registry's TLS certificate was rotated overnight, new pods across every namespace are ImagePullBackOff. Running pods are fine."

Running pods keep their image, so the cluster looks healthy but any restart or scale-up fails; this is a widening incident and I'd say so. oc describe pod confirms the message: x509: certificate signed by unknown authority means the nodes' CRI-O trust store lacks the new CA. On OpenShift, trusted registry CAs come from a ConfigMap in openshift-config referenced under additionalTrustedCA in oc get image.config.openshift.io/cluster -o yaml, keyed by registry hostname and port. If the new certificate came from a new CA, that ConfigMap still holds the old one. Fix: add the new CA to that ConfigMap; the MCO pushes it to every node's registry certificate directory, which I'd confirm with oc debug node/ under /etc/docker/certs.d/, then test with oc debug node/<n> -- chroot /host podman pull. If the registry itself serves an incomplete chain, openssl s_client -showcerts shows it and the registry team fixes it. Prevention: the rotation runbook includes updating the cluster CA ConfigMap and a test pull, plus an alert on cluster-wide ImagePullBackOff counts rising.

Scenario: "A team's pods started getting OOMKilled and evicted right after the platform team changed the namespace LimitRange. The team says they did not touch their deployment."

They are right, and the interviewer wants to hear me own it. A LimitRange sets default requests and limits for containers that declare none, applied at pod creation, so nothing changed until their next rollout or restart recreated pods with the new defaults. I'd confirm with oc describe limitrange -n <ns> and the resources block in oc get pod -o yaml, which shows the new smaller default memory limit stamped on the container. OOMKilled follows because the app needs more than the default; evictions can appear if the defaults changed the QoS class. Mitigation: raise the default back or, better, have the team set explicit resources in their deployment, which is what the LimitRange was meant to encourage. The process failure was rolling a namespace-wide change without checking who relied on the defaults. Prevention: before changing a LimitRange, query for containers without explicit resources in that namespace, communicate, and apply through GitOps with the namespace owner's review; longer term, the golden-path chart always sets resources so defaults are a safety net, not a dependency.

Scenario: "A PostgreSQL StatefulSet pod has been ContainerCreating for twenty minutes after its node failed. Events show a Multi-Attach error. What do you do?"

The volume is ReadWriteOnce and the storage system still believes it is attached to the dead node, so the new pod cannot attach it. StatefulSet pods are not force-rescheduled automatically when a node goes NotReady, precisely because two writers on one database volume would corrupt it. So before acting I confirm the old node is truly down, not just partitioned: oc get node, the hypervisor or cloud console, and oc get pods -o wide showing the old pod stuck Terminating. Only then do I force the old pod out with oc delete pod <name> --force --grace-period=0, check oc get volumeattachment for the stale attachment, and if the CSI driver does not clean it up, delete that VolumeAttachment so the controller detaches and reattaches. The new pod then starts. I'd say explicitly that I would never force this while the node might still be running the database. Prevention: fencing automation with MachineHealthCheck or the Node Health Check operator and a remediation provider, so a confirmed-dead node is deleted and its pods released within minutes, plus a database operator with replication so one pod failure is not a full outage.

Scenario: "An operator upgrade is stuck. The Subscription shows UpgradePending, there is an InstallPlan waiting, and the ClusterServiceVersion is in Failed. Walk me through it."

Two separate things are going on, and I'd separate them. First the InstallPlan: oc get installplan -n <ns>; if APPROVED is false, the Subscription uses installPlanApproval: Manual, common in banks so upgrades happen in change windows. After checking the change record, approve with oc patch installplan <name> -n <ns> --type merge -p '{"spec":{"approved":true}}'. Second the Failed CSV: oc describe csv <name> -n <ns> and read the reason. Common ones: the operator deployment cannot pull its image, a CRD conflict where a newer version owns the same CRD, a missing OperatorGroup or one whose install mode does not match, or the operator's own webhook not becoming ready. oc get catalogsource -n openshift-marketplace and oc get operatorgroup -n <ns> cover the OLM plumbing. Mitigation: fix the cause and delete the failed CSV so OLM recreates it from the Subscription, or at worst delete Subscription and CSV and reinstall the pinned version; the CRs and data remain. Prevention: pin operator channels, upgrade in staging first, and monitor csv_succeeded so a Failed CSV pages someone.

Scenario: "Prometheus in openshift-monitoring has been OOMKilled repeatedly and no alerts fired for four hours, including during a real outage. Nobody noticed. What went wrong and how do you fix it?"

Two failures: the monitoring stack ran out of memory, and the absence of alerts was itself not alerted on. I'd check oc get pods -n openshift-monitoring for restart counts and memory limits, then find what grew: prometheus_tsdb_head_series and the cardinality query topk(10, count by (__name__)({__name__=~".+"})) to find the metric family that exploded, almost always a new ServiceMonitor exposing a request-ID label or pod-per-series churn. Mitigation: drop the offending labels with metricRelabelings in that ServiceMonitor or remove it, and raise Prometheus resources through the cluster-monitoring-config ConfigMap so it comes back. Then the second failure: Alertmanager ships a Watchdog alert that fires continuously precisely so an external system can notice when it stops; I'd wire it to a dead man's switch receiver in the on-call tool that pages when the heartbeat goes silent. Prevention: user workload monitoring separated from the platform Prometheus so an app team cannot take down platform alerting, a cardinality budget per ServiceMonitor, and the heartbeat alert included in the platform's own health checks.

Scenario: "The security operations team says OpenShift logs stopped arriving in Splunk at 03:10. Applications are running normally. Where do you look?"

Logs flow from the Vector collector DaemonSet, configured by a ClusterLogForwarder, out to Splunk over HEC, so I'd walk that path outward. oc get clusterlogforwarder -n openshift-logging -o yaml status conditions say whether the output is validated and ready. oc get pods -n openshift-logging shows whether the collectors restarted at 03:10 or are CrashLoopBackOff. Then oc logs -n openshift-logging ds/<collector> for errors talking to Splunk: an expired or rotated HEC token, a TLS failure after Splunk's certificate changed, or 503 responses meaning Splunk's indexers are full and the collector is buffering. I'd also check whether logs still arrive in the in-cluster Loki store; if so, the cluster side is fine and the problem is the Splunk output or the network between them. Fix whichever it is and confirm with a test message in Splunk. Prevention: an alert on collector output errors and a Splunk-side alert for a source that goes silent, plus the forwarder in GitOps with the HEC token in a managed secret with an expiry reminder.

Scenario: "An auditor asks who deleted a Secret named payments-db-creds in the payments namespace last Tuesday. How do you answer?"

The API server audit log records every request with user, verb, object and timestamp, so this is answerable if the logs still exist. On OpenShift they live on the control-plane nodes: oc adm node-logs --role=master --path=kube-apiserver/ lists the files, then oc adm node-logs --role=master --path=kube-apiserver/audit.log piped through jq selecting .verb=="delete", .objectRef.resource=="secrets" and .objectRef.name=="payments-db-creds". The record gives the username or service account, source IP, user agent and whether it succeeded. If the deletion came through Argo CD, the user is the Argo service account and the real answer is in the Git history of the application repo, so I'd correlate timestamps. The catch is retention: node-local audit logs rotate within days, so for last Tuesday I'd go to Splunk, where the ClusterLogForwarder ships the audit input. I'd also verify the profile in oc get apiserver cluster -o yaml; the default logs metadata, and a bank usually wants WriteRequestBodies for changes. Prevention: audit logs forwarded off-cluster with retention matching the regulator, and a saved Splunk search for exactly this question.

Scenario: "On EKS, new pods are stuck ContainerCreating and the event says failed to assign an IP address to pod. The nodes have plenty of CPU. What is going on?"

This is the VPC CNI running out of IP addresses, because on EKS every pod gets a real VPC IP from the subnet through ENIs on the node. I'd confirm with kubectl describe pod, then check the two ceilings. Node ceiling: each instance type supports a fixed number of ENIs and IPs per ENI, so a small type simply cannot host more pods; kubectl describe node shows pod capacity. Subnet ceiling: no free addresses, visible in the VPC console or the available IP count from aws ec2 describe-subnets, and in kubectl logs -n kube-system ds/aws-node. Mitigation: scale into a subnet with space, or add a node group in a larger subnet. Durable fixes: prefix delegation on the CNI, which hands each ENI a /28 block instead of individual IPs and multiplies pod density on Nitro instances, and custom networking with a secondary CIDR such as 100.64.0.0/16 so pods stop consuming scarce routable addresses the bank's IPAM team guards. Prevention: monitor awscni_assigned_ip_addresses against capacity and plan subnets at design time, as in Post 27.

Scenario: "A pod using IRSA gets AccessDenied from S3. The developer insists the pod has the role. How do you debug it?"

IRSA is a chain of five links and any one breaks it, so I'd walk the chain. One: the ServiceAccount carries eks.amazonaws.com/role-arn with the right ARN, via kubectl describe sa. Two: the pod was created after the annotation, because the webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE at admission; kubectl exec and env | grep AWS proves it, and a restart fixes a pod that predates the annotation. Three: the role's trust policy names the cluster's OIDC provider, with sub matching system:serviceaccount:<ns>:<sa> exactly and aud equal to sts.amazonaws.com; a namespace typo is the classic. Four: the OIDC provider is actually registered in IAM for this cluster. Five: the permission policy and any SCP or bucket policy allow the action; aws sts get-caller-identity inside the pod shows whether I am the assumed role or fell back to the node role, the most common "has the role" mistake. Fix the failed link. Prevention: a Terraform IRSA module generating trust policy and annotation from one variable, and EKS Pod Identity as the newer path that removes the OIDC trust-policy step.

Scenario: "After upgrading the EKS control plane, the ingress controller is down and every external URL fails. What happened and how do you recover?"

Mitigate first: kubectl logs and kubectl describe on the controller pods in its namespace, and if rolling back the controller version brings it back, do that. The typical causes are compatibility, not the upgrade itself: the AWS Load Balancer Controller or NGINX ingress version does not support the new Kubernetes version, its webhook fails because a client-go API it relied on was removed, or a managed add-on such as vpc-cni, coredns or kube-proxy was left outside the supported matrix, checked with aws eks describe-addon-versions. Removed APIs are the other family: manifests still using a deprecated version fail to reconcile after the upgrade, and kubectl get events shows the conversion errors. Recovery: upgrade the controller Helm release and add-ons to compatible versions and reapply the manifests. Prevention is the senior part: run aws eks list-insights and a tool like pluto or kubent before upgrading, update add-ons and controllers in staging first, and treat the control-plane upgrade as one step in a versioned runbook covering data plane, add-on and controller compatibility, as in Post 28.

Scenario: "You updated a managed node group to a new AMI. The new instances launch, but kubectl get nodes never shows them and the old ones are being drained. Go."

Stop the bleeding first: pause the rollout so the last healthy nodes are not drained, by cancelling the update or scaling the old node group back up. Then the diagnosis, a node-layer bootstrap problem. aws eks describe-nodegroup shows health issues with reasons. On the instance, through SSM rather than SSH, journalctl -u kubelet and /var/log/cloud-init-output.log show whether bootstrap ran and what it complained about. Usual causes: the node IAM role lacks the required policies or, since access entries replaced the aws-auth ConfigMap, there is no access entry of type EC2_LINUX for the new role, checked with aws eks list-access-entries; the cluster is private and the new subnet or security group does not allow 443 to the control plane; a launch template whose custom user data does not bootstrap the new AMI family; or an AMI version newer than the control plane. Fix the cause and resume. Prevention: node group changes through Terraform with a plan review, rolled in staging first, with update_config.max_unavailable of one so a bad AMI cannot drain the cluster.

Scenario: "A bank security review asks how you would make an EKS cluster fully private and, more importantly, prove that nothing is reachable from the internet. What do you present?"

The controls, then evidence for each, because the second half is the real question. Controls: the cluster endpoint private only, so the API is reachable solely from the VPC and connected networks; nodes in private subnets with no public IPs and no internet gateway route; VPC endpoints for ECR, S3, STS, EC2, CloudWatch Logs and anything else the nodes need, so image pulls and IAM work without NAT to the internet; load balancers internal by default through annotations and enforced with policy; and security groups allowing only the expected paths. Evidence: aws eks describe-cluster showing endpointPublicAccess false, aws ec2 describe-route-tables for the node subnets showing no internet gateway, AWS Config managed rules for EKS endpoint public access and public IP assignment, Security Hub at zero findings for those controls, and a live test: a curl to the API endpoint and every load balancer hostname from outside the corporate network that fails. I'd add the Terraform module that produces this configuration so reviewers see it enforced by code, and a scheduled compliance report that re-checks it. Details are in Post 28.

Scenario: "terraform plan in the production pipeline says it wants to destroy and recreate the EKS node group. What do you check before you let anyone apply it?"

Nobody applies it until I know why. The plan marks the attribute forcing replacement with a comment like # forces replacement, so I read that line first. Common triggers: a change to an immutable attribute such as the node group name, subnet list or launch template name; a provider upgrade that renamed an attribute so Terraform sees a difference; or drift where someone changed the node group in the console and the code now disagrees. I'd compare with terraform state show for the resource and with the console. If the change is intentional, replacement still destroys every node in the group, so the safe pattern is adding a new node group alongside, migrating workloads with cordon and drain respecting PDBs, then removing the old one: two separate plans. If accidental, a moved block, lifecycle { ignore_changes } for the drifted attribute, or reverting the code removes the replacement. Prevention: lifecycle { prevent_destroy = true } on production node groups, a pipeline step that fails any plan showing a destroy, and a second approver for plans with replacements, as in Post 29.

Scenario: "A colleague's Terraform apply is failing with Error acquiring the state lock. The lock ID belongs to a pipeline run from yesterday. What do you do?"

The lock exists so two applies cannot corrupt the state, so I treat it as a signal, not an obstacle. The error prints who holds it, the operation and the timestamp. I'd confirm the holder is genuinely dead by opening yesterday's pipeline run and checking it was cancelled or the runner died mid-apply, and that no apply is in progress anywhere else on that state. Only after that confirmation do I run terraform force-unlock <lock-id>, and I'd say the phrase out loud: I'd confirm the holder is dead before force-unlocking, because unlocking under a live apply is how you get a corrupted state file. Then terraform plan to see whether the failed apply left state and the real world out of sync. The lock lives in DynamoDB for the S3 backend or, on Terraform 1.10 and later, in the S3 native lock file if enabled, so I could also inspect the lock item directly. Prevention: pipelines that always run with -lock-timeout, a concurrency group so only one apply per state runs at a time, bucket versioning for state recovery, and a runbook entry so nobody needs to guess.

Scenario: "An Argo CD application flips to OutOfSync every minute, self-heals, and flips again. The developers say Git has not changed. What is happening?"

Something in the cluster mutates the live object after every sync, so Argo sees a diff, corrects it, and the mutator changes it back. I'd start with argocd app diff <app>, because the differing field names the culprit. The classics: a HorizontalPodAutoscaler managing replicas while the manifest also sets it; a mutating admission webhook injecting a sidecar, label or default; a controller writing to metadata.annotations; or, on OpenShift, fields the API server or an operator fills in, such as a Route's host or a Service's clusterIP. Two Applications owning the same resource with different content give the same symptom, which argocd app list filtered by namespace reveals. The fix is to stop fighting: remove replicas from the manifest when an HPA exists, or add an ignoreDifferences entry in the Application with a JSON pointer or jq path for the mutated field, combined with the RespectIgnoreDifferences=true sync option so the sync itself does not overwrite it. Prevention: a base Application template in the app-of-apps carrying the known ignore rules for OpenShift-mutated fields, and a Grafana alert on sync frequency so flapping is caught early, as in Post 30.

Scenario: "During a Sev-1 last month, an engineer fixed a bad environment variable with oc edit. Two minutes later Argo CD reverted it and the outage got worse. What went wrong, and how would you handle it now?"

Two things went wrong and neither is the engineer. First, the Application had automated sync with self-heal on, which is the right default, but nobody had a documented break-glass path for incidents, so the only fast option was a manual edit that GitOps is designed to erase. Second, the incident bridge did not know self-heal was on, so the platform's behavior surprised its own operators. Now the runbook offers two sanctioned moves. The fast one is an emergency commit to the environment repo with a pre-agreed expedited approval, so the fix lands through the same path and Argo applies it in seconds, keeping the audit trail the bank's change process requires anyway. The other is to pause self-heal for that one application with argocd app set <app> --sync-policy none or by disabling auto-sync in the UI, make the live change, reconcile Git, then re-enable, with the pause and reason recorded in the incident timeline. I'd add an alert when any production application has auto-sync disabled for longer than an hour, so break-glass cannot quietly become permanent. The RCA action is the runbook and the training, not a rule against oc edit.

Scenario: "A critical CVE in a widely used base image affects about 150 services across the cluster. Security gives you a seven-day SLA. How do you run it?"

This is a programme, not a ticket, so I'd run it like an incident with a coordinator, a tracker and a daily update. Day one is inventory: pull every running image with oc get pods -A -o jsonpath, match against the scanner's findings from Red Hat Advanced Cluster Security, Quay, ECR scanning or Trivy, and group by base image and owning team; 150 services becomes perhaps five base images and twenty teams. Day one also ranks exposure: internet-facing services and those handling regulated data go first. The fix is rebuilding: the platform team publishes patched base images, CI pipelines that pin the base image by tag rebuild on the new digest, and hand-built Dockerfiles get a scripted pull request. Argo CD promotes the new images through environments with the normal gates shortened but not skipped. Verification is the scanner at zero remaining instances, plus an exception list for anything not rebuilt in time, each with an owner and a compensating control. Prevention: an ACS admission policy that blocks images with critical CVEs after a grace period, automatic base image rebuilds on upstream updates, and the inventory script as a scheduled report, as in Post 31.

Scenario: "A developer committed a production database password to a Git repository. It was pushed an hour ago. What do you do, in order?"

The password is compromised the moment it was pushed, regardless of whether anyone saw it, so step one is rotation, not cleanup. The database team rotates the credential and updates consumers through the secrets path, which on a well-run platform means Vault or AWS Secrets Manager with External Secrets, so applications pick up the new value without a manual edit. Step two is containment: check the repository's visibility and access logs for who cloned or viewed it, and the database audit for logins with that account from unexpected sources during the exposure window. Step three is cleanup: remove the secret from history with git filter-repo or BFG, which needs a force-push and coordination with everyone holding a clone, and I'd say clearly this is hygiene, not protection, because forks and CI caches may still hold it. Step four is the record: a security incident ticket, the timeline and the RCA. Prevention is where the value is: pre-commit secret scanning with a tool like gitleaks, push protection on the Git server so the commit is rejected, a rule that secrets never live in manifests, and no blame for the developer who reports it fast.

Scenario: "The Compliance Operator's CIS scan shows 40 failed checks and the external audit is next month. How do you get to a defensible state?"

Forty failures is normal for a first scan, and the audit does not need zero; it needs every item either fixed or justified. I'd list them with oc get compliancecheckresult -n openshift-compliance sorted by severity and split them into three buckets. Bucket one is automatic remediations: oc get complianceremediation -n openshift-compliance shows what the operator can apply, mostly MachineConfig or KubeletConfig changes; I'd review each, apply by setting the remediation's apply field or enabling auto-apply on the ScanSetting, and schedule the reboots per pool in change windows. Bucket two is manual fixes such as the API server audit profile, encryption at rest or RBAC cleanup, each with an owner and a date. Bucket three is checks that do not apply to this environment, handled with a TailoredProfile that disables the rule and a written justification the auditor can read. Then rerun with oc compliance rerun-now and export evidence with the oc compliance fetch-raw plugin for the ARF reports. Prevention: scheduled scans forwarded to the compliance dashboard, and any new failed check opening a ticket automatically so the count never reaches forty again.

Scenario: "A new application team has to be live on OpenShift in two weeks and has never used it. They currently deploy with Docker Compose on VMs. What is your plan?"

Two weeks is achievable only with a golden path, so I would design nothing new; I'd run the onboarding playbook from Post 26 and be honest about scope. Week one: a kickoff mapping their Compose services to deployments, services and routes; projects created through the project template with quota, LimitRange, default NetworkPolicies and the RBAC group from the corporate directory; the platform Helm chart or Kustomize base that already produces a restricted-v2-compatible workload with probes, resources and a route; and a workshop on the three things that break Compose apps on OpenShift: running as root, privileged ports, and writing to the image filesystem. Their CI builds images into the corporate registry with scanning; Argo CD deploys from their environment repo through the app-of-apps. Week two: a readiness review against the checklist, load and failover tests, log and metric verification in Splunk and Grafana, the on-call handoff with their runbook, and the go-live change record. I'd push back on anything needing a new SCC or a bespoke network design in two weeks; those need their own timeline. The output is also a better checklist for the next team.

Scenario: "Two teams share a cluster. One team's batch jobs keep starving the other team's customer-facing API. Both say the platform is at fault. How do you fix it?"

Both are partly right: the platform let it happen. I'd confirm with oc adm top pods -n <batch-ns> and node-level metrics that the batch pods consume the shared capacity, and check whether they set requests at all, because pods without requests get BestEffort QoS and crowd a node while the scheduler thinks it is empty. Then the controls, in layers. ResourceQuota on the batch namespace caps their total CPU and memory, and a LimitRange forces requests so scheduling is honest. PriorityClasses give the API pods higher priority so the scheduler preempts batch pods when capacity is tight. If the interference is on the node rather than in scheduling: dedicated nodes for the API team through a taint and toleration plus the project's default node selector annotation, and an infra pool for the batch jobs. Batch jobs also get a concurrency limit in their CronJob settings and off-peak windows. All of it lives in the namespace's GitOps definition, with quota-usage alerts at 80 percent so the batch team hears about their ceiling before hitting it. The answer to "who is at fault" is a platform standard applied to every tenant, which is what the governance bullet in the job description means in practice.