Chapter 25
The OpenShift Troubleshooting Playbook and Incident Response
Before you read, guessWhich initial commands identify the problem layer before symptoms are described?
Take ten seconds and guess — even a wrong guess makes the answer stick. Tap to see where the chapter lands, or just read on.
The first five commands (oc get clusterversion, oc get co, oc get nodes, oc get mcp, filtered oc get pods -A) tell you which layer you are in within a minute and before anyone describes the symptom to you.
The job description for this role says two things in plain language: "comfortable leading troubleshooting efforts and supporting critical production environments" and "production incident response, root cause analysis." Every other post in this track teaches you a subsystem. This one teaches you what to do at 2 a.m. when the payments team says the platform is broken and a director is on the bridge. By the end you will have a single ordered method you can state in one sentence, the exact commands for each layer, ten worked incidents with their root causes, and the vocabulary a bank uses to run an incident from first page to postmortem. That combination is what separates a candidate who has read about OpenShift from one who has been on call for it.
The interviewer is grading your method, not your answer
A scenario question in this interview will sound like: "A team says their app is down on OpenShift. What do you do?" Notice what is missing: everything. No error message, no namespace, no timeline. That is deliberate. The interviewer is not testing whether you know the answer to a puzzle with one solution. They are watching how you think when you know almost nothing, because that is exactly the condition you will be in during a real incident.
Candidates who fail this question do one of two things. They guess ("probably DNS") and start defending the guess, or they list every command they know in no particular order. Candidates who pass state a method first, then apply it, narrating what each step would tell them and what they would do with each possible result. You can get the final diagnosis wrong and still pass if the method was sound. You can get it right by luck and still fail if you skipped straight to it.
So the first thing to memorize in this post is not a command. It is a sentence.
The four-layer method, said out loud
Here it is: "I check cluster health, then node health, then the network path, then the workload, and I always read Events before I guess." Say it in the interview before you touch a single command. Then walk the layers.
- Cluster health = is the control plane itself healthy? The API server, etcd, the ClusterVersion Operator and the cluster operators that manage every platform component. If this layer is sick, every symptom below it is a consequence, not a cause.
- Node health = are the machines that run pods Ready, not under pressure, and running the configuration the Machine Config Operator expects? A NotReady node explains a hundred pod symptoms at once.
- Network path = can a packet get from the client to the pod? External load balancer, router, Route, Service, endpoints, DNS, NetworkPolicy, OVN. Most "the app is down" reports where the pod is actually Running live here.
- Workload = the pod, its image, its probes, its resources, its permissions. Only when the three layers above are clean do you spend time on the application itself.
Why this order and not the reverse? Because a fault at a higher layer explains many symptoms below it, and the checks at the higher layers are cheap. Five commands tell you the state of the entire cluster. Reading one application's logs tells you about one application. You start with the widest, cheapest view and narrow down, so you never spend forty minutes reading Java stack traces while the real problem is that half the worker nodes are NotReady.
"Events before guesses" is the discipline that holds the method together. Kubernetes writes down what it tried and why it failed: the scheduler records why it could not place a pod, the kubelet records why it could not pull an image or mount a volume, the ReplicaSet controller records why it was forbidden from creating a pod. That record is free to read and is almost always more accurate than your intuition. Every time you are tempted to guess, run oc describe or oc get events instead.
The first five commands you always run
Before you ask the reporting team a single question, run these five. They take under a minute and tell you which layer you are in.
1. Is the cluster upgrading or unhappy about its version? ClusterVersion = the object the Cluster Version Operator (CVO) uses to record the desired and actual OpenShift version. If PROGRESSING is True, an upgrade is in flight and much of what you see next is expected churn.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.16.20 True False 13d Cluster version is 4.16.202. Which platform components are complaining? ClusterOperator = one object per platform component (authentication, ingress, monitoring, network, etcd and so on) reporting Available, Progressing and Degraded. A healthy cluster shows every row as True False False. Anything else is your starting point.
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.16.20 True False True 4m OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.ocp-prod.example.internal/healthz": x509: certificate signed by unknown authority
console 4.16.20 True False False 13d
dns 4.16.20 True False False 13d
etcd 4.16.20 True False False 13d
image-registry 4.16.20 True False False 13d
ingress 4.16.20 True False False 6m
kube-apiserver 4.16.20 True False False 13d
machine-config 4.16.20 True False False 13d
monitoring 4.16.20 True False False 13d
network 4.16.20 True False False 13dThe MESSAGE column is often the diagnosis. Here, authentication went Degraded four minutes ago with an x509 error against the OAuth route, and ingress changed state six minutes ago. Two facts, one story: somebody changed the ingress certificate.
3. Are the nodes Ready?
$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0.ocp-prod.example.internal Ready control-plane,master 201d v1.29.10+67d3387
master-1.ocp-prod.example.internal Ready control-plane,master 201d v1.29.10+67d3387
master-2.ocp-prod.example.internal Ready control-plane,master 201d v1.29.10+67d3387
worker-0.ocp-prod.example.internal Ready worker 201d v1.29.10+67d3387
worker-1.ocp-prod.example.internal Ready,SchedulingDisabled worker 201d v1.29.10+67d3387
worker-2.ocp-prod.example.internal NotReady worker 201d v1.29.10+67d3387
worker-3.ocp-prod.example.internal Ready worker 201d v1.29.10+67d3387One node NotReady and one cordoned. A cordoned node (SchedulingDisabled) with no upgrade in progress is suspicious: either an operator is draining it or a human forgot to uncordon it after maintenance.
4. Is the Machine Config Operator mid-rollout or stuck? MachineConfigPool (MCP) = the group of nodes (master, worker, or a custom pool) that share one rendered machine configuration. When you change a MachineConfig or upgrade, the MCO drains and reboots nodes one pool at a time, and the pool's status tells you where it is.
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-6f1a2c9d0b8e4f7a3c5d1e2b9a8c7d6e True False False 3 3 3 0 201d
worker rendered-worker-9b3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f False True False 4 3 2 0 201dThe worker pool is UPDATING: two of four nodes have the new config, and the MCO is working on the rest. That explains the cordoned node from the previous command, and it tells you not to "fix" the cordon by uncordoning it yourself.
5. Which pods are unhappy, anywhere? Filter out the healthy states and what remains is your worklist.
$ oc get pods -A | grep -v -E 'Running|Completed'
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-monitoring prometheus-k8s-0 5/6 CrashLoopBackOff 14 3h
payments-prod payments-api-7d9c8b6f5-k2x4p 0/1 ContainerCreating 0 22m
payments-prod payments-api-7d9c8b6f5-m9q1z 0/1 Pending 0 22m
risk-batch nightly-var-28471520-xk7lp 0/1 Init:Error 0 6hSixty seconds in, you know whether this is a cluster problem, a node problem or a workload problem, and you have not had to trust anyone's description of the symptom. Now you go down the layers.
oc adm cordon a node, or scale a deployment to an image tag that does not exist, and rerun the five. The goal is to make the "healthy" output so familiar that anything abnormal jumps off the screen without you reading every line.Layer 1: cluster health
Layer 1 is where you rule out the platform before you blame the application. Three questions: are the operators healthy, is the control plane fast, and are the certificates valid?
Reading a Degraded ClusterOperator
A ClusterOperator has three conditions you care about. Available = the component is serving. Progressing = the operator is changing something (rolling out a new version or config). Degraded = the operator has been unable to reach its desired state for long enough to complain. Degraded with Available still True means "working, but not the way I was told to work", which is the common case; Available False is a real outage of that component.
oc describe co gives you the full condition messages, which are much more useful than the truncated MESSAGE column.
$ oc describe co authentication
...
Status:
Conditions:
Last Transition Time: 2026-09-08T14:02:11Z
Message: OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.ocp-prod.example.internal/healthz": x509: certificate signed by unknown authority
Reason: OAuthServerRouteEndpointAccessibleController_SyncError
Status: True
Type: Degraded
Last Transition Time: 2026-09-08T13:58:40Z
Message: All is well
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2026-08-26T09:14:02Z
Message: OAuthServerDeploymentAvailable: availableReplicas==3
Reason: AsExpected
Status: True
Type: Available
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: authentications
Group:
Name: openshift-authentication
Resource: namespacesTwo habits here. First, the Reason string names the controller inside the operator that is unhappy, and searching the Red Hat Knowledgebase for that exact string usually lands on a solution article. Second, Related Objects lists the namespaces and custom resources the operator manages; that is where you go next (oc get pods -n openshift-authentication, oc get authentication.operator cluster -o yaml). The same pattern works for ingress (related namespace openshift-ingress-operator and openshift-ingress), monitoring (openshift-monitoring) and every other operator.
The CVO and upgrade status
CVO = the Cluster Version Operator, the operator that manages all the other operators and drives upgrades by applying the release payload's manifests in order. When an upgrade is running, oc adm upgrade is the first status view.
$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.16.22: 763 of 894 done (85% complete), waiting on machine-config
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.16 (available channels: candidate-4.16, candidate-4.17, eus-4.16, fast-4.16, fast-4.17, stable-4.16, stable-4.17)
No updates available. You may still upgrade to a specific release image, but doing so may not be supported and may result in a cluster that cannot be upgraded."Waiting on machine-config" means the CVO has handed control to the MCO, which is rebooting nodes, and the upgrade will sit at this percentage until every pool finishes. A stuck upgrade is nearly always an MCO drain problem, which is Layer 2. The full story of upgrades, the MCO and pools lives in Post 20.
API slow? Look at etcd first
If oc commands take seconds, the console spins, and controllers seem to lag, the API server is the symptom and etcd is the usual cause. etcd = the key-value database that holds every Kubernetes object; the API server cannot answer a request faster than etcd can commit or read. etcd is extremely sensitive to disk write latency, because every write must be fsynced to disk on a quorum of members before it is acknowledged.
Check the etcd pods and then get inside one. In OpenShift the etcd pods ship etcdctl with its endpoints and certificates preconfigured, so you can query cluster status directly.
$ oc get pods -n openshift-etcd -l app=etcd
NAME READY STATUS RESTARTS AGE
etcd-master-0 4/4 Running 0 13d
etcd-master-1 4/4 Running 0 13d
etcd-master-2 4/4 Running 0 13d
$ oc rsh -n openshift-etcd etcd-master-0
sh-5.1# etcdctl endpoint status -w table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.0.11:2379 | 8e9e05c52164694d | 3.5.14 | 412 MB | false | false | 42 | 98217634 | 98217634 | |
| https://10.0.0.12:2379 | 91bc3c398fb3c146 | 3.5.14 | 412 MB | true | false | 42 | 98217634 | 98217634 | |
| https://10.0.0.13:2379 | fd422379fda50e48 | 3.5.14 | 413 MB | false | false | 42 | 98217631 | 98217631 | |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
sh-5.1# etcdctl endpoint health
https://10.0.0.11:2379 is healthy: successfully committed proposal: took = 9.812ms
https://10.0.0.12:2379 is healthy: successfully committed proposal: took = 8.104ms
https://10.0.0.13:2379 is healthy: successfully committed proposal: took = 41.337msWhat you are reading: three members, one leader, same raft term (no leader elections churning), and applied index within a few entries of each other (no member falling behind). The health check's "took" is a quick latency indicator; the third member at 41 ms is a hint that its disk is slower. The authoritative signals are the alerts etcdHighFsyncDurations and etcdHighCommitDurations (Red Hat's guidance is that 99th-percentile WAL fsync should stay under 10 ms) and, for a growing database, etcdDatabaseHighFragmentationRatio. A large DB size with high fragmentation is a defrag conversation, not an emergency; high fsync is a storage conversation and shows up as scenario (f) below. Quorum loss, the case where fewer than two of three members are healthy, is a documented recovery procedure with a backup restore, and it is worth reading in Post 20 before you ever need it.
etcdctl endpoint status, find the member and the disk that are slow, and fix the cause. Restarting is not a diagnosis. An interviewer who hears "restart etcd" as a first step will conclude you have never operated a control plane.Certificates: x509 errors and pending CSRs
OpenShift rotates its internal certificates automatically, but two situations surface as incidents. The first is a custom certificate you introduced (ingress, API server, an internal registry) whose chain is incomplete or whose CA the cluster does not trust; you see x509: certificate signed by unknown authority in operator conditions. The second is a cluster that was powered off long enough that the kubelet client certificates expired while it was down; when nodes come back, the kubelet cannot authenticate, requests a new certificate, and the request sits waiting for approval. CSR = CertificateSigningRequest, the object a kubelet creates to ask the cluster to sign a certificate for it.
$ oc get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-2k7vn 6m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-9pw4x 5m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-hj7f2 2m kubernetes.io/kubelet-serving system:node:worker-0.ocp-prod.example.internal <none> PendingPending CSRs plus NotReady nodes is a pattern you should recognize instantly. Scenario (c) works through it.
Cluster-wide Events, sorted by time
The last Layer 1 habit: read the most recent Events across all namespaces. This is the "Events before guesses" rule applied at cluster scale, and it frequently tells you what changed and when.
$ oc get events -A --sort-by=.lastTimestamp | tail -n 8
openshift-ingress 14m Normal Killing pod/router-default-5c7b9d8f6-x2v4q Stopping container router
openshift-ingress 14m Normal Scheduled pod/router-default-7f8a6c5d4-p9m3k Successfully assigned openshift-ingress/router-default-7f8a6c5d4-p9m3k to worker-3.ocp-prod.example.internal
openshift-ingress 13m Normal Started pod/router-default-7f8a6c5d4-p9m3k Started container router
openshift-authentication 4m Warning Unhealthy pod/oauth-openshift-6d9b8c7f5-k4j2h Readiness probe failed: Get "https://10.128.2.19:6443/healthz": x509: certificate signed by unknown authority
payments-prod 2m Warning FailedScheduling pod/payments-api-7d9c8b6f5-m9q1z 0/7 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 1 node(s) were unschedulable, 5 Insufficient memory.Four minutes of reading and you have a timeline: the router restarted fourteen minutes ago (a certificate change), OAuth started failing its readiness probe four minutes ago, and a payments pod cannot schedule because one node is unreachable, one is cordoned and the other five are out of memory. The pods for the control plane itself live in openshift-kube-apiserver, openshift-etcd, openshift-kube-controller-manager and openshift-kube-scheduler; if Layer 1 points at them, oc get pods -n openshift-kube-apiserver and oc logs on the relevant pod is the next stop.
Layer 2: node health
A node is where the platform meets Linux. OpenShift nodes run RHCOS (Red Hat Enterprise Linux CoreOS) = an immutable, image-based operating system managed by the MCO, with CRI-O as the container runtime and the kubelet as the agent. You do not SSH in and edit config files; you read the node through oc and, when needed, get a shell with oc debug.
NotReady and the Conditions block
oc describe node shows Conditions, and the Conditions block answers "why is this node NotReady" nine times out of ten.
$ oc describe node worker-2.ocp-prod.example.internal
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Mon, 08 Sep 2026 09:41:12 -0400 Mon, 08 Sep 2026 09:42:03 -0400 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Mon, 08 Sep 2026 09:41:12 -0400 Mon, 08 Sep 2026 09:42:03 -0400 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Mon, 08 Sep 2026 09:41:12 -0400 Mon, 08 Sep 2026 09:42:03 -0400 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Mon, 08 Sep 2026 09:41:12 -0400 Mon, 08 Sep 2026 09:42:03 -0400 NodeStatusUnknown Kubelet stopped posting node status.
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
node.kubernetes.io/unschedulable:NoScheduleThree distinct stories hide in this block. Kubelet stopped posting node status with everything Unknown means the control plane has lost contact: the node is off, partitioned from the network, or the kubelet has died. MemoryPressure True, DiskPressure True or PIDPressure True means the kubelet is alive but the node is running out of that resource and is evicting pods to protect itself. And a node that is Ready but tainted unschedulable is simply cordoned. Note the LastTransitionTime: it is the moment the node went bad, which you will need for the incident timeline.
Getting onto the node the OpenShift way
oc debug node/<name> starts a privileged pod on the node with the host filesystem mounted at /host. chroot /host makes it feel like a real shell on the node, with systemctl, journalctl and crictl available. This works even for a node the API considers NotReady, as long as the kubelet can still start a pod; if it cannot, you need the cloud console or out-of-band access.
$ oc debug node/worker-2.ocp-prod.example.internal
Starting pod/worker-2ocp-prodexampleinternal-debug-x7k2p ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.1.22
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# systemctl status kubelet crio --no-pager | grep -E 'Loaded|Active'
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2026-09-08 09:47:51 EDT; 2s ago
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; preset: disabled)
Active: active (running) since Mon 2026-09-08 09:41:02 EDT; 7min ago
sh-5.1# journalctl -u kubelet --since "15 min ago" --no-pager | grep -iE 'error|fail' | tail -n 3
Sep 08 09:47:49 worker-2 kubelet[3812]: E0908 09:47:49.113 "Failed to start ContainerManager" err="failed to get rootfs info: unable to find data in memory cache"
Sep 08 09:47:49 worker-2 kubelet[3812]: E0908 09:47:49.114 kubelet.go:1466] "Failed to start ContainerManager" err="[open /sys/fs/cgroup/kubepods.slice: no space left on device]"
Sep 08 09:47:51 worker-2 systemd[1]: kubelet.service: Failed with result 'exit-code'.Kubelet is crash-looping on "no space left on device". Now the rest of the node checklist confirms and locates it.
sh-5.1# df -h /sysroot /var /var/lib/containers
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 120G 120G 14M 100% /sysroot
/dev/sda4 120G 120G 14M 100% /sysroot
/dev/sda4 120G 120G 14M 100% /sysroot
sh-5.1# du -xsh /var/lib/containers/storage /var/log /var/lib/kubelet 2>/dev/null
96G /var/lib/containers/storage
1.2G /var/log
14G /var/lib/kubelet
sh-5.1# crictl ps --state exited | wc -l
212
sh-5.1# crictl images | wc -l
418
sh-5.1# chronyc tracking | grep -E 'Reference|System time|Leap'
Reference ID : 0A000005 (ntp1.example.internal)
System time : 0.000212 seconds slow of NTP time
Leap status : Normal
sh-5.1# nmcli device status
DEVICE TYPE STATE CONNECTION
ens192 ethernet connected Wired connection 1
br-ex ovs-bridge connected br-ex
lo loopback unmanaged --Each command has a reason. df -h confirms the full disk (RHCOS puts everything on one root filesystem by default, so /var/lib/containers filling up starves the kubelet too). du finds the culprit: 96 GB of container storage with hundreds of exited containers and images the kubelet's garbage collector has not reclaimed because it is not running. crictl = the CLI for the container runtime; crictl ps, crictl pods and crictl logs <container-id> let you read containers even when the kubelet and API cannot. top catches a runaway process. chronyc tracking matters more than it looks: certificate validation and etcd leader elections both break when a node's clock drifts, and a bank's NTP source is frequently a corporate server that a firewall change can silently block. nmcli confirms the primary interface and the br-ex OVS bridge OVN-Kubernetes uses; if br-ex is missing or down, that node has no pod networking at all.
Kernel and driver issues (a NIC driver resetting, a kernel oops) live in journalctl -k. On a cloud or vSphere platform, also check the provider side: a node that vanished without any kubelet errors was often terminated or vMotioned by something outside the cluster, and oc get machine -n openshift-machine-api will show the Machine's phase and provider state.
MCP Degraded, stuck drains and evictions
When the MCO cannot apply a MachineConfig to a node, it marks the node and then the pool Degraded. The two common causes are a bad MachineConfig (a typo in an Ignition file, a systemd unit that fails, a kernel argument that prevents boot) and a drain the MCO cannot complete because a PodDisruptionBudget refuses to let a pod be evicted. Both are diagnosed the same way: oc describe mcp worker for the pool's conditions, then the machine-config-daemon logs on the affected node.
$ oc get mcp worker -o jsonpath='{range .status.conditions[?(@.status=="True")]}{.type}: {.message}{"\n"}{end}'
Degraded: Node worker-1.ocp-prod.example.internal is reporting: "failed to drain node: worker-1.ocp-prod.example.internal after 1 hour. Please see machine-config-controller logs for more information"
NodeDegraded: Node worker-1.ocp-prod.example.internal is reporting: "failed to drain node ..."
$ oc logs -n openshift-machine-config-operator -l k8s-app=machine-config-controller --tail=200 | grep -E 'evict|drain' | tail -n 3
I0908 10:12:44.120 drain_controller.go:182] node worker-1.ocp-prod.example.internal: evicting pod payments-prod/payments-ledger-0
E0908 10:12:44.131 drain_controller.go:182] node worker-1.ocp-prod.example.internal: error when evicting pods/"payments-ledger-0" -n "payments-prod" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0908 10:12:49.140 drain_controller.go:182] node worker-1.ocp-prod.example.internal: evicting pod payments-prod/payments-ledger-0A node stuck in SchedulingDisabled after a failed drain is the same problem from the other side. Do not uncordon it to "unstick" the upgrade; the MCO will cordon it again. Fix the PDB (scenario (a)), or if a human drained it manually and left, oc adm uncordon is appropriate once you have confirmed nobody is mid-maintenance.
Evictions are the kubelet protecting the node: under MemoryPressure or DiskPressure it evicts BestEffort and then Burstable pods, and the evicted pod stays visible with STATUS Evicted and a message naming the resource. Many Evicted pods across nodes at once means the cluster is over-committed, which is a capacity and quota conversation with the app teams, not a bug.
Reboot or replace?
Reboot when the node's state is corrupt but the machine is fine: a full disk after cleanup, a wedged kubelet, a stale network state. Cordon, drain, reboot, uncordon.
$ oc adm cordon worker-2.ocp-prod.example.internal
$ oc adm drain worker-2.ocp-prod.example.internal --ignore-daemonsets --delete-emptydir-data --force --timeout=300s
$ oc debug node/worker-2.ocp-prod.example.internal -- chroot /host systemctl reboot
$ oc adm uncordon worker-2.ocp-prod.example.internalReplace when the machine itself is suspect (hardware faults, a hypervisor that keeps killing it, an RHCOS install that is corrupt) and the node belongs to a MachineSet. Delete the Machine object, and the MachineSet creates a fresh node from the same template; the MCO configures it, and it joins with the correct rendered config. This is the cattle-not-pets move that OpenShift's Machine API gives you, and it is usually faster and safer than forensics on a broken box.
$ oc get machine -n openshift-machine-api | grep worker-2
ocp-prod-x9k2f-worker-0-worker-2 Running m5.2xlarge ca-central-1 ca-central-1a 201d
$ oc delete machine ocp-prod-x9k2f-worker-0-worker-2 -n openshift-machine-api
machine.machine.openshift.io "ocp-prod-x9k2f-worker-0-worker-2" deletedThe Machine API drains the node before deleting it. If the node is dead and the drain would hang forever, annotate the Machine with machine.openshift.io/exclude-node-draining first. A MachineHealthCheck can automate this for nodes that stay NotReady past a threshold, which you would propose in the prevention section of an RCA rather than enable during an incident.
oc debug node for read-only investigation. Saying "SSH and edit" tells the interviewer you have run kubeadm clusters but not OpenShift.Layer 3: the network path
By the time you reach Layer 3, the cluster is healthy and the nodes are Ready, and yet the app "is down". The most common reason is that the pod is fine and the path to it is not. The rule for this layer is to trace the path hop by hop from the outside in, confirming each hop before moving to the next, so you find the exact hop where the request dies. Post 21 explains every one of these components in depth; here is how you walk them under pressure.
Route returns 503, 502 or 504
The router = the HAProxy-based ingress controller in openshift-ingress that terminates Routes. The status code it returns tells you which hop to suspect. A 503 "Application is not available" page is generated by the router itself and means it has no healthy backend for that Route: no endpoints. A 502 means the router reached a backend and the backend answered with garbage or reset the connection (wrong port, TLS mismatch on a re-encrypt Route, app crashed mid-request). A 504 means the backend accepted the connection and never answered in time (app hung, a downstream dependency timing out, or the Route's timeout annotation is shorter than the app's response time). The walk, in order:
$ oc get pods -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-7f8a6c5d4-p9m3k 1/1 Running 0 2h
router-default-7f8a6c5d4-w6r8n 1/1 Running 0 2h
$ oc get route payments-api -n payments-prod
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
payments-api payments-api-payments-prod.apps.ocp-prod.example.internal payments-api 8080 edge None
$ oc get svc,endpoints payments-api -n payments-prod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/payments-api ClusterIP 172.30.114.52 <none> 8080/TCP 42d
NAME ENDPOINTS AGE
endpoints/payments-api <none> 42d
$ oc get pods -n payments-prod -l app=payments-api
NAME READY STATUS RESTARTS AGE
payments-api-7d9c8b6f5-k2x4p 0/1 Running 0 31m
payments-api-7d9c8b6f5-m9q1z 0/1 Running 0 31mRouter pods healthy, Route exists and points at the right Service and port, Service exists, endpoints empty, pods Running but 0/1 Ready. The last scan is the readiness probe: the pods are alive but not Ready, so the endpoints controller leaves them out of the Service and the router has nowhere to send traffic. If the endpoints had been empty while the pods were 1/1, the diagnosis would be a selector mismatch between Service and pods instead. Then confirm the app is genuinely serving by talking to it directly, bypassing the whole path:
$ oc rsh -n payments-prod payments-api-7d9c8b6f5-k2x4p curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/actuator/health
200
$ oc get pod -n payments-prod payments-api-7d9c8b6f5-k2x4p -o jsonpath='{.spec.containers[0].readinessProbe}'
{"failureThreshold":3,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}The app answers on /actuator/health; the probe asks for /health. That is scenario (d), and it takes four minutes with this method versus an afternoon of "but it works on my laptop". For router-side evidence, oc logs -n openshift-ingress deploy/router-default shows backend health and reloads, and oc get ingresscontroller default -n openshift-ingress-operator -o yaml shows the router's own configuration.
DNS
Cluster DNS in OpenShift is CoreDNS, run by the DNS operator as the dns-default DaemonSet in openshift-dns, with a node-resolver DaemonSet that lets the nodes themselves resolve the internal registry. In-cluster names (svc.cluster.local) are answered locally; anything else is forwarded to the upstream resolvers, which on a bank's network are corporate DNS servers behind a firewall.
$ oc get pods -n openshift-dns -o wide | head -n 4
NAME READY STATUS RESTARTS AGE IP NODE
dns-default-4kx9p 2/2 Running 0 13d 10.128.0.7 master-0.ocp-prod.example.internal
dns-default-7qm2w 2/2 Running 0 13d 10.129.2.4 worker-0.ocp-prod.example.internal
dns-default-9zt3c 2/2 Running 0 13d 10.131.0.9 worker-1.ocp-prod.example.internal
$ oc rsh -n payments-prod payments-api-7d9c8b6f5-k2x4p
sh-5.1$ cat /etc/resolv.conf
search payments-prod.svc.cluster.local svc.cluster.local cluster.local ocp-prod.example.internal
nameserver 172.30.0.10
options ndots:5
sh-5.1$ nslookup payments-db.payments-prod.svc.cluster.local
Server: 172.30.0.10
Address: 172.30.0.10#53
Name: payments-db.payments-prod.svc.cluster.local
Address: 172.30.201.18
sh-5.1$ nslookup ratesfeed.vendor-example.com
;; connection timed out; no servers could be reachedInternal resolution works, external does not: the problem is the upstream, not CoreDNS. oc get dns.operator/default -o yaml shows spec.upstreamResolvers; if it is empty, CoreDNS uses the node's /etc/resolv.conf, which you check from an oc debug node shell. Note the ndots:5 line: any name with fewer than five dots is first tried with every search domain appended, so one external lookup becomes five queries, and a slow or rate-limited upstream turns that into visible latency. Scenario (i) is the intermittent version of this.
NetworkPolicy, EgressIP and OVN
If DNS and the Route are fine but a pod cannot reach another pod or an external system, check policy before you check plumbing. A NetworkPolicy denies silently: no error, no event, just a timeout.
$ oc get networkpolicy -n payments-prod
NAME POD-SELECTOR AGE
allow-from-openshift-ingress <none> 42d
allow-same-namespace <none> 42d
deny-by-default <none> 42d
$ oc run nettest -n payments-prod --rm -it --restart=Never --image=registry.access.redhat.com/ubi9/ubi-minimal -- \
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://risk-engine.risk-prod.svc.cluster.local:8080/ping
000
pod "nettest" deletedThe debug pod in payments-prod cannot reach risk-prod, and the three policies explain why: the namespace allows ingress from the router and from itself, and nothing else. The fix is a policy in risk-prod allowing ingress from payments-prod, requested through the app team, not a policy you delete during the incident. Cross-namespace calls that "used to work" and stopped are nearly always a new default-deny policy applied by a golden-path template (see Post 26).
EgressIP = a fixed source IP that OVN-Kubernetes assigns to traffic leaving the cluster from selected namespaces, so a bank firewall can allowlist it. If a firewall suddenly blocks an app, check that the EgressIP is actually assigned: oc get egressip shows the assigned node in its status, and it will be blank if no node carries the k8s.ovn.org/egress-assignable label or the labeled nodes are NotReady. For OVN itself, oc get pods -n openshift-ovn-kubernetes should show every ovnkube-node pod Running with all containers ready and the ovnkube-control-plane pods healthy; oc get co network reports the operator's view. A single node whose ovnkube-node pod is crash-looping means pods on that node have no networking, which looks like a random subset of an app being unreachable.
External load balancers and TLS
Two hops sit outside the cluster. The external load balancer (F5, cloud LB, HAProxy) in front of the routers performs its own health checks against the router pods, and if those checks target a port or path that changed, the LB pulls all routers out of rotation and every Route on the cluster returns an LB-generated error at once. Check the LB's pool status before you check anything inside the cluster when everything fails simultaneously. TLS problems have three shapes: a certificate chain missing its intermediate (browsers work because they cache intermediates, curl and Java clients fail), an SNI mismatch (the Route hostname does not match the certificate's names, common after a wildcard change), and on re-encrypt Routes, a destinationCACertificate that no longer matches the pod's serving certificate after the app team rotated it. openssl s_client -connect host:443 -servername host -showcerts from outside and oc get route x -o yaml for the TLS block settle all three. When you need everything about the network layer captured for support, oc adm must-gather -- gather_network_logs collects OVN databases, flows and node network state.
Layer 4: the workload
Only now do you look at the application itself. The Kubernetes half of this you already know from Post 9, Post 14 and Post 17; the table is a compressed refresher with the first check for each state. The OpenShift-specific states after it are the ones an interviewer for this role will probe.
| Pod state | What it means | First check | Usual causes |
|---|---|---|---|
| Pending | Scheduler has not placed it | oc describe pod Events: FailedScheduling message | Insufficient CPU/memory, taint without toleration, node affinity unsatisfiable, unbound PVC, quota exceeded |
| ContainerCreating | Scheduled, kubelet cannot start it | Events: FailedMount, FailedAttachVolume, FailedCreatePodSandBox | Volume attach/mount failure, missing ConfigMap or Secret, CNI failure on that node |
| ImagePullBackOff / ErrImagePull | Image cannot be pulled | Events: the registry's exact error | Wrong name or tag, missing pull secret, registry CA not trusted, registry down, ImageStream import failed |
| CrashLoopBackOff | Container starts, exits, restarts with backoff | oc logs --previous | App error, missing config, permission denied as non-root, liveness probe too aggressive |
| OOMKilled (in Last State) | Kernel killed it for exceeding memory limit | oc describe pod Last State: Reason OOMKilled; oc adm top pod | Limit too low, LimitRange default, memory leak, JVM heap not sized to the cgroup |
| Error | Container exited non-zero and will not restart (Job or restartPolicy Never) | oc logs, exit code in describe | Script failure, bad arguments, dependency unreachable |
| Completed unexpectedly | Main process exited 0 in a long-running workload | oc logs, check the command and entrypoint | Wrong command, app started in foreground mode that returns, missing -D/daemon flags |
| Terminating (stuck) | Deletion requested, not finishing | oc get pod -o yaml: finalizers, node status | Node NotReady (kubelet cannot confirm), finalizer held by a dead controller, volume unmount hung |
| Init:Error / Init:CrashLoopBackOff | An init container failed | oc logs pod -c <init-name> | Migration script failed, waiting on a dependency that never appears, permission denied |
| Unknown | Kubelet stopped reporting the pod | oc get node for the pod's node | Node NotReady or partitioned; it is a Layer 2 problem |
| Evicted | Kubelet removed it to relieve node pressure | oc describe pod Message names the resource | MemoryPressure/DiskPressure on the node; pod has no or low requests (BestEffort goes first) |
| CreateContainerConfigError | Kubelet cannot build the container spec | Events: the missing object's name | ConfigMap or Secret key referenced by env does not exist; invalid securityContext |
OpenShift-specific workload failures
SCC denial. The SecurityContextConstraint = OpenShift's admission control that decides what a pod is allowed to request (UID, capabilities, host access, volume types). The default for ordinary service accounts is restricted-v2, which forbids running as root or as a fixed UID. When a manifest asks for something the SCC does not allow, the pod is never created; the error is on the ReplicaSet, and oc get pods shows nothing at all, which confuses people who expect a failing pod.
$ oc get events -n payments-prod --sort-by=.lastTimestamp | tail -n 2
2m Warning FailedCreate replicaset/legacy-batch-6b7c8d9e5 Error creating: pods "legacy-batch-6b7c8d9e5-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000680000, 1000689999], provider restricted-v2: .containers[0].capabilities.add: Invalid value: "NET_BIND_SERVICE": capability may not be added]
$ oc get pod -n payments-prod payments-api-7d9c8b6f5-k2x4p -o yaml | grep openshift.io/scc
openshift.io/scc: restricted-v2The message tells you exactly which field violated which SCC. The fix is to make the image work as an arbitrary UID (scenario (e)), or, when there is a documented reason, to bind a less restrictive SCC to that workload's service account via a role, never to a user and never cluster-wide. The second command shows which SCC admitted a running pod, which is how you prove to an auditor what a workload is running under. Post 22 covers SCCs fully.
Arbitrary UID permission denied. The pod is admitted, then the app crashes with Permission denied writing to a directory the image owns as root. Same root cause as above, seen from inside the container.
ImageStream and registry problems. ImageStream = OpenShift's pointer to images, which can import from external registries on a schedule. oc describe is app -n ns shows import errors such as Import failed (Unauthorized) or x509; oc import-image app:latest --confirm retries it. Pods pulling from the internal registry reference image-registry.openshift-image-registry.svc:5000/ns/app@sha256:..., and if that fails, check oc get co image-registry and its storage.
BuildConfig failures. oc get builds -n ns shows phase; oc logs build/app-7 shows the build pod's output. Failures are typically a base image pull error, a Dockerfile step failing, or a push to the internal registry rejected because the builder service account lacks rights or the registry is full.
Quota exceeded. Like an SCC denial, this is a ReplicaSet FailedCreate event: exceeded quota: compute-resources, requested: limits.memory=2Gi, used: limits.memory=7Gi, limited: limits.memory=8Gi. oc describe quota -n ns shows the usage; the fix is a quota change through the team's onboarding process, not an emergency edit.
LimitRange defaults. LimitRange = per-namespace defaults for requests and limits applied to containers that do not set their own. A JVM container with no explicit memory limit inherits, say, 512Mi from the LimitRange, sizes its heap from the cgroup and is OOMKilled under load. oc get limitrange -n ns -o yaml reveals the default that the app team never knew existed.
PDB blocking a rollout. oc get pdb -n ns with ALLOWED DISRUPTIONS 0 and a rollout that never progresses, because the old pods cannot be evicted. Also scenario (a) at cluster scale.
HPA flapping. oc describe hpa shows the metric and conditions; a missing CPU request (missing request for cpu) makes the HPA unable to compute utilization, and a target set near the app's idle usage makes it scale up and down every few minutes.
Probes misconfigured. Liveness probes that hit an endpoint doing real work, or with timeoutSeconds: 1 against a slow app, produce restarts that look like crashes. oc describe pod Events say Liveness probe failed before every restart, which is the tell.
The workload commands, in the order you use them
$ oc describe pod -n payments-prod payments-api-7d9c8b6f5-k2x4p | sed -n '/Last State/,/Ready/p;/Events/,$p'
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 08 Sep 2026 11:02:14 -0400
Finished: Mon, 08 Sep 2026 11:09:51 -0400
Ready: False
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 12s (x14 over 6m) kubelet Back-off restarting failed container payments-api in pod payments-api-7d9c8b6f5-k2x4p_payments-prod
$ oc logs -n payments-prod payments-api-7d9c8b6f5-k2x4p --previous --tail=5
11:09:49.882 WARN [reconcile-7] c.e.p.LedgerCache - cache size 1,204,118 entries (heap 96%)
Terminating due to java.lang.OutOfMemoryError: Java heap space
$ oc adm top pod -n payments-prod --containers
POD NAME CPU(cores) MEMORY(bytes)
payments-api-7d9c8b6f5-k2x4p payments-api 412m 1019Mi
$ oc rollout history deployment/payments-api -n payments-prod
REVISION CHANGE-CAUSE
41 <none>
42 <none>
$ oc rollout undo deployment/payments-api -n payments-prod
deployment.apps/payments-api rolled back
$ oc rollout status deployment/payments-api -n payments-prod
deployment "payments-api" successfully rolled outThe sequence: describe for the state and Events, --previous logs for the reason the last container died, oc adm top pod for what it is using right now (this is kubectl top backed by OpenShift's metrics stack, which is always present), then history and undo when the rollout that introduced the problem is the fastest thing to reverse. Mitigation first; the memory investigation can wait until the service is back.
When you need a shell, oc rsh is kubectl exec -it with a saner default shell. When the container is crash-looping and there is nothing to exec into, or when you need root to inspect a permission problem, use oc debug: it creates a copy of the pod (from a deployment, a pod, or a node) with the command replaced by a shell, so you can poke at the filesystem, environment and mounts of a container that would otherwise never stay up.
$ oc debug deployment/legacy-batch -n payments-prod --as-root
Starting pod/legacy-batch-debug-t8x2m, command was: /opt/app/run.sh
Pod IP: 10.129.2.44
If you don't see a command prompt, try pressing enter.
sh-5.1# ls -ld /var/lib/app
drwxr-xr-x. 2 root root 6 Aug 30 10:11 /var/lib/app
sh-5.1# id
uid=0(root) gid=0(root) groups=0(root)--as-root only works if your own account is allowed an SCC that permits it (cluster-admin is), which is exactly why it is a platform engineer's tool and not one the app team can use to bypass policy. This is the OpenShift way to debug: no privileged sidecar images, no changing the real deployment's security context to "just look".
Storage and registry incidents, briefly
Storage incidents share one habit: read the PVC's Events, then the CSI driver's pods. PVC Pending means no volume was provisioned; oc describe pvc shows either waiting for a volume to be created (the provisioner is slow or broken: check the CSI driver pods in openshift-cluster-csi-drivers or the vendor's namespace) or storageclass not found (the class name is wrong or a default class is missing). Multi-Attach is the RWO volume problem after a node failure, scenario (g). The internal registry running out of storage shows up as builds failing to push and the image-registry operator going Degraded; oc adm prune images (run from a machine with access to the registry route) reclaims space, and the long-term fix is a pruning schedule and larger backing storage. On nodes, the kubelet's image garbage collection starts when the disk passes 85% used and stops at 80% by default; if a node fills up faster than GC runs, you get DiskPressure evictions, and the answer is a larger disk, a separate /var/lib/containers partition through a MachineConfig, or app teams using fewer unique image tags. All of this is covered in depth in Post 21.
Gathering data for Red Hat support
Red Hat support is part of the platform at a bank, and using it well is a skill the interviewer will ask about. The tool they will expect you to know is must-gather = a command that runs a collection image on the cluster and downloads a snapshot of every cluster operator's state, logs, nodes, events and custom resources into a local directory.
$ oc adm must-gather --dest-dir=./mg-2026-09-08-auth
[must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:...
ClusterID: 3f9d1c2e-7b4a-4e8f-9a0b-1c2d3e4f5a6b
ClusterVersion: Stable at "4.16.20"
ClusterOperators:
clusteroperator/authentication is degraded because OAuthServerRouteEndpointAccessibleControllerDegraded: ...
[must-gather ] OUT namespace/openshift-must-gather-x7k2p created
[must-gather ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... created
[must-gather-2k9pv] OUT gather logs unavailable: ...
[must-gather ] OUT namespace/openshift-must-gather-x7k2p deleted
Reprinting Cluster State:
...
$ tar cvaf mg-2026-09-08-auth.tar.gz mg-2026-09-08-auth/Notice that must-gather prints the cluster's own summary of what is wrong at the top; read it before uploading. The default image covers the core platform. Operators that ship their own must-gather image are collected by adding --image: OpenShift Data Foundation (--image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.16, matching your ODF version), the Logging operator, Virtualization, and others; the exact image tag is in each operator's documentation and tracks its version. For networking, oc adm must-gather -- gather_network_logs runs the network-specific script inside the default image. Newer oc versions accept --since to limit log volume; check oc adm must-gather --help on your client.
Three narrower tools are faster when you already know where the problem is. oc adm inspect ns/openshift-ingress --dest-dir=./inspect-ingress collects one namespace's objects, pods and logs. oc adm node-logs worker-2.ocp-prod.example.internal -u kubelet --tail=500 pulls a node's journal for a unit through the API, no debug pod needed, and --role=master -u crio does it for a whole role. For a full OS-level diagnostic of one node, run an sosreport: from oc debug node, chroot /host, start toolbox (a support container with the RHEL tooling), and run sos report -k crio.all=on -k crio.logs=on; the archive lands under /host/var/tmp and you copy it out with oc cp or oc debug ... -- cat.
Opening the case: Red Hat severities run from Severity 1 (production down or a critical business impact with no workaround; with a Premium subscription this gets 24x7 engagement with a one-hour initial response target) through Severity 2 (production severely degraded), Severity 3 (issue with a workaround, non-urgent) and Severity 4 (questions, cosmetic). Attach the must-gather at case creation; a case without one gets a first reply asking for it, which costs you hours. Before opening, search the Customer Portal Knowledgebase for the exact Reason or error string, and check Insights (oc get co insights confirms it is reporting; the console at console.redhat.com shows recommendations), because a large share of platform problems already have a solution article. Escalate when the problem is a platform fault you cannot mitigate yourself, when it involves a supported component's internals (OVN, etcd, MCO), or when a regulated process requires vendor confirmation. While the case is open you keep working the four layers, because the case exists to get help, not to hand off responsibility.
Ten worked scenarios
Each of these follows the same shape: symptom, method with the commands and what you expect to see, root cause, fix, prevention. Practise saying them aloud; the interviewer will hand you one of these or a close cousin.
(a) Upgrade stuck at 85%: the worker pool cannot drain a node
Symptom. The 4.16.20 to 4.16.22 upgrade has shown "85% complete, waiting on machine-config" for three hours. Method. oc adm upgrade confirms the CVO is waiting on the MCO. oc get mcp shows worker UPDATING True with UPDATEDMACHINECOUNT 2/4 and, after an hour, DEGRADED True. oc get nodes shows worker-1 Ready,SchedulingDisabled. oc describe mcp worker reports "failed to drain node after 1 hour". The machine-config-controller logs show the same eviction retried every five seconds: Cannot evict pod as it would violate the pod's disruption budget for payments-prod/payments-ledger-0. oc get pdb -n payments-prod shows payments-ledger with MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0, and the StatefulSet has exactly one replica. Root cause. A PDB requiring one available replica on a one-replica workload makes that pod permanently unevictable. Fix. With the app team on the bridge, either scale the StatefulSet to two (if the app supports it) so a disruption is allowed, or temporarily patch the PDB's minAvailable to 0 under an emergency change, let the drain proceed, and restore it. Never delete the PDB silently. Prevention. An admission policy (Gatekeeper or a ValidatingAdmissionPolicy) that rejects PDBs whose minAvailable equals the workload's replica count, a pre-upgrade health check script that lists such PDBs (see Post 32), and a golden-path template that ships a sane PDB by default.
(b) authentication operator Degraded after an ingress certificate replacement
Symptom. Users cannot log in to the console; oc login fails with a certificate error. Method. oc get co shows authentication Degraded, ingress recently Progressing. oc describe co authentication gives OAuthServerRouteEndpointAccessibleControllerDegraded ... x509: certificate signed by unknown authority. The change calendar shows the default ingress certificate was replaced two hours ago with a corporate CA-signed wildcard. openssl s_client -connect oauth-openshift.apps.ocp-prod.example.internal:443 -servername oauth-openshift.apps.ocp-prod.example.internal -showcerts shows a single certificate in the chain, no intermediate. Root cause. The new certificate was installed without its intermediate CA in the bundle, and the cluster's trust bundle was never told about the corporate CA, so the authentication operator's own health check of the OAuth route cannot validate it. Fix. Rebuild the TLS secret in openshift-ingress with the full chain (server certificate followed by intermediates), and add the corporate CA to a ConfigMap in openshift-config referenced by proxy/cluster's spec.trustedCA, which is the documented step people skip. The operator recovers within minutes. Prevention. A runbook for certificate replacement with a pre-check that validates the chain with openssl, and a certificate-expiry alert so this is never done in a hurry.
(c) Nodes NotReady after the cluster was shut down for two weeks
Symptom. A non-production cluster powered off for a data-centre migration comes back with every worker NotReady. Method. oc get nodes: masters Ready, workers NotReady. oc get csr: a list of Pending CSRs from node-bootstrapper. oc adm node-logs worker-0 -u kubelet --tail=50 (or the debug shell) shows x509: certificate has expired or is not yet valid when the kubelet tries to talk to the API. Root cause. Kubelet client certificates are short-lived and rotate automatically while the cluster runs; the cluster was off across a rotation window, so the kubelets came back with expired certificates and requested new ones, and the automatic approver does not approve renewals in this state. Fix. Approve the client CSRs, wait for the serving CSRs to appear, approve those too, and watch the nodes turn Ready.
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/csr-2k7vn approved
certificatesigningrequest.certificates.k8s.io/csr-9pw4x approved
$ sleep 60; oc get csr | grep Pending
csr-hj7f2 1m kubernetes.io/kubelet-serving system:node:worker-0.ocp-prod.example.internal <none> Pending
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
$ oc get nodes | grep -c ' Ready'
7Prevention. Follow the documented graceful shutdown and restart procedure, which warns about this; for long shutdowns, take an etcd backup first and plan the CSR approval into the restart runbook.
(d) Route returns 503 while curl inside the pod works
Symptom. "The API is down" from the payments team; the router shows the "Application is not available" page. Method. The Layer 3 walk above: router pods healthy, Route correct, Service correct, endpoints empty, pods Running but 0/1. oc rsh and curl to localhost returns 200, so the app is fine. The readiness probe path (/health) does not match the app's health endpoint (/actuator/health). Root cause. A new image version moved the health endpoint and the Deployment's probe was not updated. Fix. oc rollout undo to restore service immediately, then a proper change that updates the probe with the new image. Prevention. Health endpoint paths owned in the Helm chart alongside the image tag, and a CI check that hits the probe path against the built image before deployment (see Post 30). If endpoints had been empty with pods Ready, the twin root cause would be a Service selector that no longer matches the pod labels after a chart refactor.
(e) CrashLoopBackOff only on OpenShift; the same image runs fine in Docker
Symptom. A vendor image crash-loops. The team says "it works on Docker Desktop". Method. oc logs --previous shows mkdir: cannot create directory '/var/lib/app/cache': Permission denied. oc get pod -o yaml | grep scc shows restricted-v2; oc rsh into a debug copy and id shows uid=1000680000 gid=0. oc debug deployment/x --as-root and ls -ld /var/lib/app shows the directory owned by root with mode 755. Root cause. The image assumes it runs as root; OpenShift runs it as an arbitrary UID in group 0, and the directory is not group-writable. Docker ran it as root, so nobody noticed. Fix. Rebuild the image with chgrp -R 0 /var/lib/app && chmod -R g=u /var/lib/app in the Dockerfile, and if the app binds port 80, change it to 8080 (the Service and Route point at the container port anyway). If the vendor will not rebuild, mount an emptyDir or PVC at the writable path as the shortest-term workaround, and treat an SCC exception as the last resort with a ticket and an expiry. Prevention. The image guidelines in the onboarding golden path (Post 26) and a CI scan that flags images with USER root or no USER at all.
(f) etcd fsync alerts and a slow API on vSphere
Symptom. etcdHighFsyncDurations firing; oc commands take five to ten seconds; Argo CD syncs time out. Method. oc get co shows etcd and kube-apiserver Available but the etcd operator's message mentions slow members. etcdctl endpoint status from an etcd pod shows the members healthy with one or more lagging on applied index and raft terms incrementing (leader elections). In Prometheus, histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) is well above the 10 ms guidance, peaking at the same times every night. From oc debug node/master-1, run Red Hat's disk benchmark container against the etcd directory to measure fsync latency directly, and check with the vSphere team what else shares that datastore. Root cause. The control-plane VMs sit on a datastore shared with a backup job and a database cluster; storage latency spikes during the nightly backup window, and etcd's fsync waits with it. Fix. Move the master VMs' disks to a dedicated, low-latency datastore (SSD-backed), or give etcd its own disk via the supported MachineConfig procedure so it stops competing with the OS and container storage on the same device. Prevention. Run the etcd disk benchmark as part of cluster build validation, alert on the p99 fsync metric with a threshold below the failure point, and reserve control-plane VMs on the hypervisor so they never share IOPS with batch jobs.
(g) Pods stuck ContainerCreating with a Multi-Attach error after a node failure
Symptom. A hypervisor host died; its worker is NotReady; a StatefulSet pod rescheduled to another node has been ContainerCreating for twenty minutes. Method. oc describe pod Events: Warning FailedAttachVolume ... Multi-Attach error for volume "pvc-8c3f..." Volume is already used by pod(s) payments-ledger-0 or Volume is already exclusively attached to one node and can't be attached to another. oc get pods -n payments-prod shows the old pod stuck Terminating on the dead node. oc get volumeattachment | grep pvc-8c3f shows the attachment still bound to the dead node. Root cause. The volume is ReadWriteOnce (one node at a time). The dead node's kubelet cannot confirm the old pod stopped or detach the volume, so the CSI controller refuses to attach it elsewhere. Fix. Confirm the node is really dead (not partitioned and still writing), then delete the Machine (or the Node object if it is not Machine-managed) so the controllers know the node is gone; if the old pod is still Terminating, force-delete it. The attachment clears within a few minutes and the new pod starts. Prevention. A MachineHealthCheck that replaces NotReady nodes automatically after a timeout, RWX storage for workloads that genuinely need to fail over quickly, and an app-level replica for anything a bank calls critical, so a single pod's disk is never the recovery path.
(h) Image pulls failing across the cluster after a pull secret or registry certificate change
Symptom. New pods everywhere go ImagePullBackOff; existing pods are fine. Method. oc get pods -A | grep -v -E 'Running|Completed' shows the blast radius. oc describe pod Events give the registry's error, and there are two versions. Version one: unauthorized: Please login to the Red Hat Registry or no basic auth credentials right after someone updated the global pull secret; oc get secret pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq . shows the new JSON is missing the registry.redhat.io entry, or is not valid JSON at all, and oc get mcp shows the pools updating as the MCO rolls the broken file to every node. Version two: x509: certificate signed by unknown authority after the corporate registry rotated its certificate to a new CA. Root cause. The global pull secret was overwritten instead of merged, or the registry's new CA is not in the cluster's image.config.openshift.io/cluster additional trust bundle. Fix. For the pull secret, rebuild it by merging the original Red Hat credentials with the corporate registry's and re-apply: oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=pull-secret.json; the MCO distributes it to nodes. For the CA, update the ConfigMap in openshift-config referenced by spec.additionalTrustedCA, with a key named after the registry host (a port is written with two dots, as in registry.example.internal..5000). Prevention. Both files owned in Git and applied through a pipeline with a JSON lint and an oc image info pull test against every configured registry before the change is promoted to production.
(i) DNS for external hosts fails intermittently
Symptom. A market-data client sees roughly one in twenty lookups of a vendor hostname time out; internal names are fine. Method. oc get pods -n openshift-dns all healthy. Repeated nslookup from a debug pod reproduces the failure rate, and it is worse on two specific nodes. oc get dns.operator/default -o yaml shows a single upstream resolver with no fallback. On the bad nodes, oc debug node and journalctl -k | grep conntrack show nf_conntrack: table full, dropping packet, and sysctl net.netfilter.nf_conntrack_count is at the maximum. Prometheus confirms it with node_nf_conntrack_entries against node_nf_conntrack_entries_limit. Root cause. Two causes stacked: the nodes' connection-tracking table is full because a chatty app opens thousands of short-lived UDP flows, so DNS packets are dropped on those nodes, and with only one upstream and ndots:5 multiplying every external query, any drop becomes a visible timeout. Fix. Raise nf_conntrack_max through the Node Tuning Operator (a Tuned profile targeting the worker pool) rather than a hand-edited sysctl, add the second corporate resolver to spec.upstreamResolvers, and have the app use fully-qualified names with a trailing dot to skip the search-domain expansion. Prevention. An alert on conntrack utilization above 80%, two upstream resolvers from day one, and CoreDNS metrics (coredns_forward_requests_total, response codes) on the platform dashboard from Post 24.
(j) Prometheus OOMKilled and alerts silently stop
Symptom. Nobody has received an alert in six hours, which is itself suspicious. Method. The five commands show prometheus-k8s-0 and -1 in CrashLoopBackOff in openshift-monitoring. oc describe pod shows Last State: Terminated, Reason: OOMKilled for the prometheus container. oc get co monitoring is Degraded. Between crashes, the TSDB status page (or prometheus_tsdb_head_series in a brief query) shows series count tripled since yesterday; topk(10, count by (__name__)({__name__=~".+"})) names one metric from a newly onboarded app with a request-ID label. Root cause. A team enabled user-workload monitoring on a service exposing a high-cardinality label (unique per request), Prometheus' memory grew past its limit, and with Prometheus down no rules were evaluated, so no alerts fired, including the ones that would have told you Prometheus was down. Fix. Drop the label with a metricRelabelings rule on that ServiceMonitor (or delete the ServiceMonitor during the incident), then let Prometheus recover; raise Prometheus' resources in cluster-monitoring-config only if the legitimate series count justifies it. Prevention. Set enforcedSampleLimit and label limits for user-workload monitoring in the monitoring config, review ServiceMonitors in onboarding, and, most important, watch the Watchdog alert: it always fires by design, and an external dead-man's switch (PagerDuty or the enterprise monitoring tool expecting it every few minutes) is how you learn the alerting pipeline is down when it can no longer tell you itself.
Incident response at a bank
Technical skill gets service back. Process is what keeps a regulated institution safe while you do it, and the interviewer will listen for whether you know the process or resent it. The vocabulary below is ITIL, the service-management framework nearly every bank runs on, and the tooling is typically ServiceNow.
An incident = an unplanned interruption or degradation of a service. A problem = the underlying cause of one or more incidents; it is a separate record that outlives the incident. A change = any addition, modification or removal that could affect a service, recorded as an RFC (request for change) and approved before it happens. A known error = a problem with a documented root cause and workaround, stored in the known-error database so the next incident is resolved in minutes. Keeping those four words straight in the interview signals you have worked somewhere with a change calendar.
Severity levels
Definitions vary by bank, but the shape is always the same: severity is set by business impact, not by how technically interesting the failure is.
| Priority | Example definition | Response | Who is on the bridge | Update cadence |
|---|---|---|---|---|
| P1 (Critical) | Customer-facing or regulatory service down, or material financial or reputational impact; no workaround | Engage within 15 minutes, 24x7; bridge opened immediately | Incident commander, platform on-call, app owners, network, security, an executive sponsor | Every 30 minutes to stakeholders, continuous on the bridge |
| P2 (High) | Major degradation, one line of business affected, or a P1 with a working but painful workaround | Engage within 30 minutes, 24x7; bridge opened | Incident commander, on-call teams involved | Hourly |
| P3 (Medium) | Minor degradation, workaround available, limited users | Business hours, response within 4 hours | Assigned engineer, ticket-driven | Daily or on state change |
| P4 (Low) | Cosmetic or single-user issue, no business impact | Next business day or two | Assigned engineer | On resolution |
Roles, the bridge and communications
The incident commander (at many banks the Major Incident Manager) owns the incident, not the fix. They decide priorities, assign work, keep the timeline, decide when to escalate to the vendor or to executives, and are the only person who declares the incident resolved. The technical lead (often you, for a platform incident) drives diagnosis and proposes actions. A scribe records every action with a timestamp. A communications lead writes stakeholder updates so engineers are not interrupted. The bridge (war room) is the conference call and chat channel where all of it happens; nothing is done to production that was not announced on the bridge first.
Stakeholder updates follow a fixed template so anxious executives get the same shape every time: current impact, what has been done, what is being done now, and the time of the next update. "We are at 40% understanding and testing a rollback; next update at 14:30" is a good update. "Working on it" is not.
Mitigation before root cause
The single most important discipline: restore service first, understand it later. During a P1, the question is not "why did this break" but "what is the fastest safe action that ends the customer impact": roll back the deployment, fail traffic to the other cluster or region, scale up, cordon the bad node, revert the change from an hour ago. You capture evidence so the root cause is not lost when you do (a must-gather, oc get events -A to a file, the relevant pod logs, a copy of the objects before you change them), and then you act. Root cause is problem management's job, tomorrow, with a clear head.
When you may change production
In a bank, production changes go through the CAB (Change Advisory Board) with lead time, a rollback plan and a test record. During an incident, you use an emergency change: a ticket raised on the bridge, approved by the incident commander and the emergency CAB (often the change manager plus the service owner) in minutes, executed with a second person watching (four-eyes), through privileged access that is recorded (a PAM tool such as CyberArk that checks out the cluster-admin credential and logs the session, on top of OpenShift's own audit log). Change freezes around quarter-end and major releases still apply, and an emergency change during a freeze needs a higher approver. All of that sounds slow and is not: an experienced team completes it in five minutes, and the record it leaves is what lets the bank prove to a regulator what happened.
Closing and handing off
The incident closes when the commander confirms service is restored and monitored stable for an agreed period. The record is handed to problem management with the evidence, timeline and a preliminary cause. If the root cause is understood and a workaround exists but the permanent fix is pending, it becomes a known error. If not, a problem record stays open until the RCA is complete. Post-incident, the platform team owns the RCA for platform-caused incidents and contributes to app-caused ones.
Root cause analysis
An RCA (also called a postmortem or PIR, post-incident review) is a document, usually due within five business days of a P1 or P2, written for readers who were not on the bridge. Its purpose is to make the same incident impossible or harmless next time, not to identify who to blame. The structure most teams converge on:
- Summary: two sentences, what broke and for how long.
- Impact: services, customers, transactions, duration, any regulatory reporting triggered.
- Timeline: timestamped, from the change or trigger, through detection, each action, to resolution. Use one time zone and say which.
- Detection: how you found out (an alert, a customer, a user), and how long after the trigger.
- Root cause: the technical condition that, had it been different, would have prevented the incident.
- Contributing factors: things that made it worse or slower to fix.
- What went well: it matters for morale and for keeping what works.
- Action items: each with an owner and a due date, tracked in the ticketing system, reviewed at the next ops meeting.
- Lessons: what the organization now knows that it did not.
The 5 Whys is the simplest tool for getting from symptom to root cause: ask "why" of each answer until you reach a condition you can change. Blameless means the document names systems and decisions, not people, and assumes everyone acted reasonably with the information they had. "Human error" is never a root cause; it is a symptom of a system that let a reasonable person make an unsafe change without a guardrail.
A worked RCA for scenario (a)
Summary. The production OpenShift upgrade from 4.16.20 to 4.16.22 stalled for 3 h 10 min because one worker node could not be drained; no customer impact, but the maintenance window was exceeded and the upgrade completed outside the approved change window.
Impact. No service impact. One worker was cordoned for 3 hours, reducing spare capacity by 25%; the change record was breached, requiring a post-approval by the change manager.
Timeline (all times ET). 22:00 upgrade started under CHG0047112. 22:41 CVO reached "waiting on machine-config". 22:52 worker-1 cordoned by the MCO. 23:52 worker MCP reported Degraded (drain timeout). 00:05 on-call paged by the ClusterOperatorDegraded alert for machine-config. 00:12 on-call ran the five commands, identified the drain and the PDB. 00:20 app team paged; 00:41 emergency change approved on the bridge to set minAvailable: 0 on payments-ledger PDB. 00:43 drain completed. 01:10 upgrade complete; PDB restored to minAvailable: 1 under the same change.
Detection. Alert, 73 minutes after the drain first failed; the drain retry period is one hour before the MCO marks the pool Degraded, and the on-call was not watching the upgrade dashboard.
Root cause (5 Whys). Why did the upgrade stall? The MCO could not drain worker-1. Why? The PDB for payments-ledger allowed zero disruptions. Why? minAvailable: 1 on a StatefulSet with one replica. Why? The team's Helm chart defaulted to a PDB, and the app cannot run more than one replica. Why was that allowed into production? Nothing validates PDBs against replica counts at admission or in the pre-upgrade check.
Contributing factors. No pre-upgrade validation script; the alert fires only after the one-hour drain timeout; the app team's on-call rota was not on the change's notification list.
What went well. The five-command triage located the cause in seven minutes; the emergency change process took twenty-one minutes end to end; the PDB was restored, not deleted.
Action items. (1) Pre-upgrade check script listing PDBs with zero allowed disruptions, platform team, two weeks (see Post 32). (2) Admission policy rejecting PDBs that permit no disruptions, platform team, six weeks. (3) Golden-path chart: PDB only rendered when replicas are two or more, platform team, four weeks (Post 26). (4) Add app on-call groups to upgrade change notifications, change manager, one week. (5) Custom alert on machine_config_pool_updating duration over 45 minutes, monitoring owner, two weeks.
Notice how the action items are where the value lives. Items 1 and 2 become automation, item 3 becomes a platform standard, and item 5 improves detection. A postmortem with no action items, or with actions that have no owner and no date, is a story, not an RCA.
minAvailable: 1, then run oc adm drain on its node with --timeout=60s and watch it fail. Fix it two ways (scale to two, then patch the PDB) and note the time each takes. Then write a one-page postmortem using the nine headings above, with at least three action items that have owners and dates. Reading a postmortem is easy; writing one that would survive a change manager's review is the skill.Building the runbook library
A runbook = a step-by-step procedure for a specific alert or symptom that an on-call engineer who has never seen the problem can follow at 3 a.m. Good ones share a structure: the alert or symptom it covers and its severity; the business impact in one sentence; a pre-check that confirms this is the right runbook (the exact command and expected output); the diagnosis steps in order, each with the command, what good looks like and what bad looks like; the remediation, including the emergency-change requirement; verification that service is restored; rollback if the remediation makes it worse; escalation path with the vendor case template; the owner and the date it was last tested. Each of the ten scenarios above is a runbook waiting to be written.
Runbooks earn their keep when the alert links to them. In a PrometheusRule, the runbook_url annotation carries the link, and Alertmanager passes it through to the pager, so the engineer paged for etcdHighFsyncDurations opens the etcd storage runbook with one click. OpenShift's built-in alerts already carry links to Red Hat's runbooks repository; you add your own for platform-specific and app-specific alerts.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-upgrade-rules
namespace: openshift-monitoring
spec:
groups:
- name: platform.upgrade
rules:
- alert: MachineConfigPoolUpdatingTooLong
expr: max by (pool) (mcp_updating_seconds) > 2700
for: 5m
labels:
severity: warning
team: platform
annotations:
summary: "MCP {{ $labels.pool }} has been updating for over 45 minutes"
runbook_url: "https://git.example.internal/platform/runbooks/blob/main/mco/stuck-drain.md"The metric name here is illustrative; the point is the annotation. Around the runbooks sit three companions. Health-check scripts (Post 32) encode the five commands and the pre-upgrade checks, so they run the same way every time and produce an artifact you can attach to a change record. The known-errors database holds every diagnosed problem with its symptom, root cause and workaround, searchable by the error string, so scenario (h) is a two-minute lookup the second time. And all of it lives in Git: runbooks as Markdown in a repository with pull-request review, an owner per directory, a last-tested date, and a quarterly game day where the on-call rota runs a runbook against a lab cluster to prove it still works. A runbook nobody has run in a year is a rumour.
How to present all this in the interview
You will get "walk me through how you troubleshoot an OpenShift issue" in some form. Here is a 60-second answer to rehearse until it is smooth.
"I use the same ordered method every time: cluster health, then node health, then the network path, then the workload, and I read Events before I guess. Concretely, the first minute is five commands: oc get clusterversion to know whether an upgrade is in progress, oc get co to see which operator is Degraded and read its message, oc get nodes for NotReady or cordoned nodes, oc get mcp to see whether the MCO is mid-rollout or stuck, and a filtered oc get pods -A for every unhappy pod in the cluster. That tells me which layer I am in. If it is cluster, I describe the operator and check etcd and certificates. If it is node, oc describe node for conditions and oc debug node for kubelet, CRI-O, disk and time. If it is network, I walk router to Route to Service to endpoints to readiness and test from inside a pod. If it is workload, describe, previous logs, SCC and quota events. In parallel I capture a must-gather and, if service is impacted, I mitigate first through an emergency change and root-cause afterwards in a blameless postmortem with owned action items."
When you do not know the answer, say so and then show the method: "I have not seen that exact failure. Here is how I would find out: the operator's condition message and Reason string first, then the Knowledgebase for that string, then the operator's pods and logs in its related namespace, and a Sev 2 case with a must-gather if it is not resolved in thirty minutes." That answer scores higher than a confident wrong guess, because it is what the interviewer would want you to do on their cluster.
Likely interview questions
A team says their application is down. What is the very first thing you do?
State the method and run the five commands before I ask them anything, because their description of the symptom is the least reliable data I have. Cluster version, cluster operators, nodes, machine config pools, and the filtered pod list across all namespaces tell me within a minute whether this is a platform problem affecting everyone or a problem confined to their namespace, and that decides everything that follows.
A cluster operator is Degraded. How do you find out why?
oc describe co <name> and read the Degraded condition's Message and Reason; the Reason names the specific controller that is failing and is the string I search in the Red Hat Knowledgebase. Then I look at the Related Objects list, go to that namespace, and check its pods and logs. Degraded with Available True is usually a configuration the operator cannot reconcile; Available False is an outage of that component.
What is the difference between how you investigate a node on OpenShift and on a kubeadm cluster?
On OpenShift I do not SSH in and edit files, because RHCOS is managed by the Machine Config Operator, which owns the OS configuration and will revert or flag drift. I use oc debug node with chroot /host for read-only investigation (systemctl, journalctl, crictl, df, chronyc), and I make changes through a MachineConfig, KubeletConfig or Tuned profile committed to Git. If a node is unrecoverable and Machine-managed, I delete the Machine and let the MachineSet replace it.
A Route returns 503 but the pod is Running. Walk me through it.
A 503 from the router means it has no healthy endpoint. I check the router pods, the Route's service and port, the Service, and then the endpoints; if endpoints are empty and the pod is Running but 0/1 Ready, it is the readiness probe, and if the pod is 1/1 Ready, it is a selector mismatch. I confirm the app itself by oc rsh and curl to localhost. The fix is the probe or the labels, and the mitigation is usually a rollout undo.
An image runs in Docker but crash-loops on OpenShift with permission denied. Why, and what do you do?
OpenShift's default restricted-v2 SCC runs containers as an arbitrary non-root UID in group 0, so an image that assumes root cannot write to root-owned directories or bind privileged ports. I confirm with oc logs --previous, the openshift.io/scc annotation and oc debug --as-root, then have the image fixed: group-zero ownership with chmod g=u on writable paths and a port above 1024. An SCC exception on the service account is the last resort and needs a ticket, not the first move.
The API server is slow. Where do you look?
etcd. I check the etcd cluster operator and pods, run etcdctl endpoint status and endpoint health from an etcd pod to see leader stability, raft lag and commit latency, and look at the fsync and commit duration alerts. Slow etcd is nearly always slow disk under the control plane, so the fix is storage, not restarts. If etcd is healthy, I look for a client hammering the API, in the apiserver metrics and audit log.
What do you send Red Hat when you open a case, and when do you open it?
A must-gather from oc adm must-gather, plus an operator-specific one if ODF, logging or networking is involved, and an sosreport from the affected node if the problem is OS-level. I open the case as soon as I suspect a platform fault I cannot mitigate, at a severity matching business impact, and I keep troubleshooting and mitigating in parallel rather than waiting.
Explain incident, problem and change in your own words.
An incident is the outage or degradation happening now, and its goal is restoring service. A problem is the underlying cause, tracked separately until it is fixed or documented as a known error with a workaround. A change is any modification to production, approved through the CAB in advance or, during an incident, through an emergency change approved on the bridge. Incidents restore, problems prevent, changes control.
During a P1, you know the fix. What happens before you type it?
I say it on the bridge, the incident commander approves an emergency change, the ticket is linked to the incident, I execute through the privileged-access tool with a second engineer watching, and I capture evidence first so the root cause is not lost. Then I verify service is restored and tell the commander. It takes minutes and it is the only way a bank can prove to a regulator what happened.
What makes a postmortem useful rather than a formality?
A timestamped timeline, a root cause found by asking why until you reach a condition the platform can change, contributing factors including detection time, and action items with owners and due dates that are tracked to completion. Blameless, because "human error" is a symptom of a missing guardrail. The best action items become automation and platform standards so the same incident cannot recur.
Key Takeaways
- State the method before the answer: cluster health, then node health, then the network path, then the workload, and always Events before guesses. The interviewer grades the method.
- The first five commands (
oc get clusterversion,oc get co,oc get nodes,oc get mcp, filteredoc get pods -A) tell you which layer you are in within a minute and before anyone describes the symptom to you. - Cluster layer:
oc describe cofor the Reason and Message,oc adm upgradefor the CVO,etcdctl endpoint statusfor a slow API,oc get csrfor certificate trouble. - Node layer:
oc describe nodeconditions, thenoc debug nodeandchroot /hostfor kubelet, CRI-O, disk, time and network; change nodes through MachineConfig, and replace Machines rather than repair them. - Network layer: walk router to Route to Service to endpoints to readiness, test from inside a pod, and remember that NetworkPolicy denies silently and a 503 means no endpoints.
- Workload layer: describe,
--previouslogs, the SCC and quota messages on the ReplicaSet's events,oc debug --as-rootfor permission problems, andoc rollout undoas the fastest mitigation. - Restore service first under an emergency change approved on the bridge, capture a must-gather and a timeline, then write a blameless RCA whose action items become automation, standards and better alerts.
- Runbooks linked from alerts, health-check scripts, a known-errors database and a Git repository with quarterly game days are what turn one engineer's experience into a platform's resilience.