Cluster Lifecycle: Nodes, MachineConfig, Upgrades and etcd

Two bullets on the job description decide whether you're a platform engineer or someone who deploys apps onto a platform: "perform cluster upgrades, patching, lifecycle management and platform maintenance" and "manage MachineConfig". Every interviewer for this role will ask some version of "walk me through upgrading production" and "how do you back up and restore etcd", and they'll listen for whether you've done it under a change ticket with a rollback plan or only read about it. After this post you'll be able to describe, in order and with the exact commands, how you'd upgrade a production OpenShift cluster at a bank, how you'd change the operating system on 200 nodes without ever typing ssh, and what you'd do at 3 a.m. when etcd has lost quorum.

At a bank, an upgrade is a change record, not a command

In a startup, "upgrade the cluster" means someone runs a command on a Friday afternoon. In a bank it means a change record that names the version, the window, the pre-checks, the backout plan and the person who approves each gate; a CAB (Change Advisory Board) that reviewed it; a lower environment that already went through the exact same procedure; and evidence afterwards that the platform is healthy, because an auditor may ask for it a year later. The technology is the same oc adm upgrade. What the bank buys with all that ceremony is predictability: no surprise reboots, no untested versions, no "we'll figure out rollback if we need it".

The second theme of this post is the Machine Config Operator (MCO), which is how OpenShift touches the operating system on every node. On a plain Kubernetes cluster, changing a kernel parameter or an NTP server on the workers means Ansible over SSH, or a golden image and a node replacement. OpenShift nodes run RHCOS (Red Hat Enterprise Linux CoreOS), an immutable, image-based OS that is meant to be configured only through the Kubernetes API. You describe the desired OS state in a MachineConfig object, and the MCO rolls it out node by node, draining and rebooting as it goes. No SSH, no configuration drift, a full audit trail in the API server's audit log, and the same mechanism carries OS updates during a cluster upgrade. That's why the JD lists MachineConfig next to upgrades: they are one system.

We'll build up in layers: how nodes come to exist (the Machine API), how you take one out of service safely (cordon, drain, debug, CSRs), how the OS is configured (MCO), how the whole cluster moves versions (CVO and the update graph), how the state store is protected (etcd), and finally the operational routines a bank actually runs: patch cadence, certificate hygiene, graceful shutdown and fleet management with RHACM. Post 19 introduced most of these components by name; here you learn to operate them.

The Machine API: nodes as declared objects

In Kubernetes, a Node object appears when a kubelet registers; how the machine underneath got created is somebody else's problem. OpenShift makes that somebody the cluster itself. The Machine API (API group machine.openshift.io, everything lives in the openshift-machine-api namespace) is a set of controllers that create and delete virtual machines or bare-metal servers on your infrastructure provider, then boot them into the cluster with an Ignition config served by the MCO. Three objects matter:

• Machine = one node's infrastructure record: which instance it maps to, its provider-specific spec (instance type, subnet, template), and a phase (Provisioning → Provisioned → Running, or Failed/Deleting).
• MachineSet = a template plus a replica count, exactly like a ReplicaSet for Machines. Scale it and the controller creates or deletes Machines, which become Nodes.
• MachineHealthCheck = a rule that says "if a node in this set has been NotReady for longer than X, delete its Machine and let the MachineSet replace it". This is node-level self-healing.

Here's what a production cluster looks like. This one is on vSphere, which is common for Canadian banks; on a cloud provider the TYPE, REGION and ZONE columns show the instance type and availability zone.

$ oc get machines -n openshift-machine-api
NAME                           PHASE     TYPE   REGION   ZONE   AGE
prod-7k2xd-infra-0-4zt8w       Running                          390d
prod-7k2xd-infra-0-mq7cn       Running                          390d
prod-7k2xd-infra-0-zz81p       Running                          390d
prod-7k2xd-master-0            Running                          412d
prod-7k2xd-master-1            Running                          412d
prod-7k2xd-master-2            Running                          412d
prod-7k2xd-worker-0-2fhwr      Running                          412d
prod-7k2xd-worker-0-l9ktz      Running                          412d
prod-7k2xd-worker-0-p4mnd      Running                          88d
prod-7k2xd-worker-0-tk6vc      Provisioning                     41s

$ oc get machinesets -n openshift-machine-api
NAME                    DESIRED   CURRENT   READY   AVAILABLE   AGE
prod-7k2xd-infra-0      3         3         3       3           390d
prod-7k2xd-worker-0     9         9         8       8           412d

prod-7k2xd is the infra ID, a random suffix the installer generates so every cloud resource belonging to this cluster is taggable and findable. The last machine is 41 seconds old and still Provisioning because someone just scaled the worker MachineSet from 8 to 9. That someone ran:

$ oc scale machineset prod-7k2xd-worker-0 --replicas=9 -n openshift-machine-api
machineset.machine.openshift.io/prod-7k2xd-worker-0 scaled

$ oc get machines -n openshift-machine-api -w
NAME                        PHASE          TYPE   REGION   ZONE   AGE
prod-7k2xd-worker-0-tk6vc   Provisioning                          41s
prod-7k2xd-worker-0-tk6vc   Provisioned                           3m12s
prod-7k2xd-worker-0-tk6vc   Running                               7m48s

Provisioning is the provider creating the VM. Provisioned means the VM exists and has booted with the worker Ignition config. Running means a Node object with the same name has joined and is Ready. In between, the new kubelet requested certificates via CSRs, which the cluster-machine-approver auto-approved because it could match the request to a Machine it knows about. Keep that detail; it's the key to a classic failure in the next section. There's also a cluster autoscaler (ClusterAutoscaler plus per-MachineSet MachineAutoscaler objects) that scales MachineSets for you, but banks usually keep capacity static and change it through a ticket, so manual oc scale is what you'll actually do.

Infra nodes: a node pool with a label and a taint

An infra node is a worker reserved for platform components: the ingress routers, the internal image registry, monitoring (Prometheus, Alertmanager), logging (Loki) and, in bigger clusters, ODF storage. Red Hat does not count infra nodes against your OpenShift subscription as long as they run only those components, which is why every bank has them. Mechanically it's just a second MachineSet whose template adds a node label and a taint. The providerSpec block is copied from the existing worker MachineSet.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: prod-7k2xd-infra-0
  namespace: openshift-machine-api
  labels:
    machine.openshift.io/cluster-api-cluster: prod-7k2xd
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: prod-7k2xd
      machine.openshift.io/cluster-api-machineset: prod-7k2xd-infra-0
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: prod-7k2xd
        machine.openshift.io/cluster-api-machine-role: infra
        machine.openshift.io/cluster-api-machine-type: infra
        machine.openshift.io/cluster-api-machineset: prod-7k2xd-infra-0
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      taints:
        - key: node-role.kubernetes.io/infra
          value: reserved
          effect: NoSchedule
        - key: node-role.kubernetes.io/infra
          value: reserved
          effect: NoExecute
      providerSpec:
        value:
          # copied verbatim from the worker MachineSet: template, network,
          # numCPUs, memoryMiB, diskGiB, userDataSecret (worker-user-data)

Two things the interviewer wants to hear you say. First, these nodes still boot with the worker Ignition config (userDataSecret: worker-user-data), so they also carry the node-role.kubernetes.io/worker label; the infra label is added on top. Second, the taint alone doesn't move anything. You then have to point each platform component at the pool: spec.nodePlacement on the default IngressController, spec.nodeSelector on the image registry config, the cluster-monitoring-config ConfigMap for Prometheus and friends, each with a matching toleration. That's a day-2 procedure in the docs and a common item on a bank's build standard. We'll create a matching infra MachineConfigPool later, so OS config for infra nodes can differ from ordinary workers.

MachineHealthCheck

apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: worker-notready-remediation
  namespace: openshift-machine-api
spec:
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-machine-role: worker
      machine.openshift.io/cluster-api-machine-type: worker
  unhealthyConditions:
    - type: Ready
      status: Unknown
      timeout: 300s
    - type: Ready
      status: "False"
      timeout: 300s
  maxUnhealthy: 40%
  nodeStartupTimeout: 10m

Read it as a sentence: "for machines with the worker role, if the Node's Ready condition is False or Unknown for five minutes, delete the Machine, unless more than 40% of the pool is already unhealthy, in which case do nothing because this is probably a network partition and deleting machines would make it worse." maxUnhealthy is the circuit breaker; a bank sets it conservatively. MachineHealthChecks don't remediate control-plane machines unless the cluster has an active ControlPlaneMachineSet to rebuild them, and they only help when the MachineSet can actually replace the node; on user-provisioned bare metal with nothing to create, they're pointless.

Cloud vs bare metal

On AWS, Azure, GCP or vSphere the Machine API talks to the provider's API and the whole lifecycle is automatic. On bare metal with the installer-provisioned flow, the provider is Metal3/Ironic and the extra object is a BareMetalHost: a record of a physical server with its BMC (iDRAC, iLO) address and credentials. A Machine "consumes" a BareMetalHost, Ironic powers it on and writes the RHCOS image, and it joins like any other node.

$ oc get baremetalhosts -n openshift-machine-api
NAME        STATE                    CONSUMER                     ONLINE   ERROR   AGE
master-0    externally provisioned   prod-7k2xd-master-0          true             412d
worker-0    provisioned              prod-7k2xd-worker-0-2fhwr    true             412d
worker-1    provisioned              prod-7k2xd-worker-0-l9ktz    true             412d
worker-7    ready                                                 false            2d

worker-7 is racked, inspected and waiting: the next time you scale the MachineSet, it gets consumed. With user-provisioned infrastructure (UPI: you built the VMs yourself), there are no Machine objects at all, node adds are manual, and CSR approval is manual too. Newer releases are introducing upstream Cluster API (cluster.x-k8s.io) alongside the Machine API, but the Machine API is what you administer in the 4.14 to 4.19 clusters a bank runs today.

Node maintenance: cordon, drain, debug, approve

Everything else in this post (MachineConfig rollouts, upgrades, hardware maintenance) is built on one primitive: taking a node out of service without hurting the apps on it. You learned the Kubernetes version in Post 9; the OpenShift commands are the same with oc adm in front.

• Cordon = mark the node unschedulable. Nothing new lands; existing Pods keep running. oc adm cordon <node>.
• Drain = cordon plus evict every Pod through the Eviction API, so controllers reschedule them elsewhere and PodDisruptionBudgets are honoured. oc adm drain <node> --ignore-daemonsets --delete-emptydir-data --force.
• Uncordon = put it back. oc adm uncordon <node>.

$ oc adm cordon prod-7k2xd-worker-0-l9ktz
node/prod-7k2xd-worker-0-l9ktz cordoned

$ oc adm drain prod-7k2xd-worker-0-l9ktz --ignore-daemonsets --delete-emptydir-data --force
node/prod-7k2xd-worker-0-l9ktz already cordoned
WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-4lm7v, openshift-dns/dns-default-p5c2r, openshift-image-registry/node-ca-hjt9s, openshift-machine-config-operator/machine-config-daemon-x2ln8, openshift-monitoring/node-exporter-tmznb, openshift-multus/multus-6b2vr, openshift-ovn-kubernetes/ovnkube-node-7ktqn
evicting pod payments-prod/ledger-api-7d9f8c6b5-k2v9x
evicting pod payments-prod/ledger-api-7d9f8c6b5-qq4tz
evicting pod fraud-scoring/scorer-5c6d7b8a9-x1z2y
error when evicting pods/"ledger-api-7d9f8c6b5-qq4tz" -n "payments-prod" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/ledger-api-7d9f8c6b5-k2v9x evicted
pod/fraud-scoring/scorer-5c6d7b8a9-x1z2y evicted
error when evicting pods/"ledger-api-7d9f8c6b5-qq4tz" -n "payments-prod" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

The three flags each answer a question the drain would otherwise refuse to guess at. --ignore-daemonsets: DaemonSet Pods (the OVN node agent, the machine-config-daemon, node-exporter) can't be evicted because the DaemonSet would just recreate them on the same node, so skip them; without the flag the drain refuses to start. --delete-emptydir-data: Pods with an emptyDir volume lose that data on eviction, confirm you accept that. --force: Pods with no controller (a bare Pod someone created by hand) can't be rescheduled, delete them anyway.

The last two lines are the interesting part. The drain is retrying every five seconds because evicting ledger-api-…-qq4tz would violate its PodDisruptionBudget (PDB): the app team declared a minimum number of ready replicas, and evicting this one would go below it. The Eviction API says no, and drain politely waits. It will wait forever. Your options, in order of preference at a bank:

1. Look: oc get pdb -n payments-prod. If ALLOWED DISRUPTIONS is 0, find out why. Often the app has two replicas and minAvailable: 2, or it has three replicas but one is CrashLooping so only two are ready. The PDB is doing its job; the app is the problem.
2. Fix the cause with the app team: get the crashlooping replica healthy, or scale up so the budget has headroom, or correct a PDB that can never be satisfied (minAvailable: 100% is a real thing people write).
3. Only with the app owner's agreement and a note in the change record: bypass it. oc adm drain … --disable-eviction deletes Pods instead of evicting them, which ignores PDBs. This is the "break glass" option, not a habit.

This matters far beyond manual maintenance: the MCO uses the same eviction path when it rolls out a MachineConfig or an upgrade, so a stuck PDB stalls the whole pool. We'll see what that looks like.

Getting a shell on the node without SSH

RHCOS nodes do accept SSH as the core user if you injected a key at install, but the supported day-to-day way is oc debug node/<name>. It starts a privileged Pod on that node with the host filesystem mounted at /host; chroot /host makes the host's binaries and journal yours.

$ oc debug node/prod-7k2xd-worker-0-l9ktz
Temporary namespace openshift-debug-8x2ml is created for debugging node...
Starting pod/prod-7k2xd-worker-0-l9ktz-debug-4mzq2 ...
To use host binaries, run `chroot /host`
Pod IP: 10.20.31.14
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# systemctl status kubelet --no-pager | head -5
sh-5.1# journalctl -u kubelet --since "30 min ago" --no-pager | grep -iE "error|fail" | tail -20
sh-5.1# journalctl -u crio --since "30 min ago" --no-pager | tail -20
sh-5.1# crictl ps | grep -v Running
sh-5.1# crictl logs <container-id>
sh-5.1# rpm-ostree status
sh-5.1# df -h /var /sysroot; free -g; uptime

Those are the eight commands you'll run in ninety percent of node investigations: is the kubelet up, what is it complaining about, is CRI-O (the container runtime) complaining, which containers are dead, what does the OS think it's running, and is the disk full. For the kubelet journal alone there's a shortcut that needs no debug Pod: oc adm node-logs prod-7k2xd-worker-0-l9ktz -u kubelet --tail=200. Remember to exit twice; the debug Pod and its temporary namespace are removed for you. And note what you did not do: you did not edit anything on the host. We'll come back to why.

Certificate signing requests and the "NotReady forever" node

When a kubelet joins, it needs two certificates, and it asks for them through the Kubernetes CertificateSigningRequest (CSR) API. First a client certificate, requested by the bootstrap identity (node-bootstrapper) so the kubelet can authenticate to the API server. Once that's approved, the kubelet registers the Node and immediately requests a serving certificate, so the API server can talk back to it (for oc logs, oc exec, metrics). Two rounds, two CSRs, and the Node isn't fully usable until both are approved.

$ oc get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                                                                   REQUESTEDDURATION   CONDITION
csr-2lm7c   6m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
csr-9vq2k   2m    kubernetes.io/kubelet-serving                 system:node:prod-7k2xd-worker-0-tk6vc                                       <none>              Pending
csr-b8xh4   3h    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Approved,Issued

$ oc adm certificate approve csr-2lm7c
certificatesigningrequest.certificates.k8s.io/csr-2lm7c approved

$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' \
    | xargs --no-run-if-empty oc adm certificate approve

The one-liner approves everything Pending; it's straight from the Red Hat docs and worth memorising. On Machine API clusters the cluster-machine-approver does this automatically because it can prove the CSR belongs to a Machine it created. It will not approve a CSR it can't match: a UPI node you built by hand, a node whose Machine was deleted, or the second-round serving CSR of a node that changed its address. And after a long shutdown, kubelet certificates on every node may have expired, so on restart you get a wall of Pending CSRs and no Ready nodes until you approve them. Approve, wait a minute, run oc get csr again for the second round, approve again.

The Machine Config Operator: the OS as an API object

Now the centrepiece. The MCO is four pieces that live in the openshift-machine-config-operator namespace, and you should be able to name them:

• machine-config-operator: the parent, a ClusterOperator like any other (oc get co machine-config). It installs and watches the other three.
• machine-config-controller: merges MachineConfigs into a single rendered config per pool, decides which node updates next, and enforces maxUnavailable.
• machine-config-server (MCS): an HTTPS endpoint on the control plane (port 22623) that serves the Ignition config to a node booting for the first time. This is how a fresh VM knows what to become.
• machine-config-daemon (MCD): a DaemonSet, one Pod per node, running privileged. It compares what's on disk with the rendered config the node should have, applies the difference, and reboots the node when needed.

And the objects, all cluster-scoped, all in API group machineconfiguration.openshift.io:

• MachineConfig (MC) = a fragment of desired OS state for one role: files, systemd units, kernel arguments, the OS image, SSH keys, and extensions (extra RPM packages such as usbguard or kernel-devel). Its body is written in Ignition format, the same JSON schema RHCOS uses for first boot; on OpenShift you write spec.config.ignition.version: 3.2.0 (newer schema versions such as 3.4.0 are accepted too).
• MachineConfigPool (MCP) = a set of nodes (selected by node label) plus the set of MCs that apply to them (selected by the MC's role label). Every cluster has master and worker; you add custom pools such as infra.
• rendered MachineConfig = the merge of every MC that matches a pool, in name order, producing one object named rendered-<pool>-<hash>. Nodes never run "a MachineConfig"; they run a rendered config, and the hash is what you'll see in every status column.

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-2f9a7c1d4e0b3a8c6d5e4f3a2b1c0d9e    True      False      False      3              3                   3                     0                      390d
master   rendered-master-8c1b2a3d4e5f60718293a4b5c6d7e8f9   True      False      False      3              3                   3                     0                      412d
worker   rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d   False     True       False      9              8                   7                     0                      412d

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
00-worker                                          a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
01-master-container-runtime                        a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
01-master-kubelet                                  a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
01-worker-container-runtime                        a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
01-worker-kubelet                                  a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
99-master-generated-registries                     a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
99-master-ssh                                                                                 3.2.0             412d
99-worker-chrony                                                                              3.2.0             41m
99-worker-generated-kubelet                        a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             120d
99-worker-generated-registries                     a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             412d
99-worker-ssh                                                                                 3.2.0             412d
rendered-master-8c1b2a3d4e5f60718293a4b5c6d7e8f9   a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             12d
rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d   a1b2c3d4e5f60718293a4b5c6d7e8f9012345678   3.4.0             41m

Read the MC list like a layer cake. 00-* and 01-* are the base OS, CRI-O and kubelet configs that Red Hat ships; you never touch them. 99-*-generated-* are produced by the controller from higher-level CRs (a KubeletConfig, an ImageDigestMirrorSet). 99-*-ssh carries the SSH key from install time. And 99-worker-chrony, 41 minutes old with no GENERATEDBYCONTROLLER value, is one a human applied. Its arrival produced a new rendered-worker-…1c0d, and the worker pool is now UPDATING towards it: nine machines, seven already on the new rendered config, one being worked on, one still to go. The infra pool is separate and will get its own rendered config, because a custom pool inherits worker MCs plus its own.

How a change rolls out

When the rendered config for a pool changes, the controller picks up to maxUnavailable nodes (default 1) and sets the annotation machineconfiguration.openshift.io/desiredConfig on each to the new hash. The MCD on that node notices its currentConfig differs from desiredConfig and runs the sequence you must be able to recite: cordon → drain → write files and units, apply kernel args and OS image via rpm-ostree → reboot → validate on-disk state against the rendered config → set currentConfig = desiredConfig → uncordon. Then the controller picks the next node. Budget ten to fifteen minutes per node when everything is healthy, so a nine-node worker pool with maxUnavailable: 1 takes about two hours, and a 60-node pool takes a working day unless you raise maxUnavailable (a bank might allow 2 or 3 on the worker pool and never more than 1 on master, where it's fixed anyway).

Not every change reboots. The MCD knows a short list of paths and settings it can apply live (container registry mirrors, SSH keys, the kubelet CA bundle), and since 4.17 the MachineConfiguration CR has a nodeDisruptionPolicy section where you can declare that a particular file only needs a service reload rather than a reboot. Everything else reboots. Assume reboot unless you've verified otherwise.

Watching it happen is the skill. Three views, from coarse to fine:

$ oc get mcp worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d   False     True       False      9              8                   7                     0                      412d

$ oc describe mcp worker | sed -n '/^Status:/,/^Events:/p'
Status:
  Conditions:
    ...
    Message:               All nodes are updating to MachineConfig rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d
    Status:                True
    Type:                  Updating
    Status:                False
    Type:                  Degraded
    Status:                False
    Type:                  NodeDegraded
    Status:                False
    Type:                  RenderDegraded
  Degraded Machine Count:    0
  Machine Count:             9
  Ready Machine Count:       8
  Unavailable Machine Count: 1
  Updated Machine Count:     7

$ oc get nodes -o custom-columns='NAME:.metadata.name,STATE:.metadata.annotations.machineconfiguration\.openshift\.io/state,CURRENT:.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig,DESIRED:.metadata.annotations.machineconfiguration\.openshift\.io/desiredConfig' | grep worker
prod-7k2xd-worker-0-2fhwr   Done      rendered-worker-5a4b3c2d...   rendered-worker-5a4b3c2d...
prod-7k2xd-worker-0-l9ktz   Working   rendered-worker-9e8d7c6b...   rendered-worker-5a4b3c2d...
prod-7k2xd-worker-0-p4mnd   Done      rendered-worker-9e8d7c6b...   rendered-worker-9e8d7c6b...

The three columns UPDATED / UPDATING / DEGRADED are a state machine: a healthy idle pool is True/False/False; a rolling pool is False/True/False; a pool with a node that failed is False/True-or-False/True. The per-node view tells you which node is Working right now (its CURRENT and DESIRED differ) and which are still waiting (both columns show the old hash). Newer releases also expose a MachineConfigNode object per node with a finer-grained phase; check oc get machineconfignodes if your version has it. Beyond that, the source of truth is the MCD log on the working node, which we'll read in the troubleshooting section.

Pausing a pool

A MachineConfigPool has spec.paused. While it's true, the controller still renders new configs but never assigns them to nodes: no drains, no reboots. You use it for exactly one reason: to control when nodes reboot rather than whether they do.

$ oc patch mcp worker --type merge -p '{"spec":{"paused":true}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

$ oc get mcp worker -o jsonpath='{.spec.paused}{"\n"}'
true

$ oc patch mcp worker --type merge -p '{"spec":{"paused":false}}'

Why a bank pauses the worker pools during a control-plane upgrade: the control plane can be upgraded in a daytime window with zero application disruption (apps don't run on masters), while the worker reboots, which do disrupt applications, are held until the approved Saturday night window and can be released pool by pool: infra first, then the workers that host non-critical apps, then the payments workers with the app team on the bridge. It also protects you from an accident: with the worker pool paused, a colleague applying a MachineConfig at 2 p.m. doesn't start rebooting production. The cost is that a paused pool receives nothing, including a new kubelet CA bundle. Recent versions push certificate material to nodes even when paused and raise the alert MachineConfigControllerPausedPoolKubeletCA if a pause would block a rotation; treat either as "unpause this week, not this quarter".

Custom MachineConfigs you will actually write

Four requests cover most of what a platform team gets asked for: point the nodes at the bank's NTP servers, set a kernel argument for a database team, enable a system service the security standard requires, and make the nodes pull images from the internal registry mirror. Each maps to one MachineConfig shape. Naming convention first: <NN>-<role>-<purpose>, and use 99- for your own, because MCs merge in name order and a later name wins when two write the same path. Label it with the role it targets; a role of worker also reaches custom pools that inherit worker configs (our infra pool), while master needs its own copy.

Adding a file: chrony.conf

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-chrony
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 420
          overwrite: true
          contents:
            source: data:text/plain;charset=utf-8;base64,c2VydmVyIG50cDEuY29ycC5leGFtcGxlLmNvbSBpYnVyc3QKc2VydmVyIG50cDIuY29ycC5leGFtcGxlLmNvbSBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkK

The pieces: path is where it lands on the host; mode: 420 is decimal for octal 0644 (Ignition wants decimal; 0600 is 384); overwrite: true says replace whatever the base image put there; and contents.source is a data URL whose payload is base64 of the file. Decode it yourself (echo '<payload>' | base64 -d) and you'll see two server ntp?.corp.example.com iburst lines, a driftfile, makestep, rtcsync and a logdir. To produce the payload, base64 -w0 chrony.conf on Linux (base64 -i chrony.conf on macOS). Or skip the hand encoding entirely: Red Hat ships Butane, a tool that takes a human-readable YAML (variant: openshift, version: 4.16.0, inline: file contents) and emits the MachineConfig for you; butane 99-worker-chrony.bu -o 99-worker-chrony.yaml. Either way, the YAML goes in Git and Argo CD applies it (Post 30), which is how you get a reviewed, reversible history of every OS change on the platform.

Kernel arguments

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-kargs-thp
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  kernelArguments:
    - transparent_hugepage=never

No Ignition body at all; spec.kernelArguments is a top-level MachineConfig field and the MCD applies it with rpm-ostree kargs, which always means a reboot. Verify on a node afterwards with cat /proc/cmdline inside oc debug node. Note what this is not for: sysctls. Runtime kernel parameters such as vm.max_map_count or net.core.somaxconn belong either in a file under /etc/sysctl.d/ delivered by a MachineConfig, or better, in a Tuned profile from the Node Tuning Operator, which can apply them per node label without a reboot.

A systemd unit: enabling kdump

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-kdump
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  kernelArguments:
    - crashkernel=256M
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - name: kdump.service
          enabled: true

Kernel crash dumps are something Red Hat support asks for when a node panics, and a bank's build standard often requires them. This MC reserves crash-kernel memory and enables the unit that already exists in RHCOS. To ship a unit of your own, add a contents: | block under the unit with the usual [Unit], [Service], [Install] sections; the MCD writes it to /etc/systemd/system/ and enables it.

Container registry configuration: mirrors

Nodes decide where to pull images from using /etc/containers/registries.conf. You could write it as a file MC, and for one-off drop-ins under /etc/containers/registries.conf.d/ people do. But mirrors, which every disconnected or security-conscious bank cluster has, have a proper API: ImageDigestMirrorSet (IDMS, for images referenced by digest) and ImageTagMirrorSet (ITMS, by tag). These replaced the older ImageContentSourcePolicy (ICSP) in 4.13, and the MCO renders them into 99-worker-generated-registries and 99-master-generated-registries for you.

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  name: bank-registry-mirrors
spec:
  imageDigestMirrors:
    - source: quay.io/openshift-release-dev/ocp-release
      mirrors:
        - registry.bank.example:5000/openshift/release-images
    - source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
      mirrors:
        - registry.bank.example:5000/openshift/release
    - source: registry.redhat.io
      mirrors:
        - registry.bank.example:5000/redhat

The result on every node is a [[registry]] block per source with the mirror listed first and a fallback to the original. Registry changes are one of the live-apply cases: no reboot, and recent versions skip the drain as well. Two related knobs live elsewhere: which registries are allowed, blocked or insecure is set in image.config.openshift.io/cluster under spec.registrySources, and the CA for your internal registry goes into a ConfigMap referenced by spec.additionalTrustedCA on the same object. Neither is a MachineConfig you write; both end up as files the MCO manages.

KubeletConfig and ContainerRuntimeConfig

Some node settings are so common that Red Hat gave them typed CRs instead of making you write kubelet config files by hand. A KubeletConfig sets kubelet parameters for a pool; a ContainerRuntimeConfig sets CRI-O parameters. Both select a pool via machineConfigPoolSelector, and both are turned into 99-<pool>-generated-kubelet / 99-<pool>-generated-containerruntime MachineConfigs, so the rollout is the same drain-and-reboot per node. The default pools carry a label pools.operator.machineconfiguration.openshift.io/<name> you can match; for custom pools the documented pattern is to add your own label to the MCP first (oc label mcp infra custom-kubelet=infra-limits).

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-kubelet-limits
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    maxPods: 500
    podPidsLimit: 4096
    containerLogMaxSize: 50Mi
    containerLogMaxFiles: 5
---
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: worker-crio-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    logLevel: info
    overlaySize: 20G
    pidsLimit: 4096

What each line is for, because the interviewer will pick one and ask: maxPods raises the default of 250; before you do, check the cluster network's hostPrefix (default /23, about 510 usable Pod IPs per node) and note that podsPerCore also caps it and the lower value wins. podPidsLimit stops a fork bomb in one container from starving the node. containerLogMaxSize and containerLogMaxFiles bound how much of the node disk one chatty container can eat before rotation. In the CRI-O config, overlaySize caps each container's writable layer, pidsLimit is the older CRI-O-level PID cap (prefer the kubelet one), and logLevel: debug is what support will ask you to flip during a runtime investigation. The MCO numbers the generated MachineConfigs (99-worker-generated-kubelet, then -1, -2...) and only honours a small number of them per pool, so keep one KubeletConfig per pool rather than one per request. When you're unsure which fields your version accepts, oc explain kubeletconfig.spec.kubeletConfig and oc explain containerruntimeconfig.spec.containerRuntimeConfig are the honest answer.

Troubleshooting the MCO

MCO failures come in three flavours and each has a signature in oc get mcp. Learn the signatures, then the fix.

1. A node is Degraded: "unexpected on-disk state"

$ oc get mcp worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d   False     True       True       9              8                   7                     1                      412d

$ oc describe mcp worker | grep -A3 'Type:.*NodeDegraded' -B4
    Message:  Node prod-7k2xd-worker-0-l9ktz is reporting: "unexpected on-disk state validating against rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d: content mismatch for file \"/etc/chrony.conf\""
    Reason:   1 nodes are reporting degraded status on sync
    Status:   True
    Type:     NodeDegraded

$ oc get pods -n openshift-machine-config-operator -o wide | grep worker-0-l9ktz
machine-config-daemon-x2ln8   2/2   Running   0   3d   10.20.31.14   prod-7k2xd-worker-0-l9ktz

$ oc logs -n openshift-machine-config-operator machine-config-daemon-x2ln8 -c machine-config-daemon --tail=30
I0908 03:12:44.118 daemon.go:1567] Validating against current config rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d
E0908 03:12:44.201 writer.go:200] Marking Degraded due to: unexpected on-disk state validating against rendered-worker-5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d: content mismatch for file "/etc/chrony.conf"

The pool tells you which node; the node's MCD Pod tells you which file. "Content mismatch" means the file on disk doesn't match what the rendered config says it should be, almost always because a human edited it, occasionally because a previous apply was interrupted mid-write (a power loss during a reboot). The cleanest fix is to put the file back the way the MCO expects and let the MCD re-validate on its next loop. When you can't easily reconstruct it, use the force file: it tells the MCD to skip validation and re-apply the desired config from scratch, reboot included.

$ oc debug node/prod-7k2xd-worker-0-l9ktz -- chroot /host touch /run/machine-config-daemon-force
$ oc get mcp worker -w      # node goes Working, reboots, returns Done; DEGRADEDMACHINECOUNT returns to 0

If the node is beyond that (disk corruption, an rpm-ostree deployment that won't boot), stop fixing it: oc delete machine prod-7k2xd-worker-0-l9ktz -n openshift-machine-api drains and deletes it and the MachineSet builds a replacement from the current rendered config. On UPI, oc delete node and reprovision the VM. Cattle, not pets; a bank's runbook says the same thing.

2. The pool is stuck UPDATING for hours, nothing Degraded

Find the node in Working state with the custom-columns command from earlier, then read its MCD log. The usual cause is a drain that can't complete:

I0908 14:02:11.532 drain.go:120] Draining node prod-7k2xd-worker-0-p4mnd (attempt 3)
E0908 14:02:41.877 drain.go:159] error when evicting pods/"ledger-api-7d9f8c6b5-qq4tz" -n "payments-prod": Cannot evict pod as it would violate the pod's disruption budget.
I0908 14:07:41.880 drain.go:120] Draining node prod-7k2xd-worker-0-p4mnd (attempt 4)

Same PDB story as before, only now nobody is watching a terminal. The MCD retries for an hour, then marks the node Degraded with "failed to drain node" and the alert MCDDrainError fires. Fix the PDB or the app, and the MCD resumes on its own. Two less obvious causes of a stuck pool: the node rebooted and never came back (check vSphere or the BMC, then oc get nodes), and a node that was already NotReady before the rollout started, because with maxUnavailable: 1 the controller counts that node as the one unavailable machine and refuses to touch any other. Fix or delete the sick node and the pool moves.

3. RenderDegraded

The controller couldn't merge the MCs into a rendered config: an invalid Ignition body, an unknown field, a bad base64 payload. oc describe mcp names the MC; fix or delete it. No node was touched, which is the one comforting thing about this failure.

Cluster upgrades: the CVO, channels and the update graph

Everything about a cluster's version is one object: ClusterVersion, named version, owned by the Cluster Version Operator (CVO). Post 19 described the CVO as the operator of operators; in lifecycle terms it's the thing that knows which release image the cluster is running, asks Red Hat which release images it may move to, and drives the move. A release image is a single container image manifest that pins the exact versions of every component: every cluster operator, CRI-O, the kubelet, the RHCOS image. Upgrading means "reconcile the cluster towards a different release image".

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.20   True        False         33d     Cluster version is 4.16.20

$ oc adm upgrade
Cluster version is 4.16.20

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.16 (available channels: candidate-4.16, candidate-4.17, eus-4.16, fast-4.16, fast-4.17, stable-4.16, stable-4.17)

Recommended updates:

  VERSION     IMAGE
  4.16.30     quay.io/openshift-release-dev/ocp-release@sha256:7c1e9d2a8b4f0e6c3d5a1b7f9e2c4d6a8b0f1e3c5d7a9b1f3e5c7d9a1b3f5e7c
  4.16.29     quay.io/openshift-release-dev/ocp-release@sha256:9a0f4b2c6d8e1a3f5c7b9d2e4a6c8f0b1d3e5a7c9f2b4d6e8a0c1f3b5d7e9a2c
  4.16.28     quay.io/openshift-release-dev/ocp-release@sha256:2b3c5d7e9f1a4c6e8b0d2f4a6c8e1b3d5f7a9c2e4b6d8f0a1c3e5b7d9f2a4c6e

Updates with known issues:

  Version: 4.16.27
  Image: quay.io/openshift-release-dev/ocp-release@sha256:e4f6a8c0b2d4f6a8c1e3b5d7f9a2c4e6b8d0f2a4c6e8b1d3f5a7c9e2b4d6f8a0
  Reason: OVNKubernetesPodCrashLoop
  Message: Clusters running OVN-Kubernetes with more than 250 nodes may experience ovnkube-node restarts after updating. https://issues.redhat.com/browse/OCPBUGS-38612

Read that output top to bottom because you'll be asked to explain it. Upstream is the update service the CVO asks: by default Red Hat's hosted OpenShift Update Service (the "Cincinnati" graph), which a disconnected cluster replaces with a local instance. Channel is which slice of the graph you're subscribed to:

• stable-4.x: releases Red Hat has watched in the field for a while; what production runs.
• fast-4.x: the same releases, offered as soon as they're GA; what your lower environments run so you hit problems first.
• candidate-4.x: pre-release builds, no support; for a sandbox only.
• eus-4.x: Extended Update Support, exists only for even-numbered minors (4.14, 4.16, 4.18), and carries the extra edges that make an EUS-to-EUS jump possible.

The update graph is a directed graph whose nodes are releases and whose edges are supported upgrade paths. Two facts about it explain most of the surprises people hit. First, edges are between specific z-streams, not between minors: 4.16.20 may have an edge to 4.17.9 and to 4.17.12 but not to 4.17.4, because Red Hat only tests and publishes paths from the versions that were current when the target shipped. That's why "I want 4.17" sometimes becomes "first go to 4.16.30, then 4.17.12". Second, edges get removed: when a bug is found in a particular path, Red Hat pulls the edge, and a version you saw yesterday is gone today. That's a feature. The list under "Updates with known issues" (older oc prints "Supported but not recommended updates") is the middle ground, conditional updates: the edge exists, but Red Hat has published a risk with a PromQL expression the CVO evaluates against your cluster; if you match the risk (here, more than 250 nodes) it stays hidden unless you pass --include-not-recommended and accept it explicitly. A bank does not do that without a support case saying it's fine.

The rule that follows from all this: you can move between z-streams of one minor freely, you can move to the next minor only via a published edge, and you cannot skip a minor. 4.16 to 4.18 is always two hops of the control plane, even in the EUS-to-EUS procedure. Red Hat's update-path tool on the customer portal draws the path for you; put a screenshot of it in the change record.

Running the upgrade and what actually happens

The commands are short. The understanding of what they set in motion is what you're being hired for.

$ oc adm upgrade channel stable-4.16          # set or change the channel (also: oc patch clusterversion version --type merge -p '{"spec":{"channel":"stable-4.16"}}')
$ oc adm upgrade --to=4.16.30                 # a specific recommended version
Requested update to 4.16.30

$ oc adm upgrade --to-latest=true             # the newest recommended version in the channel
$ oc adm upgrade --to-image=registry.bank.example:5000/openshift/release-images@sha256:7c1e9d2a... --allow-explicit-upgrade   # disconnected: by digest

$ oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.20   True        True          14s     Working towards 4.16.30: 9 of 903 done (0% complete)
version   4.16.20   True        True          6m      Working towards 4.16.30: 117 of 903 done (12% complete), waiting on kube-apiserver
version   4.16.20   True        True          38m     Working towards 4.16.30: 681 of 903 done (75% complete), waiting on machine-config
version   4.16.30   True        False         2m      Cluster version is 4.16.30

Here's the sequence behind "Working towards". The CVO verifies the release image's signature against the keys it trusts (Red Hat's, or the signature ConfigMap you imported on a disconnected cluster), pulls it, and reads the ~900 manifests inside it. It updates itself first. Then it applies manifests in run-level order, which puts the foundations first: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, then the OpenShift API server and OAuth, then networking (OVN-Kubernetes), then everything else. Each cluster operator upgrades its own operand; the control-plane operators do it as rolling static Pod revisions on the masters, one master at a time, with no reboot. You can watch that in oc get co:

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.30   True        False         False      14m
etcd                                       4.16.30   True        False         False      412d
kube-apiserver                             4.16.30   True        True          False      412d    NodeInstallerProgressing: 1 node is at revision 27; 2 nodes are at revision 28
kube-controller-manager                    4.16.30   True        False         False      412d
machine-config                             4.16.20   True        True          False      412d    Working towards 4.16.30
monitoring                                 4.16.30   True        False         False      9d
network                                    4.16.30   True        True          False      412d    DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" update is rolling out (7 out of 12 updated)
...

The last operator to finish is machine-config, and it's the only one that reboots anything. The new release image carries a new RHCOS image, so the MCO renders a new config for every pool and starts the drain-reboot dance from the previous section: the master pool first, one control-plane node at a time (etcd keeps quorum with 2 of 3 throughout). When the master pool is done, the MCO reports the new version, the CVO declares "Cluster version is 4.16.30", and then the MCO starts on the worker pool. That's what "control plane upgrades first, then the MCO rolls workers" means in practice, and it has a consequence you must internalise:

Timing for the change record, from experience on mid-sized clusters: control plane 60 to 90 minutes, then workers at roughly 10 to 15 minutes per node divided by maxUnavailable. A 30-node cluster is an evening. Newer oc builds also have oc adm upgrade status, which summarises control plane, worker pools and estimated completion in one screen; in 4.16 to 4.18 it's behind the environment variable OC_ENABLE_CMD_UPGRADE_STATUS=true, so check oc adm upgrade --help for your version.

EUS-to-EUS: two control-plane hops, one worker reboot

Because worker reboots are the expensive part for the business, Red Hat supports a procedure for jumping between EUS releases (4.14 to 4.16, 4.16 to 4.18) that reboots each worker once instead of twice. Kubernetes allows a kubelet to be two minor versions behind the API server, so workers can stay at 4.14 while the control plane passes through 4.15 to reach 4.16. The steps:

1. Confirm every OLM operator has versions compatible with both 4.15 and 4.16 (next section), and that no workload needs a 4.15 kubelet.
2. Pause the worker pool and every custom pool: oc patch mcp worker --type merge -p '{"spec":{"paused":true}}', same for infra.
3. oc adm upgrade channel eus-4.16. Only the EUS channel carries both hops.
4. Acknowledge any API-removal gate for 4.15, then oc adm upgrade --to=4.15.z. Wait for ClusterVersion, oc get co and the master pool. Workers stay put.
5. Acknowledge the gate for 4.16 if there is one, then oc adm upgrade --to=4.16.z. Wait again.
6. Unpause the pools. Each worker drains and reboots once, landing directly on the 4.16 RHCOS and kubelet.

The intermediate state, a 4.15 control plane with 4.14 workers, is supported only as a transit state; don't camp there. And if OLM operators (logging, ODF, GitOps) each need their own upgrade between hops, that's part of the plan too, which is why an EUS-to-EUS at a bank is a multi-week project with the actual command execution being the short part.

There is no downgrade

Red Hat does not support reverting a cluster to a previous version. Not with oc adm upgrade, not by pointing the CVO at an older image. When the change record asks for a backout plan, the honest answer has four layers and you should be able to say them in this order: (1) fix forward with a Sev-1 Red Hat support case; most "failed upgrades" are one stuck operator or one degraded node, and support has seen it; (2) restore etcd from the pre-upgrade backup, understanding that Red Hat only supports restoring a backup onto control-plane nodes at the same z-stream it was taken from, so this covers "the upgrade went wrong before the release actually moved" and "we rebuilt the control plane at the old version", not a casual version rollback; (3) fail traffic over to the DR cluster, which a bank has, and redeploy applications there from Git; (4) for a single application that breaks on the new version, roll back the application, not the platform. Saying "we'd just roll the cluster back" is the fastest way to end an interview.

Compatibility gates: operators and removed APIs

Two things block a minor upgrade in practice, and both are things you check before you raise the change, not after the CVO tells you.

OLM operators

$ oc get sub -A
NAMESPACE                     NAME                          PACKAGE                       SOURCE             CHANNEL
openshift-compliance          compliance-operator           compliance-operator           redhat-operators   stable
openshift-gitops-operator     openshift-gitops-operator     openshift-gitops-operator     redhat-operators   gitops-1.14
openshift-logging             cluster-logging               cluster-logging               redhat-operators   stable-6.1
openshift-operators-redhat    loki-operator                 loki-operator                 redhat-operators   stable-6.1
openshift-storage             odf-operator                  odf-operator                  redhat-operators   stable-4.16

$ oc get clusterversion version -o jsonpath='{.status.conditions[?(@.type=="Upgradeable")]}' | jq
{
  "type": "Upgradeable",
  "status": "False",
  "reason": "ClusterOperatorsNotUpgradeable",
  "message": "Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking minor version upgrades to 4.17 or higher: maximum supported OCP version for openshift-storage/odf-operator.v4.16.5 is 4.16"
}

Operators installed through OLM (Post 23) can declare olm.maxOpenShiftVersion. When one says "I only support up to 4.16", OLM sets Upgradeable=False on its ClusterOperator, the CVO copies that into ClusterVersion, and oc adm upgrade refuses any 4.17 target. Note the wording: between minor versions. Upgradeable=False never blocks a z-stream. ODF is the classic case because its version is pinned to the OpenShift minor; the sequence is OpenShift 4.16 → ODF 4.17 → OpenShift 4.17. Your pre-upgrade job is to read each operator's compatibility matrix, upgrade the ones that need it in the lower environment first, and confirm oc get csv -A shows every operator Succeeded before the window.

Removed APIs

Every Kubernetes minor removes some beta APIs. If a workload, a CI pipeline or an operator still calls them, it breaks the moment the API server moves. OpenShift records who is calling what in APIRequestCount objects, one per API version, with per-user and per-user-agent detail for the last 24 hours. Here's the check on a lower environment that is ahead, at 4.18, planning 4.19 (Kubernetes 1.32, which removed flowcontrol/v1beta3):

$ oc get apirequestcounts -o jsonpath='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.status.requestCount}{"\t"}{.metadata.name}{"\n"}{end}' | sort
1.32    48      flowschemas.v1beta3.flowcontrol.apiserver.k8s.io
1.32    48      prioritylevelconfigurations.v1beta3.flowcontrol.apiserver.k8s.io

$ oc get apirequestcounts flowschemas.v1beta3.flowcontrol.apiserver.k8s.io -o jsonpath='{range .status.last24h[*].byNode[*].byUser[*]}{.username}{"\t"}{.userAgent}{"\n"}{end}' | sort -u
system:serviceaccount:platform-tools:legacy-audit-exporter    audit-exporter/2.3.1

$ oc get clusterversion version -o jsonpath='{.status.conditions[?(@.type=="Upgradeable")].message}{"\n"}'
Kubernetes 1.32 and therefore OpenShift 4.19 remove several APIs which require admin consent. Please see the following Red Hat KCS for more details: https://access.redhat.com/articles/... Please set 'ack-4.18-kube-1.32-api-removals-in-4.19' to 'true' in the admin-acks configmap in the openshift-config namespace once you have evaluated the impact.

$ oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.18-kube-1.32-api-removals-in-4.19":"true"}}' --type=merge
configmap/admin-acks patched

The first command lists every API version that is going away and how much it's used; the second names the caller so you can go and fix it (here, a tool in the platform team's own namespace, which is usually how it goes). The third is the admin-acks gate: when a minor removes APIs, the CVO refuses the upgrade until an administrator writes a specific key into the admin-acks ConfigMap. Not every minor has this gate, and the exact key differs each time, so don't memorise keys; memorise that the Upgradeable condition message tells you what to set. At a bank the ack itself is an approved step in the change record with the APIRequestCount evidence attached, because it's you certifying that nothing in production still uses the removed APIs.

The bank-grade pre-upgrade checklist

This is the list you recite when asked "walk me through how you'd upgrade production". Say it as a numbered sequence, with the command or the artefact for each step; that's what distinguishes a runbook from a vibe.

1. Read the release notes and known issues for every z-stream between you and the target, plus the Red Hat "update risks" for your path. Note anything touching your platform (vSphere, OVN, ODF) and any bug fixed that you've been carrying a workaround for.
2. Confirm the cluster is healthy before you touch it. oc get co: every operator Available=True, Progressing=False, Degraded=False. oc get mcp: every pool UPDATED=True, DEGRADED=False, none paused unless you meant it. oc get nodes: all Ready. oc get pods -A | grep -vE 'Running|Completed': nothing surprising. No firing critical alerts in Alertmanager. An upgrade never fixes a degraded cluster; it hides the cause under a second problem.
3. Confirm capacity and disruption budgets. Each node will be drained; can the rest absorb it? oc adm top nodes for headroom. oc get pdb -A and look for ALLOWED DISRUPTIONS 0, because each one is a stuck drain waiting to happen. Get app teams to fix those before the window, not during it.
4. Take an etcd backup and verify it (next section). Copy it off the cluster. Record the filename in the change.
5. Check operator compatibility. oc get sub -A, oc get csv -A, the Upgradeable condition on ClusterVersion, and each operator's support matrix for the target minor. Upgrade operators first where required.
6. Check deprecated API usage with oc get apirequestcounts; fix the callers; only then set the admin-ack, with the evidence attached.
7. Check certificates. Custom ingress and API certificates aren't expiring inside the window; the CSR queue is empty (oc get csr | grep Pending); no cluster certificate alerts.
8. Confirm the change ticket, the window and the comms. CAB approval, the maintenance window long enough for the worker rollout (or a paused worker pool with a second window booked), the app teams notified, the on-call bridge scheduled, the Red Hat support case pre-opened as a courtesy for large clusters.
9. Test on a lower environment first, the same path. Same source version, same target, same operators, same procedure, ideally the same automation. If dev went to 4.16.30 via 4.16.27 last week, that's your rehearsal.
10. Write the backout plan down using the four layers above: fix forward with support, etcd restore (and its z-stream limitation), DR failover, application rollback. Name who decides and by when.
11. Have the post-upgrade validation script ready before you start, and run it before the upgrade too so you have a baseline to diff against. Post 32 builds exactly that script.

Post-upgrade validation

1. oc get clusterversion at the target, Progressing=False, no Failing condition.
2. oc get co: all at the target VERSION, none Progressing or Degraded.
3. oc get mcp: all pools UPDATED, none Degraded, none paused (unless the worker window is later, in which case the ticket says so).
4. oc get nodes: all Ready, all on the new kubelet version, none SchedulingDisabled.
5. oc get pods -A | grep -vE 'Running|Completed': only what you expected.
6. Alertmanager: no new critical alerts, and any alert that fired during the window resolved.
7. Platform smoke tests: log in through OAuth, create a project, deploy a test app, get a Route, push and pull from the internal registry, query Prometheus, search Loki, run a Job that mounts a PVC.
8. Application health: the app teams' own synthetic checks green; error rates and latency on the dashboards back to baseline.
9. Take a fresh etcd backup. The pre-upgrade one is now historical.
10. Close the change with the evidence attached: command outputs before and after, the validation script report, the alert timeline.

Patching: RHCOS, cadence and disconnected clusters

A question that catches people coming from RHEL: "how do you patch the OS on the nodes?" The answer is that you don't, not as a separate activity. RHCOS is delivered as an image inside the release payload, and the only supported way to change it is to move the cluster to a release that contains a newer one. There is no dnf update, no Satellite channel for the nodes. Every z-stream release includes the current RHCOS with the current kernel and CVE fixes, and the MCO applies it during the upgrade using rpm-ostree (bootc in the newest releases), as one more rendered-config change with a reboot. So "patching" and "z-stream upgrading" are the same operation, and a bank's patch policy for OpenShift is really a z-stream cadence.

$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.16.30-x86_64 | sed -n '1,8p;/Release Metadata/,/Component Versions/p;/Component Versions/,+3p'
Name:      4.16.30
Digest:    sha256:7c1e9d2a8b4f0e6c3d5a1b7f9e2c4d6a8b0f1e3c5d7a9b1f3e5c7d9a1b3f5e7c
Created:   2025-01-14T03:12:41Z
OS/Arch:   linux/amd64
Manifests: 719
Metadata files: 1

Release Metadata:
  Version:  4.16.30
  Upgrades: 4.15.35, 4.15.36, 4.15.37, 4.15.38, 4.16.20, 4.16.21, 4.16.22, ..., 4.16.29
  Metadata:
    url: https://access.redhat.com/errata/RHSA-2025:0193

Component Versions:
  kubernetes 1.29.11
  machine-os 416.94.202501100914-0 Red Hat Enterprise Linux CoreOS

Three things to point at in that output. The Upgrades: line is the graph's incoming edges for this release: every version you're allowed to come from. The url: is the errata; RHSA means a security advisory, and its severity (Critical, Important, Moderate, Low) is what your vulnerability-management process keys on. And machine-os is the RHCOS build, which you quote in the change record when the security team asks "which kernel will the nodes be on".

Z-stream vs minor. A z-stream (4.16.20 → 4.16.30) is bug and security fixes only, no feature or API changes, weekly-ish releases from Red Hat. A minor (4.16 → 4.17) brings a new Kubernetes version, possible API removals, operator compatibility work, and needs the full checklist. A realistic bank cadence: z-streams monthly, staged dev → UAT → production a week apart, one z-stream behind the newest on the stable channel; minors twice a year or EUS-to-EUS once a year, as a project. CVE-driven emergency patches break the cadence: a Critical CVE in the kernel, CRI-O or OVN triggers an expedited change, still with the checklist, still lower environment first, but with hours between environments instead of days. The same errata mechanism tells you whether you're affected: oc adm release info for what's in the fix, and the Insights advisor for whether your cluster matches the risk.

Insights. The insights-operator uploads anonymised cluster configuration to Red Hat, and Insights Advisor on console.redhat.com turns it into recommendations: known-bad configurations, upgrade risks specific to your cluster, CVE exposure. Banks in a disconnected setup either disable it or feed it through a proxy with the data-collection policy signed off by security; either way, know it exists, because an interviewer at a Red Hat shop will mention it.

Disconnected clusters. Many bank clusters can't reach quay.io or registry.redhat.io. Then a release is a thing you physically bring in. The tool is oc-mirror (v2 in 4.16 and later). You describe what you need in an ImageSetConfiguration, mirror it to disk on a connected machine, carry it across the air gap, push it into the internal registry, and apply the IDMS, ITMS and CatalogSource objects the tool generates.

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
      - name: stable-4.16
        minVersion: 4.16.20
        maxVersion: 4.16.30
    graph: true
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
      packages:
        - name: cluster-logging
        - name: loki-operator
        - name: compliance-operator
        - name: odf-operator
  additionalImages:
    - name: registry.redhat.io/ubi9/ubi:latest

$ oc-mirror -c imageset.yaml file:///data/mirror --v2                                       # connected side: mirror to disk
$ oc-mirror -c imageset.yaml --from file:///data/mirror docker://registry.bank.example:5000 --v2   # disconnected side: disk to registry
$ oc apply -f /data/mirror/working-dir/cluster-resources/                                    # IDMS, ITMS, CatalogSource, signature ConfigMap
$ oc adm upgrade --to-image=registry.bank.example:5000/openshift/release-images@sha256:7c1e9d2a... --allow-explicit-upgrade

graph: true also mirrors the update graph data so you can run the OpenShift Update Service inside the network; point spec.upstream on ClusterVersion at it and oc adm upgrade behaves as if connected, with the graph limited to what you mirrored. Without OSUS you upgrade by digest with --allow-explicit-upgrade, and you must apply the release signature ConfigMap first, or the CVO rejects the image as unverified. The unsupported shortcut is --force, which skips signature verification; at a bank it isn't an option. The mirror registry itself becomes a tier-1 dependency (nodes pull from it on every reboot), so it's redundant, monitored and in scope for the DR plan.

etcd operations: health, backup, restore

One-line recap from Post 2: etcd is the key-value store that holds every object in the cluster, so losing it is losing the cluster, and it needs a majority of its members (2 of 3) to accept writes. On OpenShift, etcd runs as static Pods on the three control-plane nodes, managed by the cluster-etcd-operator (CEO), which handles membership when a master is replaced, rotates etcd certificates, defragments automatically, and enforces the backend quota. You read its state from inside the etcd Pod, where etcdctl and its connection variables are already set up.

$ oc get pods -n openshift-etcd -l app=etcd
NAME                        READY   STATUS    RESTARTS   AGE
etcd-prod-7k2xd-master-0    4/4     Running   0          12d
etcd-prod-7k2xd-master-1    4/4     Running   0          12d
etcd-prod-7k2xd-master-2    4/4     Running   0          12d

$ oc rsh -n openshift-etcd etcd-prod-7k2xd-master-0
sh-5.1# etcdctl endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.20.30.11:2379 | 3f8d2a1c9b7e6f5a |  3.5.16 |  318 MB |     false |      false |        87 |   41982731 |           41982731 |        |
| https://10.20.30.12:2379 | 8a1b2c3d4e5f6071 |  3.5.16 |  318 MB |      true |      false |        87 |   41982731 |           41982731 |        |
| https://10.20.30.13:2379 | c7d6e5f4a3b2c1d0 |  3.5.16 |  319 MB |     false |      false |        87 |   41982731 |           41982731 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
sh-5.1# etcdctl endpoint health
https://10.20.30.11:2379 is healthy: successfully committed proposal: took = 9.12ms
https://10.20.30.12:2379 is healthy: successfully committed proposal: took = 7.84ms
https://10.20.30.13:2379 is healthy: successfully committed proposal: took = 11.03ms
sh-5.1# etcdctl member list -w table

What a healthy table looks like: three endpoints, one leader, identical RAFT INDEX and APPLIED INDEX (nobody is lagging), similar DB SIZE, a RAFT TERM that isn't climbing every time you look (a rising term means leader elections, which means a member or its disk is unhealthy), and an empty ERRORS column. The health check's took is a real write round-trip; single-digit milliseconds is good, hundreds is a disk or network problem.

Backup

Red Hat ships the backup script on every control-plane node. You run it through oc debug; no SSH needed.

$ oc debug node/prod-7k2xd-master-0 -- chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup
Temporary namespace openshift-debug-k4n2p is created for debugging node...
Starting pod/prod-7k2xd-master-0-debug-7pq2m ...
To use host binaries, run `chroot /host`
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-28
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-14
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-11
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-9
{"level":"info","ts":"2026-09-08T02:00:03.112Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2026-09-08_020003.db.part"}
{"level":"info","ts":"2026-09-08T02:00:04.938Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetched snapshot","endpoint":"https://10.20.30.11:2379","size":"318 MB","took":"1.8s"}
Snapshot saved at /home/core/assets/backup/snapshot_2026-09-08_020003.db
{"hash":2318932456,"revision":41982731,"totalKey":38122,"totalSize":318357504}
snapshot db and kube resources are successfully saved to /home/core/assets/backup

Removing debug pod ...

$ oc debug node/prod-7k2xd-master-0 -- chroot /host ls -lh /home/core/assets/backup
-rw-------. 1 root root 304M Sep  8 02:00 snapshot_2026-09-08_020003.db
-rw-------. 1 root root  92K Sep  8 02:00 static_kuberesources_2026-09-08_020003.tar.gz

Two files, and you should be able to say what each is. snapshot_<timestamp>.db is the etcd snapshot itself: every object in the cluster at that revision. static_kuberesources_<timestamp>.tar.gz is the static Pod resources for etcd and the control-plane components from that node, and, if etcd encryption at rest is on, the encryption keys the snapshot was written with. That single fact drives the storage policy: the backup pair contains every Secret in the bank's cluster and the keys to read them. So: off the cluster immediately (a control-plane disk is the one place a backup is useless when you need it), encrypted at rest, in a location the platform team can reach when the cluster is dead, with retention that matches the change window (daily for 30 days, plus one kept per change record, plus quarterly restore tests). Verify before you trust it: the script's own JSON line (hash, revision, totalKey) is the snapshot status, and the revision should match what endpoint status showed; on the bastion where the off-cluster copy lands, etcdutl snapshot status snapshot_….db -w table re-checks the copy.

Nobody runs that by hand every night. The usual automation is a CronJob on the control-plane nodes; a trimmed shape:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: ocp-etcd-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: etcd-backup      # granted the privileged SCC (Post 22)
          nodeSelector:
            node-role.kubernetes.io/master: ""
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          hostPID: true
          restartPolicy: Never
          containers:
            - name: backup
              image: registry.redhat.io/openshift4/ose-cli:latest
              securityContext:
                privileged: true
              command: ["/bin/bash", "-c"]
              args:
                - chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup
                  && /scripts/push-encrypted-copy.sh /host/home/core/assets/backup
                  && chroot /host find /home/core/assets/backup -mtime +7 -delete
              volumeMounts:
                - { name: host, mountPath: /host }
          volumes:
            - name: host
              hostPath: { path: /, type: Directory }

The push script is where the bank-specific part lives: encrypt, copy to object storage or an NFS share outside the cluster, alert on failure, and prune. Newer releases also ship a Technology Preview automated backup (a Backup CR the CEO acts on); until it's GA, the CronJob or an Ansible job (Post 32) is what production uses. And take a manual one before every change, named for the change ticket.

Restore

When do you restore? Only when the cluster's state is gone or wrong as a whole: quorum loss (two of three control-plane nodes lost and unrecoverable), etcd data corruption, or a catastrophic mistake that touched the whole cluster. Not for one dead master: that is "replacing an unhealthy etcd member" (etcdctl member remove, delete the member's secrets in openshift-etcd, delete and recreate the Machine; the CEO adds the new member), which is a routine procedure with no data loss. And not for "someone deleted a namespace": an etcd restore rewinds everything to the snapshot, so for application recovery you want OADP (Velero) backups of namespaces and volumes, not the cluster's brain. Restore is the last resort, and a bank rehearses it in a lab twice a year precisely because nobody wants their first attempt to be on production.

The procedure, from the Red Hat docs, at the level you need to narrate it:

1. Pick one surviving control-plane node as the recovery host and copy the backup pair onto it. You need SSH as core to all control-plane nodes here, because the API may be down; keep those keys in the vault for this reason.
2. On every other control-plane node, stop the static Pods: move /etc/kubernetes/manifests/etcd-pod.yaml and kube-apiserver-pod.yaml out of the manifests directory, confirm with crictl ps | grep etcd that they're gone, and move /var/lib/etcd/ aside.
3. On the recovery host: sudo -E /usr/local/bin/cluster-restore.sh /home/core/assets/backup. It restores the snapshot into a single-member etcd and starts it with the saved static Pod resources.
4. Restart the kubelet on all control-plane nodes (sudo systemctl restart kubelet.service), then approve any pending CSRs from the oc side.
5. Confirm a single etcd member is running, then force the CEO to redeploy etcd so the other two rejoin: oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$(date --rfc-3339=ns)"'"}}' --type=merge, and wait for 3/3 Pods.
6. Force the same redeployment for kubeapiserver, kubecontrollermanager and kubescheduler; newer docs add restarting the OVN-Kubernetes control-plane and node Pods so the network database matches the restored state.
7. Verify: oc get co, oc get nodes, etcdctl endpoint status, and then the application teams, because anything created after the snapshot no longer exists in the API while its volumes and external side effects still do.

etcd performance: defrag, disk latency, encryption

etcd's on-disk database only grows unless it's defragmented, and etcd stops accepting writes when it hits its quota (8 GiB by default; 4.15 and later let you raise it with spec.backendQuotaGiB on the etcd CR, and you can't lower it again). The CEO defragments members automatically when fragmentation is high and the cluster is healthy, so manual defrag is rare, but you should know it: one member at a time, non-leaders first, leader last, and clear the alarm if the quota was hit.

sh-5.1# unset ETCDCTL_ENDPOINTS
sh-5.1# etcdctl --command-timeout=30s --endpoints=https://localhost:2379 defrag
Finished defragmenting etcd member[https://localhost:2379]
sh-5.1# etcdctl endpoint status -w table        # DB SIZE should drop
sh-5.1# etcdctl alarm list
memberID:9143847829583234432 alarm:NOSPACE
sh-5.1# etcdctl alarm disarm

The alerts you'll see in Post 24's Alertmanager, in the order they matter: etcdInsufficientMembers and etcdMembersDown (quorum at risk, page immediately); etcdHighFsyncDurations and etcdHighCommitDurations (the disk under etcd is too slow: the 99th-percentile WAL fsync is over the alert threshold, where healthy is under 10 ms); etcdHighNumberOfLeaderChanges (usually the same disk or network problem showing up as elections); etcdDatabaseQuotaLowSpaceUsed and etcdExcessiveDatabaseGrowth (something is creating objects fast: an operator in a loop, a CI job spraying ConfigMaps, or events). Disk latency is the one banks get wrong most, by putting control-plane VMs on the same shared datastore as everything else. The fix is a dedicated low-latency disk for /var/lib/etcd (there's a MachineConfig-based procedure to move it), and Red Hat's etcd-perf container to prove the disk meets the fsync requirement before you go live. The etcd CR's controlPlaneHardwareSpeed field can loosen heartbeat and election timeouts for genuinely slow platforms; it's a mitigation, not a fix.

Encryption at rest. By default etcd stores Secrets as plaintext on the control-plane disks (the disks themselves may be encrypted, which is a separate control). Turning on API-level encryption is one field on the apiserver CR, and a bank turns it on:

$ oc patch apiserver cluster --type=merge -p '{"spec":{"encryption":{"type":"aesgcm"}}}'
$ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
EncryptionCompleted
All resources encrypted: routes.route.openshift.io
$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
EncryptionCompleted
All resources encrypted: secrets, configmaps

aesgcm (4.13 and later) or aescbc; the encrypted resources are Secrets, ConfigMaps, Routes and OAuth tokens; the keys live as Secrets in openshift-config-managed, are rotated automatically, and, once more, end up in your backup tar. Encryption is a migration, not a flag flip: the API servers roll new revisions and rewrite every affected object, which takes minutes to hours depending on size, and the Encrypted condition tells you when it's done.

Certificates: what rotates itself and what doesn't

Almost every certificate inside an OpenShift cluster is generated and rotated by an operator: the kubelet client and serving certificates (short-lived, renewed through CSRs that the controller manager auto-approves for existing nodes), the control-plane signers and their leaf certificates, etcd's peer and serving certificates (rotated by the CEO), the service serving certificates apps get from the service.beta.openshift.io/serving-cert-secret-name annotation (Post 22). You watch them rather than manage them: cluster certificate alerts in Alertmanager, and oc adm ocp-certificates, which in current oc has subcommands for monitoring and regenerating cluster certificates; run oc adm ocp-certificates --help for what your version offers. One long-lived exception that bit old clusters: the machine-config-server serving CA, valid ten years from install, which is why that command family includes regenerate-machine-config-server-serving-cert and update-ignition-ca-bundle-for-machine-config-server for the day a 2020-era cluster's Ignition CA runs out.

Rotation happens only while the cluster is running. That's the shutdown trap: switch a cluster off for weeks (a DR test, a data-centre power event) and the kubelet certificates expire in the dark. On startup the nodes come back NotReady with a queue of Pending CSRs, which you approve in two rounds exactly as in the node section. If a signing CA itself expires while the cluster is off, you're into Red Hat's certificate-recovery procedure rather than a CSR approval, so before any planned shutdown you check the date on the signer that matters:

$ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}{"\n"}'
2027-08-05T14:37:50Z

Two certificates are yours, not the cluster's, because users see them: the wildcard certificate for *.apps.<cluster domain> that the ingress routers present, and the certificate for api.<cluster domain>. Both come from the bank's internal PKI (or a public CA for internet-facing clusters), both expire on the PKI's schedule, both are replaced as a change. The docs procedure, which you should be able to recite:

$ oc create configmap custom-ca --from-file=ca-bundle.crt=bank-root-ca.pem -n openshift-config
$ oc patch proxy/cluster --type=merge -p '{"spec":{"trustedCA":{"name":"custom-ca"}}}'

$ oc create secret tls apps-wildcard-2026 --cert=apps-wildcard.crt --key=apps-wildcard.key -n openshift-ingress
$ oc patch ingresscontroller.operator default -n openshift-ingress-operator --type=merge \
    -p '{"spec":{"defaultCertificate":{"name":"apps-wildcard-2026"}}}'

$ oc create secret tls api-cert-2026 --cert=api.crt --key=api.key -n openshift-config
$ oc patch apiserver cluster --type=merge \
    -p '{"spec":{"servingCerts":{"namedCertificates":[{"names":["api.prod.bank.example"],"servingCertificate":{"name":"api-cert-2026"}}]}}}'

The first pair is the custom CA bundle: the bank's root CA goes into a ConfigMap in openshift-config (the installer creates one called user-ca-bundle if you supplied additionalTrustBundle at install) and the proxy object's trustedCA points at it; the network operator merges it with the system roots into trusted-ca-bundle and injects it into any ConfigMap labelled config.openshift.io/inject-trusted-cabundle=true, which is how Pods learn to trust internal services. The ingress change rolls the router Pods; the API change rolls kube-apiserver revisions, and every kubeconfig and automation that pinned the old CA must trust the new chain before you do it. Track expiry in the same place you track any other bank certificate, and consider the cert-manager Operator for OpenShift to renew the wildcard automatically from the internal CA.

Graceful shutdown and startup

Banks shut clusters down on purpose: annual data-centre power maintenance, a DR exercise that proves the standby site, a hardware refresh. Doing it in the wrong order costs hours on the way back up. The procedure:

1. Take an etcd backup and copy it off. Check the signer certificate date from the previous section against how long the cluster will be off.
2. Tell the app teams; stop inbound traffic at the load balancer so users get a maintenance page rather than errors.
3. Optionally cordon and drain the workers first so applications shut down through their normal termination hooks instead of a hard node halt: for n in $(oc get nodes -l node-role.kubernetes.io/worker -o name); do oc adm cordon $n; oc adm drain $n --ignore-daemonsets --delete-emptydir-data --force; done. Skip if the window is short and the apps are stateless.
4. Shut the nodes down from inside, workers first, control plane last. The docs use a one-minute delayed halt through oc debug:

$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[*].metadata.name}'); do
    oc debug node/${node} -- chroot /host shutdown -h 1
  done
$ for node in $(oc get nodes -l node-role.kubernetes.io/master -o jsonpath='{.items[*].metadata.name}'); do
    oc debug node/${node} -- chroot /host shutdown -h 1
  done

Startup is the mirror image. Power on the control-plane nodes and wait until oc get nodes answers and the three etcd Pods are Running (they need each other to form quorum; one alone won't). Power on infra, then workers. Then the two things people forget: oc get csr and approve anything Pending (two rounds), and oc adm uncordon every node you drained. Finally the same validation as after an upgrade: oc get co with nothing Degraded, oc get mcp, oc get pods -A for anything not Running, a smoke test through the router, and only then reopen the load balancer. Budget at least 30 minutes for the control plane to settle before you start diagnosing operators that are just still starting.

Fleet: Red Hat Advanced Cluster Management

"Large-scale environments" in the JD means more than one cluster: production in two data centres, DR, UAT, dev, maybe an EKS cluster or two, each with its own version, its own operators and its own compliance posture. Nobody runs the checklist above by hand across twenty clusters. Red Hat Advanced Cluster Management (RHACM) is the fleet layer: one hub cluster running the operator, and every other cluster imported as a managed cluster by installing a small agent (the klusterlet) that phones home.

$ oc get managedclusters
NAME              HUB ACCEPTED   MANAGED CLUSTER URLS                             JOINED   AVAILABLE   AGE
local-cluster     true           https://api.hub.bank.example:6443                True     True        200d
ocp-prod-tor      true           https://api.prod-tor.bank.example:6443           True     True        180d
ocp-prod-mtl      true           https://api.prod-mtl.bank.example:6443           True     True        180d
ocp-uat           true           https://api.uat.bank.example:6443                True     True        190d
eks-dev-ca        true           https://A1B2C3.gr7.ca-central-1.eks.amazonaws.com  True   True        60d

Four things it does that map straight onto this post. Cluster lifecycle: create clusters from the hub (Hive ClusterDeployment and ClusterPool objects, or zero-touch provisioning for bare-metal sites), and upgrade them from the hub with a ClusterCurator, which can run Ansible pre- and post-hooks around oc adm upgrade, so your checklist becomes code that runs the same way on every cluster. Governance: a Policy declares a desired configuration and RHACM reports (inform) or enforces (enforce) it everywhere a placement selects, giving you one screen that says which clusters are on an approved version, have etcd encryption on, have the right IDMS, and this is the compliance evidence Post 31 talks about. Observability: a Thanos-backed view of metrics and alerts across the fleet. Applications: integration with Argo CD ApplicationSets so a change in Git lands on every cluster (Post 30).

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: approved-ocp-version
  namespace: fleet-policies
spec:
  remediationAction: inform
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: approved-ocp-version
        spec:
          severity: high
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: config.openshift.io/v1
                kind: ClusterVersion
                metadata:
                  name: version
                status:
                  desired:
                    version: 4.16.30

Bound to a Placement that selects the production clusters, this Policy turns "are all prod clusters on the approved z-stream?" into a green or red row on the governance dashboard, which is the kind of answer an auditor accepts and a spreadsheet isn't. In the interview, one sentence is enough: "RHACM gives us hub-based lifecycle, policy-driven compliance and fleet observability, and it's how we'd upgrade twenty clusters with the same runbook."

Likely interview questions See Post 33 and Post 34

Next up: OpenShift networking and storage end to end, from OVN-Kubernetes, Routes and the ingress operator to CSI drivers, ODF and the storage classes a bank actually allows, and how to troubleshoot each layer when an app team says "it's the platform".