Chapter 19
OpenShift Architecture: What Red Hat Adds on Top of Kubernetes
Before you read, guessWhat role does the Cluster Version Operator play, and which command is critical for diagnosing a failing cluster?
Take ten seconds and guess — even a wrong guess makes the answer stick. Tap to see where the chapter lands, or just read on.
The Cluster Version Operator owns the cluster's version and installs everything from a single release image; every platform component reports a ClusterOperator. oc get co is the first command on any sick cluster, and AVAILABLE, PROGRESSING and DEGRADED each mean something specific.
You already know Kubernetes: pods, deployments, services, RBAC, ingress, the whole of Posts 01–17. OpenShift is that same Kubernetes, but wrapped in an opinionated, supported, integrated platform where the operating system, the container runtime, the networking, the registry, the login flow, the monitoring stack and the upgrade path all ship together and are all managed by the cluster itself. Interviewers for an OpenShift platform role rarely ask "what is a pod?" They ask "where does Kubernetes end and OpenShift begin?", and they listen for whether you can name the pieces Red Hat added, why they exist, and which command you type first when one of them breaks. After this post you will be able to draw the OpenShift 4 stack on a whiteboard, read oc get co like a dashboard, walk through the openshift-* namespaces, describe a bank-scale reference architecture out loud, and translate any Kubernetes concept into its OpenShift equivalent in one sentence.
Where Kubernetes ends and OpenShift begins
Here is the shortest true statement about OpenShift: OpenShift Container Platform (OCP) = upstream Kubernetes, unmodified at the API level, plus a curated operating system, a container runtime, a networking plugin, a set of platform services (registry, ingress router, OAuth, monitoring, console) and an army of Operators that install, configure and upgrade all of it as one product. Every Kubernetes object from Track 1 works exactly the same way, and kubectl works against an OpenShift cluster with no changes. What OpenShift adds is everything upstream Kubernetes deliberately leaves to you: which Linux to run, how to patch it, how users log in, how traffic gets in, where images live, how you upgrade, and who to call when it breaks.
That last point matters at a bank. With kubeadm you own every decision and every failure. With OpenShift, Red Hat owns the integration and the support contract, and the platform team owns operating it correctly. The job sits exactly on that boundary: you are the person who understands both layers and knows which one a given problem lives in.
kubeadm plus your own CNI, ingress, registry, auth and monitoring) or run the one a vendor stands behind. The kernel is still the kernel.For every new concept in this post, ask: "Is this Kubernetes that OpenShift ships, or something Red Hat added?" That question is the interview answer.
The OpenShift 4 stack, layer by layer
OpenShift 4 (4.1 in 2019 through 4.19 today) was a ground-up redesign of OpenShift 3. Its defining idea is that the cluster manages itself, top to bottom, including the operating system. Walk the stack from the metal up.
Layer 1: RHCOS, the immutable operating system
Red Hat Enterprise Linux CoreOS (RHCOS) = a minimal, container-focused variant of RHEL, built for OpenShift, shipped as part of the OpenShift release and managed by the cluster rather than by a sysadmin. Three properties define it:
- Immutable. The OS is a versioned image, not a pile of packages. Updates are applied atomically with rpm-ostree (the new OS version is staged whole, the node reboots into it, and a failed update rolls back to the previous whole). There is no
yum updateon a node. - Cluster-managed. The Machine Config Operator (MCO) owns every node's OS configuration: kernel arguments, systemd units, files under
/etc, the kubelet and CRI-O configs, SSH keys and the OS image version itself. You change nodes by writing aMachineConfig; the MCO renders it, then drains, applies, reboots and uncordons each node in turn. Post 20 goes deep on this. - Provisioned by Ignition. Ignition = the first-boot provisioning tool for CoreOS-style systems. A fresh RHCOS machine fetches an Ignition config from the cluster's Machine Config Server, lays down disks, users, files and units, and joins the cluster. No Ansible run, no golden image, no hardening script that drifts.
As of 4.19 RHCOS is effectively the only node OS: control-plane nodes were always RHCOS, and Red Hat has retired support for RHEL worker nodes. When you need something on the host that RHCOS does not ship (a hardware driver, a security agent), the sanctioned path is on-cluster layering: build a custom RHCOS image with extra RPMs layered on top and let the MCO roll it out to a pool. You still never touch a node by hand. The supported way to look at one is a debug pod that mounts the host filesystem, not an SSH session:
$ oc debug node/ocp-dev-worker-2.bank.example.com
Starting pod/ocp-dev-worker-2bankexamplecom-debug-xk9p2 ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.140.22
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# rpm-ostree status
State: idle
Deployments:
* ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c1e9d...
Digest: sha256:7c1e9d...
Version: 418.94.202505151234-0 (2025-05-15T12:34:56Z)
sh-5.1# crictl ps | head -3
CONTAINER IMAGE CREATED STATE NAME POD ID
a91f3c2e0d4b 6b3d5e1f2a9c 3 hours ago Running router 1f0e2d3c4b5a
c02d4e5f6a7b 9d8c7b6a5f4e 3 hours ago Running node-exporter 2a1b0c9d8e7frpm-ostree status shows the node booted from a single versioned OS image; the version string 418.94.… encodes "OpenShift 4.18, based on RHEL 9.4", and the CustomOrigin line often shown beneath it says Managed by machine-config-operator, the whole philosophy in five words. crictl, not docker, inspects containers on the host, because of layer 2.
Degraded, and upgrades are blocked until someone reconciles it. The right answer: express the setting as a MachineConfig targeted at the right pool and let the MCO roll it out node by node. Use oc debug node/ to look, never to fix.Layer 2: CRI-O, the container runtime
CRI-O = a lightweight runtime built purely to implement the Kubernetes Container Runtime Interface (CRI), the gRPC contract the kubelet uses to say "pull this image, start this container." There is no Docker daemon on an OpenShift 4 node; CRI-O uses runc (or crun, the default since 4.18) to launch processes, and its version tracks the Kubernetes minor it ships with, so OpenShift 4.18 ships CRI-O 1.31 with Kubernetes 1.31. It is simply the "runtime" box from Post 2, with a specific minimal choice tied to the release so it upgrades with everything else.
Layer 3: the Kubernetes core
Above the runtime sits the Kubernetes you know: etcd, kube-apiserver, kube-controller-manager and kube-scheduler on the control plane; kubelet and kube-proxy-equivalent networking on every node. Each OpenShift minor pins one Kubernetes minor: 4.14 ships 1.27, 4.15 ships 1.28, 4.16 ships 1.29, 4.17 ships 1.30, 4.18 ships 1.31 and 4.19 ships 1.32. Memorise it, because an experienced interviewer hears "we ran 4.16", maps it to "Kubernetes 1.29" and checks whether your stories line up.
Layer 4: the cluster operators
This layer makes OpenShift OpenShift. Every platform component (API servers, etcd, networking, DNS, ingress, registry, authentication, monitoring, console, machine and node management, storage drivers) is installed, configured, health-checked and upgraded by its own Operator: a controller that watches a custom resource describing the desired state of that component and continuously reconciles the real component toward it, the reconciliation loop from Post 1 pointed at platform software instead of your app. The next section is entirely about this layer, because most day-2 operations and most interview questions live here.
"Everything is an Operator": CVO, ClusterOperators and the release payload
In Kubernetes, "the cluster" is a set of processes you installed. In OpenShift 4, it is a set of Operators that installed themselves from a single versioned bundle and keep reporting their own health.
The Cluster Version Operator and the release image
The Cluster Version Operator (CVO) = the top-level operator, in openshift-cluster-version, responsible for the version of the whole cluster. It reads a release image (the release payload): a single container image, published at quay.io/openshift-release-dev/ocp-release, containing the manifests for every platform component plus a digest-pinned list of the exact component images that make up that OpenShift version. On install the CVO applies those manifests in run-level order (filenames start with 0000_20_, 0000_50_ and so on, so etcd and the kube-apiserver operator come up before ingress and console); on upgrade it swaps the payload and does it again. If something is part of the platform, the CVO put it there. Its own custom resource is the ClusterVersion object, always named version:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.18.14 True False 41d Cluster version is 4.18.14
$ oc get clusterversion version -o jsonpath='{.status.desired.image}{"\n"}'
quay.io/openshift-release-dev/ocp-release@sha256:3f0c8b9d2e1a7c6b5d4e3f2a1b0c9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c
$ oc adm release info 4.18.14 | head -8
Name: 4.18.14
Digest: sha256:3f0c8b9d2e1a7c6b5d4e3f2a1b0c9d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c
Created: 2025-06-03T09:12:41Z
Manifests: 771
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:3f0c8b...AVAILABLE True means the cluster is at the version it claims. PROGRESSING True means an upgrade is in flight, and STATUS becomes a live message like Working towards 4.18.15: 412 of 903 done (45% complete). spec.channel and spec.desiredUpdate on the same object are how upgrades are requested, which is Post 20's territory.
ClusterOperators and how to read oc get co
Each platform operator publishes a ClusterOperator object (short name co) summarising its component's health in a few standard conditions. The CVO watches these to decide whether the cluster is healthy and whether an upgrade may proceed. Listing them is the single most useful command in OpenShift operations:
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.18.14 True False False 41d
baremetal 4.18.14 True False False 88d
cloud-controller-manager 4.18.14 True False False 88d
cloud-credential 4.18.14 True False False 88d
config-operator 4.18.14 True False False 88d
console 4.18.14 True False False 41d
control-plane-machine-set 4.18.14 True False False 88d
dns 4.18.14 True False False 88d
etcd 4.18.14 True False False 88d
image-registry 4.18.14 True False True 3h12m ImageRegistryAvailable: The registry is degraded: storage backend is not reachable...
ingress 4.18.14 True False False 41d
insights 4.18.14 True False False 88d
kube-apiserver 4.18.14 True False False 88d
kube-controller-manager 4.18.14 True False False 88d
kube-scheduler 4.18.14 True False False 88d
machine-api 4.18.14 True False False 88d
machine-config 4.18.14 True False False 41d
marketplace 4.18.14 True False False 88d
monitoring 4.18.14 True False False 41d
network 4.18.14 True False False 88d
node-tuning 4.18.14 True False False 41d
olm 4.18.14 True False False 41d
openshift-apiserver 4.18.14 True False False 41d
openshift-controller-manager 4.18.14 True False False 41d
operator-lifecycle-manager 4.18.14 True False False 88d
service-ca 4.18.14 True False False 88d
storage 4.18.14 True False False 88dKnow the columns cold:
- VERSION is the version this operator has reached. During an upgrade the operators move one at a time, so a mix of
4.18.14and4.18.15means the CVO is mid-rollout. - AVAILABLE answers "is the component doing its job right now?"
Falseis an outage of that component: ifingressis not Available, no Route works. - PROGRESSING answers "is the operator changing something?" Normal during an upgrade or after a config change; a warning sign if it stays
Truewith no upgrade running, because the operator keeps trying and failing to converge. - DEGRADED answers "is something wrong that a human should look at, even if the component still mostly works?" Scan it first. The
image-registryline above isAVAILABLE TrueandDEGRADED Trueat once, a common and meaningful pair: the registry still serves images, but its storage backend is unreachable, so pushes fail. A degraded operator also typically blocks the CVO from starting the next upgrade. - SINCE is how long the conditions have been in this state;
3h12mnext to a degraded operator dates the incident for your RCA. MESSAGE is the operator's own explanation, truncated; the full text lives in the conditions.
A fourth condition, Upgradeable, has no column. When it is False, the operator is saying "do not start a minor-version upgrade until you fix this" (a deprecated API still in use, an unsupported configuration, a pending cert rotation); you see it in oc adm upgrade output and in oc get co <name> -o yaml. To get from the table to the actual reason, describe the operator and read its conditions:
$ oc get co | grep -v 'True False False'
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.18.14 True False True 3h12m ImageRegistryAvailable: The registry is degraded: storage backend is not reachable...
$ oc describe co image-registry | sed -n '/Conditions:/,/Extension:/p'
Conditions:
...
Last Transition Time: 2025-09-08T06:51:02Z
Message: ImageRegistryAvailable: The registry is degraded: storage backend is not reachable: RequestError: send request failed: dial tcp 10.20.30.40:443: i/o timeout
Reason: StorageBackendUnreachable
Status: True
Type: Degraded
$ oc get co image-registry -o jsonpath='{.status.relatedObjects[*].name}{"\n"}'
openshift-image-registry cluster image-registry-operator ...The grep -v trick (hide every line that is Available, not Progressing, not Degraded) prints only the header on a healthy cluster. relatedObjects is the operator telling you which namespace and resources to look at next, which is exactly what oc adm must-gather uses to decide what to collect.
oc get co, then oc get clusterversion, oc get nodes and oc get mcp, in that order, before any oc get pods -A. On Kubernetes you start from pods because there is no higher-level summary; on OpenShift the platform summarises itself, and a candidate who starts grepping pods across sixty openshift-* namespaces has not internalised that. Then the follow-through: find the degraded operator, oc describe co it, read its conditions and relatedObjects, and only then go to the pods and logs in the namespace it points at. Post 25 builds a full playbook around that method.oc get co is that board. "Available but Degraded" is a trade still on site but with a hand raised; "not Upgradeable" is a trade saying "don't start the next phase until I have sorted this out."Don't edit the operand, edit the operator's config
The Deployments, DaemonSets and ConfigMaps that operators create (their operands) are owned by the operator, which overwrites any direct change within seconds. For three router replicas instead of two, you do not scale the router-default Deployment; you edit spec.replicas on the IngressController named default in openshift-ingress-operator. To change cluster DNS behaviour you edit the DNS object named default, not the dns-default DaemonSet. The config objects are almost always singletons named cluster or default, and oc explain ingresscontroller.spec or oc explain dns.spec lists every knob. Where an operator has no knob for something, the choice is an unsupported override that Red Hat support will ask you to remove, or a support case requesting the feature.
The control plane in OpenShift
An OpenShift control plane is the Kubernetes control plane from Post 2 with three decisions made for you and a second API server bolted on.
Three control-plane nodes, always. A standard cluster has exactly three control-plane nodes (labelled both master and control-plane for compatibility), and etcd runs co-located on them as static pods in openshift-etcd. The installer creates three and the etcd operator expects a three-member quorum; you cannot run two, and five is not a supported topology. Losing one node leaves a working cluster on two members; losing two takes etcd below quorum and the API stops accepting writes, which is why "how do you recover etcd" is a favourite Post 20 question.
Static pods managed by operators. etcd, kube-apiserver, kube-controller-manager and kube-scheduler run as static pods (defined by files on the node's disk and run directly by the kubelet, so they do not depend on the API server). Each has its own operator (openshift-etcd-operator, openshift-kube-apiserver-operator and so on) that writes the static pod manifests, rotates certificates, rolls configuration changes one node at a time and reports a ClusterOperator. kube-apiserver PROGRESSING True with NodeInstallerProgressing: 1 node is at revision 47; 2 nodes are at revision 48 is that operator rolling a new revision across the three masters.
Two API servers. OpenShift's extra API groups (route.openshift.io, project.openshift.io, build.openshift.io, image.openshift.io, template.openshift.io, security.openshift.io for SCCs, and a few more) are served by a separate openshift-apiserver that plugs into kube-apiserver through the standard API aggregation layer, rather than by patching kube-apiserver. A small oauth-apiserver serves user.openshift.io and oauth.openshift.io. As a client you never notice: everything hits api.<cluster>:6443 and kube-apiserver proxies it. As an operator it matters: if openshift-apiserver is down, oc get pods works but oc get routes, oc get projects and oc new-project fail, and that pattern is a diagnostic clue.
Two controller managers. kube-controller-manager runs the upstream controllers; openshift-controller-manager runs the OpenShift-only ones (builds, image stream imports, the default RoleBindings in new projects, ServiceAccount pull secrets for the internal registry). Since 4.12 a small route-controller-manager owns the Route controllers, including the one that turns Ingress objects into Routes.
The openshift-* namespaces
A fresh cluster has roughly sixty openshift- namespaces plus kube-system, kube-public and a mostly empty default. You must be able to say what lives in the important ones, because "which namespace do you look in when X is broken?" separates people who have run the platform from people who have read about it.
| Namespace | What lives there | When you go there |
|---|---|---|
openshift-cluster-version | The CVO deployment; the cluster-scoped ClusterVersion object's home. | Upgrades stuck or refusing to start; oc logs -n openshift-cluster-version deploy/cluster-version-operator. |
openshift-etcd | The three etcd static pods (etcd-<master>) and backup/defrag helpers; operator in openshift-etcd-operator. | Quorum loss, slow API, disk latency alerts; oc rsh into an etcd pod for etcdctl endpoint status. |
openshift-kube-apiserver | The kube-apiserver static pods and their installer/pruner pods; operator in openshift-kube-apiserver-operator. | API errors, cert problems, a stuck revision rollout. |
openshift-apiserver | The aggregated OpenShift API server (Routes, Projects, Builds, ImageStreams, Templates, SCCs). | OpenShift calls fail while plain Kubernetes calls work. |
openshift-machine-config-operator | The MCO, the Machine Config Server (serves Ignition), the Machine Config Controller, and one machine-config-daemon pod per node. | A Degraded MCP, a node refusing a MachineConfig, an upgrade stuck on a pool; read that node's MCD logs. |
openshift-machine-api | The Machine API controllers: Machine, MachineSet, MachineHealthCheck, ControlPlaneMachineSet; the cloud/vSphere actuators. | MachineSet scaling does nothing, a Machine stuck Provisioning, cloud credential failures. |
openshift-ovn-kubernetes | The OVN-Kubernetes CNI: ovnkube-control-plane pods and one ovnkube-node pod per node. Config is network.config/cluster; operator in openshift-network-operator. | Pod-to-pod or pod-to-service failures, odd NetworkPolicy behaviour, a node whose pods have no network. |
openshift-dns | The dns-default (CoreDNS) and node-resolver DaemonSets; operator in openshift-dns-operator. | Name resolution failures or slow DNS inside pods. |
openshift-ingress | The router-default Deployment: HAProxy pods that implement Routes. The IngressController config lives in openshift-ingress-operator. | Routes returning 503, *.apps certificate problems, adding a router shard. |
openshift-authentication | The oauth-openshift pods (the integrated OAuth server); operator in openshift-authentication-operator; the OAuth object named cluster is cluster-scoped. | Nobody can log in, an identity provider is misconfigured. |
openshift-image-registry | The integrated registry pods and the cluster-image-registry-operator; config is configs.imageregistry.operator.openshift.io/cluster. | Builds cannot push, internal pulls fail, registry storage full or unreachable. |
openshift-monitoring | The platform Prometheus pair, Alertmanager, Thanos Querier, kube-state-metrics, node-exporter and the cluster-monitoring-operator; config is the cluster-monitoring-config ConfigMap. Sibling: openshift-user-workload-monitoring. | Alerts not firing, Prometheus out of disk, moving monitoring to infra nodes. |
openshift-operator-lifecycle-manager | OLM itself: the olm-operator, catalog-operator and packageserver. | A stuck Subscription or InstallPlan. Post 23. |
openshift-marketplace | The CatalogSource pods (redhat-operators, certified-operators, community-operators, redhat-marketplace) that OperatorHub reads from. | OperatorHub empty, catalogs failing to pull, pointing at a mirrored catalog. |
openshift-console | The web console pods and the downloads pod that serves the oc binary; operator in openshift-console-operator. | Console down while oc works; console customisation. |
Two more: openshift-config holds cluster-wide Secrets and ConfigMaps (identity provider credentials, custom CA bundles, the pull secret), and openshift-gitops appears once the Argo CD operator is installed, which at a bank it will be.
Node roles and cluster topology
Every node carries a role label of the form node-role.kubernetes.io/<role>="", and the role decides its MachineConfigPool, which workloads may land on it, and (for one role in particular) how much you pay Red Hat.
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-prod-ctrl-0.bank.example.com Ready control-plane,master 88d v1.31.9+f2ed37e
ocp-prod-ctrl-1.bank.example.com Ready control-plane,master 88d v1.31.9+f2ed37e
ocp-prod-ctrl-2.bank.example.com Ready control-plane,master 88d v1.31.9+f2ed37e
ocp-prod-infra-0.bank.example.com Ready infra,worker 60d v1.31.9+f2ed37e
ocp-prod-infra-1.bank.example.com Ready infra,worker 60d v1.31.9+f2ed37e
ocp-prod-infra-2.bank.example.com Ready infra,worker 60d v1.31.9+f2ed37e
ocp-prod-worker-0.bank.example.com Ready worker 88d v1.31.9+f2ed37e
ocp-prod-worker-1.bank.example.com Ready worker 88d v1.31.9+f2ed37e
ocp-prod-worker-2.bank.example.com Ready worker 88d v1.31.9+f2ed37e
ocp-prod-worker-3.bank.example.com Ready worker 88d v1.31.9+f2ed37e
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
infra rendered-infra-7b2c4f0e9a1d3c5b8e6f0a2d4c6e8b1a True False False 3 3 3 0 60d
master rendered-master-1a9c3e5b7d0f2a4c6e8b0d2f4a6c8e1b True False False 3 3 3 0 88d
worker rendered-worker-4e6a8c0b2d4f6a8c0e2b4d6f8a0c2e4b True False False 4 4 4 0 88d- Control-plane (master) nodes run etcd and the API servers, carry the taint
node-role.kubernetes.io/master:NoScheduleby default so ordinary workloads never land on them, and belong to themasterpool. You do not scale them with a MachineSet; since 4.12 theControlPlaneMachineSetmanages their lifecycle and can replace a failed master automatically on supported platforms. - Worker nodes run your applications, belong to the
workerpool, are created and scaled through MachineSets, and are what your OpenShift subscription is counted on. - Infra nodes are worker nodes dedicated to platform services: the routers, the image registry, monitoring, logging, and often the Argo CD and service-mesh control planes. Red Hat's subscription terms exclude cores running only these Red Hat-provided components, which at bank scale is real money; infra nodes also isolate platform services from noisy application workloads and let you size them differently (routers want network, Prometheus and Loki want fast disk and memory). You create the role yourself: a MachineSet whose nodes get
node-role.kubernetes.io/infra=""(keepingworkertoo, by convention, so they still take worker MachineConfigs) plus a taint such asnode-role.kubernetes.io/infra:NoSchedule, then node placement on each platform operator:spec.nodePlacementon the IngressController,spec.nodeSelectoron the image registry config, and node selectors and tolerations in thecluster-monitoring-configConfigMap. A custominfraMachineConfigPool, as in the output above, lets you roll OS updates to infra nodes separately from the general worker pool.
The Machine API, briefly
The Machine API (namespace openshift-machine-api) manages nodes as Kubernetes objects: a Machine = one virtual or physical host the cluster can create and delete through a cloud or vSphere provider; a MachineSet = a template plus a replica count, the ReplicaSet of machines; a MachineHealthCheck = a policy that replaces a Machine whose node has been NotReady for too long. Scaling a worker pool is oc scale machineset ocp-prod-worker-us-east-1a -n openshift-machine-api --replicas=5; a few minutes later a new RHCOS VM has fetched its Ignition config and joined the cluster with its CSR auto-approved. Machine API answers "how many nodes, on what infrastructure"; the MCO's MachineConfigPool answers "what is on each node's disk". On UPI bare metal without a provider the Machine API has nothing to drive, and you add nodes by booting RHCOS with an Ignition URL yourself. Post 20 goes deep on both.
Topologies: standard, compact, single-node, hosted
Standard is what you saw above: 3 control-plane nodes, 2 or more workers (a real environment has many more), optionally 3 infra nodes. A compact cluster is 3 nodes that are both control plane and worker, made by setting the masters schedulable (oc edit scheduler cluster, set mastersSchedulable: true); it is supported and common at the edge or in small non-production environments, but not for bank production, because a busy application competes with etcd for disk and CPU. Single-node OpenShift (SNO) is one node that is everything, with no high availability, for edge sites and labs; OpenShift Local (CRC) is effectively an SNO in a VM on your laptop. Hosted control planes (HyperShift) flip the model: the control plane (etcd, API servers, controllers) runs as ordinary pods inside a management cluster (managed through RHACM's multicluster engine), and only worker nodes exist in the hosted cluster's own infrastructure. Twenty small clusters no longer need sixty control-plane VMs, control planes spin up in minutes, and ROSA with HCP is built on this. The trade is cheaper, faster clusters against a shared blast radius and a management cluster that must be robust; most on-prem bank estates are still standalone clusters with dedicated control planes today.
Installation types and where OpenShift runs
How OpenShift gets installed reveals whether you have done it and whether you separate the cluster from the infrastructure under it.
IPI, UPI, agent-based and assisted
- IPI (Installer-Provisioned Infrastructure): you give
openshift-installa smallinstall-config.yaml(base domain, cluster name, platform credentials, network CIDRs, pull secret, SSH key) and it creates everything: VPCs or vSphere folders, load balancers, DNS records, the bootstrap machine, the three masters and the initial workers, then hands the cluster to the CVO. Because the installer created the infrastructure, the Machine API can manage it afterwards, so scaling isoc scale machineset. Preferred wherever supported: AWS, Azure, GCP, IBM Cloud, vSphere, Nutanix, OpenStack, bare metal with BMC access. - UPI (User-Provisioned Infrastructure): you create the load balancers, DNS, machines and networking yourself (usually with Terraform or the bank's existing VM automation), the installer generates Ignition configs, and you boot RHCOS with them. You use UPI where corporate rules forbid a tool creating infrastructure (very common in banks, where network and VM provisioning belong to other teams) or where there is no IPI support. The cost is weaker day-2 automation: without a provider, adding a node is a manual boot, not a MachineSet scale.
- Agent-based installer (GA since 4.12): you generate a bootable ISO containing the Assisted Installer's agent, boot your bare-metal or vSphere machines from it, and they discover each other and install the cluster with no external provisioning service. It is the modern answer for disconnected on-prem installs and for compact and single-node clusters, and the Machine API can still manage bare-metal hosts afterwards.
- Assisted Installer: the same idea delivered as a SaaS from
console.redhat.com, with a web UI that validates hosts before installing. Handy when connected; not what an air-gapped bank uses for production.
The installer is a single Go binary. Name openshift-install create install-config (the interactive wizard that writes the YAML) and openshift-install create cluster --dir ./ocp-prod; between them, openshift-install create manifests lets you drop in custom manifests (MachineConfigs, a network config, infra MachineSets) before anything exists. A minimal install-config.yaml, so the fields are familiar:
apiVersion: v1
baseDomain: bank.example.com
metadata:
name: ocp-prod
controlPlane:
name: master
replicas: 3
compute:
- name: worker
replicas: 4
networking:
networkType: OVNKubernetes
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
serviceNetwork:
- 172.30.0.0/16
machineNetwork:
- cidr: 10.20.0.0/22
platform:
vsphere:
apiVIPs: [10.20.0.10]
ingressVIPs: [10.20.0.11]
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'Every cluster exposes two endpoints whatever the platform: api.ocp-prod.bank.example.com:6443 (the Kubernetes API) and the wildcard *.apps.ocp-prod.bank.example.com (every Route). On vSphere and bare metal those are virtual IPs kept alive by keepalived on the masters and ingress nodes; on the clouds they are load balancers. When the bank's network team asks what DNS and load balancer entries you need, those two names plus the machine network are the answer.
Managed versus self-managed, and OKD versus OCP
Self-managed OCP is what this post describes: you install and operate it, on your infrastructure or in your cloud account, and Red Hat supports the software. The managed offerings, ROSA (Red Hat OpenShift Service on AWS, jointly supported with AWS, now primarily the hosted-control-plane flavour), ARO (Azure Red Hat OpenShift, jointly with Microsoft) and OpenShift Dedicated (Red Hat-operated on AWS or GCP), give you the same oc and the same objects but no access to the control plane, some MachineConfigs off-limits, and upgrades run by Red Hat SREs on a schedule you influence rather than control; a bank with an AWS footprint will often weigh ROSA against EKS (Post 27) for cloud workloads, so be ready to compare them. OKD is the community distribution: the same source, built on CentOS Stream CoreOS instead of RHCOS, with no subscription and no support, fine for learning and not something a regulated bank runs in production, for the same reason it does not run CentOS Stream on its core banking hosts.
What "large-scale environment" means at a bank
"Large Red Hat OpenShift environments" in the job description looks, concretely, like this, and you should be able to describe something similar as a reference architecture:
- Many clusters per environment, not one big one. Separate clusters for dev, UAT/SIT, pre-prod and production, often several production clusters split by data centre, by regulatory boundary (payments and card systems may need their own) or by business line. Twenty to fifty clusters is normal; a few hundred nodes in the largest production clusters is normal.
- On-prem first, cloud alongside. Production is usually on vSphere in two data centres (active/active at the application layer, each cluster confined to one site because stretching etcd across sites is a bad idea), with bare-metal clusters for latency-sensitive or GPU workloads, and ROSA or self-managed OCP on AWS for cloud-native workloads. Everything on-prem is disconnected or proxied, so images and Operator catalogs are mirrored into an internal registry (Quay or Artifactory) with
oc-mirror, and upgrades use a mirrored OpenShift Update Service. - Fleet management with RHACM. Red Hat Advanced Cluster Management (RHACM) = a hub cluster that imports every other cluster, shows fleet-wide health, pushes governance policies (required SCC settings, forbidden namespaces, certificate expiry checks, compliance reports), deploys applications across clusters via Argo CD ApplicationSets, and hosts the multicluster engine for hosted control planes. "How do you keep fifty clusters consistent?" expects RHACM policies plus GitOps.
- Shared services around the clusters. A central identity provider (Active Directory via LDAP or an OIDC broker), an enterprise registry with image scanning, a central logging destination (Splunk or a Loki cluster) fed by each cluster's log forwarder, central Prometheus/Thanos or a vendor APM receiving remote-write, HashiCorp Vault or a cloud KMS for secrets, and a change-management system that every upgrade goes through.
Practise saying that in ninety seconds; the interviewer is checking whether you know what such an estate looks like, not whether you know theirs.
What OpenShift adds for the people using the cluster
Everything so far was the platform's own machinery. This section is what an application team sees the first day they log in. Take each item as "Kubernetes gives you X; OpenShift adds Y because Z."
oc versus kubectl
oc = the OpenShift command-line client, a superset of kubectl: it embeds the same kubectl code, so every kubectl verb works unchanged, and it adds verbs for the OpenShift-only objects and workflows:
oc login: authenticates against the integrated OAuth server (username/password or a token) and writes your kubeconfig context for you;kubectlassumes someone handed you a kubeconfig.oc project,oc projects,oc new-project: switch to, list, and request Projects.oc new-app: from a Git URL, an image name or a template, generate and create all the objects an application needs in one command.oc status: a human-readable summary of what is deployed in the current project and how it is wired together.oc rsh(remote shell into a pod, likekubectl exec -it … -- shbut shorter),oc rsync(copy directories in and out of a pod),oc port-forward(same as kubectl).oc debug: a temporary copy of a pod, a Deployment's pod or a node with the entrypoint replaced by a shell, which is how you inspect a CrashLoopBackOff container or a node's host filesystem safely.oc adm: the administrator toolbox:oc adm upgrade,oc adm top nodes,oc adm drain,oc adm must-gather,oc adm policy add-scc-to-user,oc adm policy add-role-to-user,oc adm groups sync,oc adm release info,oc adm node-logs,oc adm inspect.oc explain,oc api-resources: identical to kubectl but aware of the OpenShift API groups, sooc explain route.spec.tlsworks.
A first session on a new cluster:
$ oc login https://api.ocp-dev.bank.example.com:6443 -u devuser
Console URL: https://console-openshift-console.apps.ocp-dev.bank.example.com
Authentication required for https://api.ocp-dev.bank.example.com:6443 (openshift)
Username: devuser
Password:
Login successful.
You have access to the following projects and can switch between them with 'oc project <projectname>':
* payments-dev
shared-tools
Using project "payments-dev".
$ oc whoami
devuser
$ oc whoami --show-console
https://console-openshift-console.apps.ocp-dev.bank.example.com
$ oc whoami -t
sha256~Zk8vQ2s1... # the bearer token oc is using; also what the console's "Copy login command" gives youProjects versus Namespaces
A Project = a Kubernetes Namespace plus a few OpenShift annotations plus a self-service request workflow. Every Project is literally a Namespace; oc get project payments-dev and oc get namespace payments-dev show the same thing from two API groups. The differences are practical:
- Annotations. A Project carries
openshift.io/display-name,openshift.io/description,openshift.io/requester(who asked for it, gold for audit), and the SCC-relatedopenshift.io/sa.scc.uid-range,openshift.io/sa.scc.supplemental-groupsandopenshift.io/sa.scc.mcs, which give every namespace its own block of allowed UIDs and an SELinux label. Those last three are why a container in one project cannot run as the same UID as a container in another, a security property OpenShift gives you for free. - The request flow. Creating a Namespace needs cluster-level permission. OpenShift adds a
ProjectRequestAPI: any authenticated user holding theself-provisionerClusterRole (bound by default tosystem:authenticated:oauth) can runoc new-project, openshift-apiserver instantiates a project template, and the default template creates the namespace and binds the requester asadmin. Banks customise that template (oc adm create-bootstrap-project-template, edit, then reference it inprojects.config.openshift.io/cluster) so every new project also gets a ResourceQuota, a LimitRange, a default-deny NetworkPolicy and the bank's required labels; many also remove the self-provisioner binding entirely and create projects only through a ticket-driven GitOps pipeline. Post 26 is about exactly this.
$ oc new-project payments-demo --display-name="Payments demo" --description="Sandbox for the payments onboarding demo"
Now using project "payments-demo" on server "https://api.ocp-dev.bank.example.com:6443".
$ oc get project payments-demo -o jsonpath='{.metadata.annotations}' | tr ',' '\n'
{"openshift.io/description":"Sandbox for the payments onboarding demo"
"openshift.io/display-name":"Payments demo"
"openshift.io/requester":"devuser"
"openshift.io/sa.scc.mcs":"s0:c27,c14"
"openshift.io/sa.scc.supplemental-groups":"1000730000/10000"
"openshift.io/sa.scc.uid-range":"1000730000/10000"}oc new-project and oc create namespace?" Both produce a namespace, but oc create namespace needs permission to create Namespaces directly (normally cluster-admin), skips the project template entirely (so no quota, no default NetworkPolicy, none of the bank's guardrails), and records no requester. In a regulated environment ordinary users should not have that second path, and you find out whether they do with oc adm policy who-can create namespaces. Bonus points: the SCC UID-range annotations are added to any namespace, however it was created, by openshift-controller-manager, so they are not the difference.Routes versus Ingress
A Route = OpenShift's original object for exposing a Service to the outside world by hostname. Routes predate the Kubernetes Ingress resource (OpenShift 3 had them in 2015, before Ingress went beta), which is why they exist at all. A Route names a host, a target Service and port, an optional path, and a TLS mode: edge (the router terminates TLS and talks plain HTTP to the pod), passthrough (the router forwards the encrypted stream untouched and the pod terminates TLS), or re-encrypt (the router terminates TLS and opens a new TLS connection to the pod, which most bank security teams require for anything sensitive). The router is an HAProxy deployment in openshift-ingress, managed by the ingress operator, that watches Routes and reloads its config. With no host given you get <route>-<project>.apps.<cluster>.<domain> under the wildcard DNS record the installer created.
Ingress objects work too: a controller in openshift-route-controller-manager generates an equivalent Route (with an ownerReference back to the Ingress) for each one, so a Helm chart written for vanilla Kubernetes gets a working hostname on OpenShift with no changes. Teams that live on OpenShift use Routes directly because they are simpler, expose TLS termination modes as first-class fields, and are what the console and oc expose produce. Post 21 covers router sharding, custom domains and certificates.
The integrated OAuth server and identity providers
Upstream Kubernetes has no users; it trusts whatever certificate, token or OIDC assertion you present and leaves the login experience to you. OpenShift ships an OAuth server (pods in openshift-authentication) as the front door: oc login and the console both redirect to it, it authenticates you against one or more configured identity providers (HTPasswd for break-glass accounts, LDAP for Active Directory, OpenID Connect for Entra ID, Okta or Keycloak, plus GitHub, GitLab, Google and a request-header option), and it hands back an OpenShift bearer token that kube-apiserver accepts. Successful logins create User and Identity objects, and Group objects can be synced from LDAP with oc adm groups sync so RBAC bindings can reference AD groups. The one-time kubeadmin user the installer prints should be deleted once a real identity provider works. Post 22 walks through configuring this.
The integrated image registry and ImageStreams
Every OpenShift cluster can run its own internal image registry (openshift-image-registry), reachable in-cluster at image-registry.openshift-image-registry.svc:5000 and optionally through a Route; builds push there by default and every ServiceAccount gets a pull secret for it automatically. On top of the registry sits the ImageStream = a named, versioned pointer to images, wherever they live. A tag such as payments-api:prod resolves to a specific digest; you can point it at an external registry (oc import-image), have it re-check periodically, and trigger rebuilds or rollouts when it changes. At a bank the internal registry is usually deprioritised in favour of a central Quay or Artifactory with scanning and signing, but ImageStreams still appear in oc new-app output and in older deployments, so you need to recognise them.
BuildConfigs, Source-to-Image, Templates and DeploymentConfigs
OpenShift can build images inside the cluster. A BuildConfig describes how: from a Dockerfile in a Git repo (the Docker strategy), or with Source-to-Image (S2I), where a builder image such as nodejs:20-ubi9 or openjdk-17 takes your source code, runs the language's build inside itself, and produces a runnable image without a Dockerfile. oc new-app https://github.com/… inspects the repository, picks a builder image, and creates the ImageStream, BuildConfig, Deployment and Service in one go. It is superb for onboarding developers who have never written a Dockerfile. At scale banks mostly build in CI (Jenkins, GitHub Actions or Tekton pipelines) and push to the enterprise registry, and Red Hat's newer Builds for OpenShift (based on Shipwright) is the direction of travel, so know S2I, be able to demo it, and be honest that production pipelines usually live elsewhere.
Templates are OpenShift's original parameterised-manifest format: a YAML file listing objects with ${PARAMETER} placeholders, processed with oc process or instantiated from the console's catalog. The openshift namespace ships sample ones, but Helm charts and Operators have replaced them for anything new. DeploymentConfigs (DC) were OpenShift's pre-Deployment rollout object, with image-change triggers and lifecycle hooks, deprecated in OpenShift 4.14; oc new-app has created a plain Kubernetes Deployment by default for years, and existing DCs should be migrated. If asked "Deployment or DeploymentConfig?", say "Deployment, DCs are deprecated since 4.14", and add that DCs are OpenShift-only, do not work with standard tooling, and their triggers are better served by GitOps.
The web console
The web console (pods in openshift-console, URL console-openshift-console.apps.…) is a full-featured UI, not a bolt-on dashboard. Historically it had two perspectives: Administrator (cluster health, operators, nodes, MachineConfigPools, monitoring, OperatorHub, every object) and Developer (a topology view of a project's apps, one-click "add from Git or from a catalog", builds, pipelines, Helm releases). Recent releases fold the Developer features into a single unified console; in 4.19 the separate Developer perspective is off by default and an administrator can re-enable it from the Console operator config. For the interview, the console is where an application team can be self-sufficient and where you, as platform, expose OperatorHub, dashboards and a "Copy login command" that gives them a token for oc.
Putting it together: one app, from zero to a URL
$ oc new-app https://github.com/sclorg/nodejs-ex --name=hello
--> Found image 9a1c3b2 (3 weeks old) in image stream "openshift/nodejs" under tag "20-ubi9" for "nodejs"
Node.js 20
----------
Node.js 20 available as container is a base platform for building and running various Node.js 20 applications...
* The source repository appears to match: nodejs
* A source build using source code from https://github.com/sclorg/nodejs-ex will be created
* The resulting image will be pushed to image stream tag "hello:latest"
--> Creating resources ...
imagestream.image.openshift.io "hello" created
buildconfig.build.openshift.io "hello" created
deployment.apps "hello" created
service "hello" created
--> Success
Build scheduled, use 'oc logs -f buildconfig/hello' to track its progress.
Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
'oc expose service/hello'
Run 'oc status' to view your app.Follow the build, expose the Service, and look at what you got:
$ oc get builds
NAME TYPE FROM STATUS STARTED DURATION
hello-1 Source Git@a3f9c1d Complete 2 minutes ago 58s
$ oc expose service/hello
route.route.openshift.io/hello exposed
$ oc get route hello
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
hello hello-payments-demo.apps.ocp-dev.bank.example.com hello 8080-tcp None
$ curl -s http://hello-payments-demo.apps.ocp-dev.bank.example.com | grep -o '<title>.*</title>'
<title>Welcome to your Node.js application on OpenShift</title>
$ oc status
In project Payments demo (payments-demo) on server https://api.ocp-dev.bank.example.com:6443
http://hello-payments-demo.apps.ocp-dev.bank.example.com to pod port 8080-tcp (svc/hello)
deployment/hello deploys istag/hello:latest <-
bc/hello source builds https://github.com/sclorg/nodejs-ex on openshift/nodejs:20-ubi9
deployment #1 running for 3 minutes - 1 podAn empty TERMINATION column means plain HTTP. oc create route edge hello-tls --service=hello gives a TLS route using the router's wildcard certificate, and oc create route reencrypt is what you reach for when the pod itself speaks TLS. Every object created above is a normal Kubernetes object except the ImageStream, the BuildConfig and the Route, which is a neat summary of this whole section.
crc setup, crc start, then eval $(crc oc-env)) or sign up for the free Red Hat Developer Sandbox. On CRC, log in as kubeadmin and run oc get clusterversion, oc get co, oc get nodes and oc get ns | grep -c openshift-; then oc debug node/$(oc get nodes -o name | head -1 | cut -d/ -f2), chroot /host and run rpm-ostree status and crictl ps | wc -l. On either environment, run the oc new-project / oc new-app / oc expose sequence above with your own project name, follow the build with oc logs -f bc/hello, then oc get all to see which objects have OpenShift API groups and which are plain Kubernetes. Finally, apply a vanilla Ingress for the hello Service and run oc get route again to watch the generated Route appear. (The Sandbox does not let you list ClusterOperators or debug nodes, since you are not cluster-admin there; that limitation is itself worth noticing.)Networking: the map
Four facts give you the shape of OpenShift networking; Post 21 fills in the detail.
- OVN-Kubernetes is the CNI. Default for new clusters since 4.12 and the only option since 4.17, when the older OpenShift SDN plugin was removed. It implements pod networking, Services (it replaces kube-proxy, so do not go looking for kube-proxy pods), NetworkPolicy, egress IPs and egress firewalls, using Open Virtual Network with a per-node
ovnkube-nodepod inopenshift-ovn-kubernetes. It supports multiple networks per pod through Multus, which is how banks give a pod a second interface on a specific VLAN. - The CIDRs are set at install time and are effectively permanent. The cluster (pod) network, service network and machine network from
install-config.yamlmust not overlap with each other or with anything the cluster needs to reach on the corporate network. The defaults are10.128.0.0/14for pods with a/23per node (510 pod IPs per node, 1,024 nodes maximum) and172.30.0.0/16for services. Changing them after install is documented but painful, so at a bank every cluster's CIDRs are allocated from a central IPAM before installation.oc get network.config/cluster -o yamlshows what your cluster has. - Ingress is the router. The ingress operator (
openshift-ingress-operator) manages one or more IngressControllers; each becomes a set of HAProxyrouter-*pods inopenshift-ingress, usually placed on infra nodes, published either through a cloud load balancer or by binding host ports on the infra nodes behind an external load balancer or VIP. The default IngressController serves*.apps.<cluster>.<domain>; banks add sharded IngressControllers for internal-only or partner-facing domains with different certificates and different network paths. - DNS is an operator too. The DNS operator runs CoreDNS as the
dns-defaultDaemonSet inopenshift-dns, with anode-resolverDaemonSet that lets nodes resolve the internal registry name, and theDNSobject nameddefaultis where you add forwarders for corporate zones.
Security: the map
The single OpenShift security concept that surprises Kubernetes people most is the Security Context Constraint (SCC) = a cluster-scoped OpenShift object that defines what a pod is allowed to request: which UIDs, which SELinux contexts, which capabilities, which volume types, host network or PID access, privileged mode. Every pod is admitted against the SCCs its ServiceAccount (or user) is allowed to use, and the most restrictive matching SCC that satisfies the pod is applied and recorded in the annotation openshift.io/scc. The default for every workload since 4.11 is restricted-v2: no privileged containers, no host access, all capabilities dropped except NET_BIND_SERVICE, seccomp profile runtime/default, and, most importantly, the container must run as a non-root, arbitrary UID from the project's assigned range rather than whatever UID the image asks for.
$ oc get scc -o custom-columns=NAME:.metadata.name,PRIV:.allowPrivilegedContainer,RUNASUSER:.runAsUser.type,CAPS:.allowedCapabilities,PRIORITY:.priority
NAME PRIV RUNASUSER CAPS PRIORITY
anyuid false RunAsAny <none> 10
hostnetwork false MustRunAsRange <none> <none>
nonroot-v2 false MustRunAsNonRoot [NET_BIND_SERVICE] <none>
privileged true RunAsAny [*] <none>
restricted-v2 false MustRunAsRange [NET_BIND_SERVICE] <none>
$ oc get pod hello-7d9c6b5f4-x2kqp -o jsonpath='{.metadata.annotations.openshift\.io/scc}{" uid="}{.spec.containers[0].securityContext.runAsUser}{"\n"}'
restricted-v2 uid=1000730000That uid=1000730000 came from the project's openshift.io/sa.scc.uid-range annotation you saw earlier, not from the image. This is why "the image works in Docker and on EKS but crashes on OpenShift" is the most common onboarding ticket a platform team receives: an image built to run as root (the official nginx, many vendor images, anything that writes to a root-owned path or binds port 80 without the capability) fails with permission denied or exits immediately. The fix hierarchy: rebuild the image to be UID-agnostic (write to a directory that is group-writable by GID 0, listen on a port above 1024, or use the vendor's unprivileged variant); if that is impossible, grant a specific ServiceAccount a more permissive SCC such as nonroot-v2 or, as a last resort with a security exception on file, anyuid via oc adm policy add-scc-to-user anyuid -z <sa> -n <project>. Grant SCCs to ServiceAccounts, never to users or groups, and never privileged to an application. OpenShift also syncs the upstream Pod Security Admission labels onto namespaces based on the SCCs in use, so PSA warnings and SCC enforcement coexist. Post 22 goes deep, including RBAC differences and the OAuth setup.
anyuid SCC. What do you do?" First confirm the failure really is UID-related (oc describe pod, oc logs, look for permission denied on a path or a port), then push back toward fixing the image, then, if a grant is unavoidable, grant the narrowest SCC to the specific ServiceAccount with a documented exception and expiry. A candidate who just says "add-scc-to-user anyuid" has told the bank they will hand out root on request; one who says "SCC grants go through security review and are tracked in Git" has told them the opposite.oc create deployment nginx-root --image=docker.io/library/nginx:1.27 and watch it fail: oc get pods shows CrashLoopBackOff and oc logs deploy/nginx-root shows nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied). Then run oc create deployment nginx-ok --image=docker.io/nginxinc/nginx-unprivileged:1.27, which listens on 8080 and writes to paths it owns, and watch it start. Compare oc get pod -o yaml for both and find the openshift.io/scc annotation and the injected runAsUser. Being able to narrate that ten-minute experiment is worth more in the interview than memorising every SCC field.Monitoring and logging: the map
OpenShift ships a complete monitoring stack that you did not have to install and must not modify by hand. The cluster-monitoring-operator in openshift-monitoring runs a highly available Prometheus pair, Alertmanager, Thanos Querier (one query endpoint across both Prometheus replicas), kube-state-metrics, node-exporter, and a metrics server for oc adm top, pre-loaded with hundreds of platform alerting rules (etcd latency, ClusterOperator degraded, node not ready, certificate expiry, pods crash-looping) that the console's Observe section reads from. Configuration is a single ConfigMap, cluster-monitoring-config, where you set retention, persistent storage, node placement on infra nodes, and enableUserWorkload: true, which starts a second, isolated Prometheus in openshift-user-workload-monitoring that application teams target with their own ServiceMonitor and PrometheusRule objects without touching the platform instance. Alertmanager is where you route platform alerts to the bank's paging system.
Logging is an add-on Operator, not part of the base payload: the Red Hat OpenShift Logging operator deploys Vector as the per-node log collector and Loki (as a LokiStack) as the store, with a ClusterLogForwarder object that says which logs (application, infrastructure, audit) go where (the in-cluster Loki, Splunk, Kafka, a syslog endpoint, CloudWatch). The older Elasticsearch/Fluentd/Kibana stack (EFK) is retired; an interviewer who mentions Kibana is describing what they ran three years ago. Audit logs from the API servers and OAuth server are written on the control-plane nodes and are what compliance teams want forwarded. Finally the Insights Operator ships anonymised configuration data to Red Hat, which returns proactive health recommendations in the console and on console.redhat.com. Post 24 covers all of this.
Upgrades: the map
Because the CVO owns the cluster's version, an upgrade is a request rather than a procedure. Every cluster subscribes to a channel (stable-4.18, fast-4.18, candidate-4.18, or eus-4.18 for the even-numbered Extended Update Support releases banks prefer), and the CVO periodically asks the OpenShift Update Service (Red Hat's hosted "Cincinnati" graph, or a mirrored copy in a disconnected estate) which versions are safe to move to from the exact version you are on. The result is an update graph: not every version can go to every other, and Red Hat withdraws edges when it finds a bug, which is why "I'll just jump to the latest" is not a decision you make; you look at what the graph offers.
$ oc adm upgrade
Cluster version is 4.18.14
Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.18 (available channels: candidate-4.18, candidate-4.19, eus-4.18, fast-4.18, fast-4.19, stable-4.18)
Recommended updates:
VERSION IMAGE
4.18.17 quay.io/openshift-release-dev/ocp-release@sha256:9b7e2c...
4.18.16 quay.io/openshift-release-dev/ocp-release@sha256:5d3a8f...
4.18.15 quay.io/openshift-release-dev/ocp-release@sha256:c41e7b...
$ oc adm upgrade --to=4.18.17
Requested update to 4.18.17From there the CVO pulls the new payload, upgrades operators in run-level order (control-plane components first), and hands the new RHCOS image to the MCO, which drains and reboots nodes one pool at a time, respecting each pool's maxUnavailable and every PodDisruptionBudget. A minor-version upgrade (4.18 to 4.19) is the same mechanism but first requires every operator to report Upgradeable=True and an administrator to acknowledge removed Kubernetes APIs. Watching it happen is oc get clusterversion, oc get co (the VERSION column moving) and oc get mcp (UPDATING, UPDATEDMACHINECOUNT). Post 20 covers planning, pausing worker pools, EUS-to-EUS upgrades, etcd backups before you start, and what to do when it sticks.
oc adm upgrade for recommended edges and any Upgradeable=False conditions, check oc get co and oc get mcp are clean, take an etcd backup with /usr/local/bin/cluster-backup.sh on a master, confirm every add-on Operator (logging, GitOps, service mesh, storage) is at a version compatible with the target, pause the worker MachineConfigPools if you need to control when nodes reboot, raise the change ticket, then request the update and watch the three commands above until every operator shows the new version and every pool shows UPDATED. That sequence is the interviewer's checklist too.Kubernetes to OpenShift: the translation table
When you are asked about a Kubernetes concept in an OpenShift interview, answer with the Kubernetes concept and then its OpenShift form. This table is the cheat sheet.
| Kubernetes (upstream or DIY) | OpenShift equivalent | What to say about the difference |
|---|---|---|
| Namespace | Project | Same object underneath; Project adds annotations, a requester, SCC UID ranges, and a templated self-service request flow. |
| Ingress + an ingress controller you chose (nginx, Traefik) | Route + IngressController (HAProxy router) | Routes came first and expose TLS termination modes directly; Ingress objects are translated into Routes; the router is operator-managed and shardable. |
| Deployment | Deployment (DeploymentConfig deprecated in 4.14) | Use Deployments; migrate any remaining DCs. |
| Pod Security Admission (restricted/baseline/privileged) | SecurityContextConstraints, default restricted-v2, with PSA labels synced | SCCs are older, finer-grained, and enforce arbitrary non-root UIDs; grants go to ServiceAccounts. |
| Kubernetes Dashboard (optional, rarely used) | Web console (built in, operator-managed) | Administrator and Developer views, OperatorHub, monitoring, login integration. |
| kubeadm plus your own install scripts | openshift-install (IPI/UPI/agent) plus the CVO | The installer bootstraps; the CVO installs and owns every component afterwards. |
| cert-manager or manual rotation for control-plane certs | Built-in rotation by each operator (service-ca, kube-apiserver operator, etcd operator) | Cluster certificates rotate automatically; cert-manager is still used for application certificates. |
docker build in CI, push to a registry | BuildConfig / S2I in-cluster, or CI plus the enterprise registry | S2I is great for onboarding; production builds usually stay in CI. |
kubectl | oc (superset) | Adds login, project, new-app, rsh, debug, adm. |
| Helm | Helm + OperatorHub (OLM) + Templates | Helm works unchanged; Operators handle stateful, lifecycle-heavy software; Templates are legacy. |
| Node OS patching (Ansible, SSM, golden AMIs) | RHCOS managed by the MCO | OS updates ride along with cluster upgrades; config is declarative MachineConfig; no SSH-and-fix. |
| Your chosen CNI (Calico, Cilium, VPC CNI) | OVN-Kubernetes, operator-managed | One supported CNI, integrated with the network operator; Multus for extra interfaces. |
| Prometheus/Grafana you installed | Cluster monitoring stack, operator-managed, plus user workload monitoring | Platform alerts are pre-built; application metrics go through the user workload instance. |
| OIDC flags on the API server, no users | Integrated OAuth server with identity providers; User, Identity, Group objects | Login is a platform feature; AD groups map straight to RBAC. |
The two thirty-second answers you must have ready
"What's the difference between Kubernetes and OpenShift?"
Say this, in your own words, in one breath: "OpenShift is a Kubernetes distribution from Red Hat. The Kubernetes inside it is unmodified, so every object and every kubectl command works. What Red Hat adds is the whole platform around it, delivered and supported as one product: an immutable operating system, RHCOS, that the cluster manages itself through the Machine Config Operator; CRI-O as the runtime; OVN-Kubernetes networking; an HAProxy router with Routes; an integrated OAuth login with identity providers; an internal registry with builds; a full monitoring stack; a web console; and OperatorHub. All of that is installed and upgraded by operators under a single Cluster Version Operator, so the first thing you type on a sick cluster is oc get co. The trade-off is fewer choices and a subscription, in exchange for an integrated, supported, compliance-friendly stack."
"Why do enterprises pay for OpenShift when Kubernetes is free?"
Six words, each with a sentence behind it. Support: a bank cannot open a GitHub issue during an outage; it needs a vendor with an SLA and a lifecycle (each OpenShift minor is supported for eighteen months, EUS releases for two years, with defined upgrade paths). Integration: the OS, runtime, CNI, ingress, auth, registry and monitoring are tested together for each release, so the platform team is not the integrator. Compliance: FIPS mode, SELinux everywhere, non-root by default through SCCs, audit logging, CIS benchmark and Compliance Operator profiles, and a vendor who publishes CVE fixes and errata, which is what auditors ask for. Lifecycle: one-command upgrades of the entire stack including the OS, driven by a tested update graph. Operators: a certified catalog of databases, messaging, GitOps, service mesh and storage that install and upgrade the same way the platform does. RHCOS: no separate OS patching programme, no configuration drift, no SSH culture. Finish with the honest counterpoint: the price is real, some flexibility is gone, and a team with strong Kubernetes skills and no regulatory pressure might reasonably choose EKS plus open-source tooling instead, which is exactly the comparison this job will ask you to make every week.
Likely interview questions on this post
"Walk me through the OpenShift 4 stack from the hardware up."
RHCOS, an immutable RHEL-based OS provisioned by Ignition and managed by the Machine Config Operator; CRI-O as the runtime, versioned in lockstep with Kubernetes; the upstream control plane and kubelets, with etcd co-located on three control-plane nodes; then the cluster operators (network, DNS, ingress, auth, registry, monitoring, console, machine API, MCO) that manage every platform service, coordinated by the Cluster Version Operator from a single release image. On top: OLM and OperatorHub for add-ons, and user-facing additions such as Projects, Routes, BuildConfigs and the console.
"What does oc get co show you, and what do the columns mean?"
One line per ClusterOperator: VERSION (reached by that operator), AVAILABLE (is the component working), PROGRESSING (is the operator changing something), DEGRADED (a problem needing a human), SINCE and MESSAGE. Healthy is True/False/False at the current version on every line; during an upgrade VERSION advances operator by operator. Available-but-Degraded means working but broken in a way that will bite, and oc describe co <name> gives the full reason plus relatedObjects pointing at the namespace to investigate.
"What is the Cluster Version Operator and what is a release image?"
The CVO is the operator that owns the cluster's version. A release image is a single container image from quay.io/openshift-release-dev/ocp-release bundling the manifests for every platform component and a digest-pinned list of the component images for that exact version. The CVO applies those manifests in run-level order on install and again on upgrade, reports through the ClusterVersion object named version, and fulfils upgrade requests made with oc adm upgrade --to.
"Why can't I just SSH into an RHCOS node and fix a config file?"
Because the Machine Config Operator owns node configuration: it renders MachineConfigs into a per-pool rendered config and applies it through the Machine Config Daemon on each node. The MCD detects drift in files it manages and marks the pool Degraded, which blocks upgrades, and the change would be lost on the next OS image rollout anyway. The right path is a MachineConfig targeted at the pool; oc debug node/ is for inspecting, not changing.
"Project versus Namespace?"
A Project is a Namespace with OpenShift annotations (display name, requester, and the SCC UID range and SELinux label that give each namespace its own identity range) plus a request workflow: self-provisioners run oc new-project, which instantiates the cluster's project template, where a bank injects quotas, limit ranges and default-deny network policies. oc create namespace bypasses the template and needs cluster-level rights, so a regulated cluster reserves it for the platform team or a GitOps pipeline.
"Route versus Ingress?"
Route is OpenShift's older native object for exposing a Service by hostname, implemented by the operator-managed HAProxy router pods, with edge, passthrough and re-encrypt TLS as first-class fields. Ingress objects are supported and translated into Routes by a controller, so vanilla charts work. For a mixed OpenShift and EKS estate, standardise shared charts on Ingress or Gateway API and let the platform's IngressController defaults handle Route-specific TLS.
"What are infra nodes and how do you create them?"
Workers dedicated to Red Hat platform components (router, registry, monitoring, logging) so those cores do not count toward the subscription, platform services are isolated from application load, and they can be sized and upgraded separately. Mechanism: a MachineSet whose nodes carry the node-role.kubernetes.io/infra label and a matching taint, optionally an infra MachineConfigPool, and node placement on each operator's CR (IngressController nodePlacement, registry nodeSelector, cluster-monitoring-config) so the operators move their own pods.
"IPI or UPI, and which did you use?"
IPI lets openshift-install create the infrastructure from an install-config and leaves the Machine API able to scale nodes; UPI has you build load balancers, DNS and machines yourself and boot RHCOS with Ignition, which fits banks where other teams own the infrastructure but weakens day-2 automation. Answer with the one you actually used, why the environment forced it, and one concrete pain point (UPI: keeping the external load balancer and DNS in sync when adding nodes; IPI on vSphere: the privileges the installer's vCenter account needs).
"Describe a large OpenShift environment you'd expect at a bank."
Many clusters rather than one: dev, test, pre-prod and several production clusters split by data centre and regulatory domain, mostly on vSphere or bare metal on-prem in disconnected mode with mirrored registries and a mirrored update service, plus ROSA or self-managed OpenShift on AWS. RHACM is the hub for fleet visibility, policy governance and Argo CD ApplicationSets, with shared services (identity, registry, logging, metrics, Vault, change management) around it. Say it as a picture: hub in the middle, clusters around it, shared services at the edge.
"Why did your last team pick OpenShift over plain Kubernetes, or EKS?"
Give the six reasons from the thirty-second answer (support and lifecycle, integrated tested stack, compliance posture, full-stack upgrades, the certified Operator catalog, RHCOS removing OS patching), then be balanced: name the costs (subscription, fewer choices, the SCC learning curve) and say when you would choose EKS instead (cloud-native teams, no on-prem estate, strong internal Kubernetes skills). A bank that runs both wants someone who can argue either side honestly.
Key Takeaways
- OpenShift is unmodified Kubernetes plus an integrated, supported platform: RHCOS managed by the MCO, CRI-O, OVN-Kubernetes, an HAProxy router, an OAuth server, a registry, a monitoring stack and a console, all installed and upgraded by operators. Kubernetes is the kernel; OpenShift is the distribution.
- The Cluster Version Operator owns the cluster's version and installs everything from a single release image; every platform component reports a ClusterOperator.
oc get cois the first command on any sick cluster, and AVAILABLE, PROGRESSING and DEGRADED each mean something specific. - Never fix an RHCOS node by hand. Configuration is a MachineConfig applied by the MCO; drift makes a pool Degraded and blocks upgrades. Use
oc debug node/to look. - Know what lives where: etcd in
openshift-etcd, the router inopenshift-ingress, OVN inopenshift-ovn-kubernetes, login inopenshift-authentication, MachineConfigs inopenshift-machine-config-operator, OLM inopenshift-operator-lifecycle-manager, Prometheus inopenshift-monitoring. Edit operator config objects (usually namedclusterordefault), never the operands. - Three control-plane nodes with co-located etcd, workers scaled by MachineSets, and infra nodes (labels plus taint plus operator node placement) for routers, registry, monitoring and logging, for licensing, isolation and control.
- Installation is IPI, UPI or agent-based; production at a bank is many clusters, mostly on-prem and disconnected, with RHACM for fleet management and ROSA or EKS alongside in the cloud.
- For users, OpenShift adds
oc(a kubectl superset with login, project, new-app, rsh, debug and adm), Projects with a templated request flow, Routes with TLS modes, integrated login, an internal registry with ImageStreams and S2I builds, and a console. DeploymentConfigs are deprecated; use Deployments. - Security defaults to the
restricted-v2SCC and arbitrary non-root UIDs, which is why root-assuming images fail; fix the image first and grant narrower SCCs to ServiceAccounts only as a documented exception.