OpenShift Networking and Storage, End to End

Pull the ticket queue of any bank's OpenShift platform team and two titles dominate: "my route returns 503" and "my pod can't reach the database". A third, "my PVC is stuck Pending", is not far behind. None of these are hard once you have an ordered method and know which OpenShift-specific object sits at each hop. This post gives you that map: OVN-Kubernetes underneath every pod, the Ingress Operator and its HAProxy routers, Routes and their three TLS modes, cluster DNS, NetworkPolicy and egress control the way a regulated environment actually uses them, and then the storage side: CSI, OpenShift Data Foundation, the platform components that quietly need disks, and the storage failures you will be paged for. By the end you will be able to walk an interviewer from a browser to a pod, and from a pod to a Ceph OSD, naming the command you would run at every step.

Two tickets, one habit

You already know Services, Ingress and PV/PVC from Post 5, Post 10 and Post 7. What changes on OpenShift is not the Kubernetes model. It is that every layer is owned by an Operator with its own custom resource, and every hop has an OpenShift name. "Route returns 503" is a north-south problem (traffic entering the cluster): browser → load balancer → router pod → Service → endpoint. "Pod can't reach the database" is usually an east-west problem (pod to pod) or an egress problem (pod to something outside): pod → OVN overlay → node → firewall → database.

The habit this post builds is the same one from Post 14: name the path, then test each hop from the outside in, and never guess a cause before you have isolated a layer. The interview version is: say the path out loud, then say the commands. Everything below is organised so you can do exactly that.

OVN-Kubernetes: the wires under every pod

Every Kubernetes cluster needs a CNI plugin (Container Network Interface: the component that gives each pod an IP and moves packets between pods). On OpenShift 4.14 and later that plugin is OVN-Kubernetes, and it is the only one Red Hat supports. The older OpenShift SDN was deprecated in 4.14 and removed in 4.17; a cluster still running OpenShift SDN cannot upgrade to 4.17 until it is migrated. If an interviewer asks "which SDN do you run", the answer in 2026 is OVN-Kubernetes, full stop.

OVN-Kubernetes is built from two Open vSwitch projects. OVS (Open vSwitch) is a software switch that runs on every node and actually forwards packets. OVN (Open Virtual Network) sits above it and lets you describe a virtual network (logical switches, logical routers, ACLs, NAT rules) in a database; OVN then programs OVS flows on each node to make that logical network real. OVN-Kubernetes is the controller that watches Kubernetes objects (pods, Services, NetworkPolicies, EgressIPs) and translates them into OVN logical objects.

The pods live in the openshift-ovn-kubernetes namespace. Two kinds matter:

• ovnkube-control-plane: a Deployment (normally two replicas) running the cluster manager. It hands out a pod subnet to each node, allocates egress IPs, and does other cluster-wide bookkeeping. It does not sit in the packet path.
• ovnkube-node: a DaemonSet, one pod per node, with about eight containers: the OVN northbound and southbound databases (nbdb, sbdb), northd, ovn-controller, the ovnkube-controller that watches the API, an ACL audit logger and two metrics proxies. Since 4.14 OpenShift uses OVN's interconnect architecture, which is why every node runs its own OVN databases instead of talking to a central one on the masters. A node's networking keeps working even if it loses the control plane for a while.

$ oc get pods -n openshift-ovn-kubernetes -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP           NODE
ovnkube-control-plane-6d9f8c7b5d-4kx2p   2/2     Running   0          12d   10.0.12.41   master-0
ovnkube-control-plane-6d9f8c7b5d-q7wzn   2/2     Running   0          12d   10.0.12.43   master-2
ovnkube-node-2ns8f                       8/8     Running   0          12d   10.0.12.41   master-0
ovnkube-node-7b6kd                       8/8     Running   1          9d    10.0.14.22   worker-1
ovnkube-node-9pk4c                       8/8     Running   0          12d   10.0.14.23   worker-2
ovnkube-node-m2xzt                       8/8     Running   0          12d   10.0.12.42   master-1
ovnkube-node-w8q3r                       8/8     Running   0          12d   10.0.12.43   master-2

Notice the IP column: these pods carry node IPs (10.0.x.x), not pod IPs. That is because they run with hostNetwork: true, which we will come back to.

Three address spaces you must be able to name

Every OpenShift cluster has three distinct IP ranges, and half of all "can't reach X" tickets come from someone confusing them:

1. Cluster network (pod network): the range pods get IPs from. Default 10.128.0.0/14 with hostPrefix: 23, meaning each node is handed its own /23 (510 usable pod IPs) carved out of the /14. That arithmetic caps a default cluster at 512 nodes, which is why large clusters are sized with a bigger CIDR at install time; you cannot enlarge it afterwards without a disruptive procedure.
2. Service network: the range ClusterIPs come from. Default 172.30.0.0/16. These IPs never appear on a wire; OVN rewrites them to a pod IP on the way out.
3. Machine network (host network): the real subnet your nodes' NICs are on, set as networking.machineNetwork in install-config.yaml. Node IPs, the API VIP, ingress VIP and egress IPs all live here.

$ oc get network.config/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  clusterNetworkMTU: 1400
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16

To see which slice a particular node received, read its OVN annotation:

$ oc get node worker-1 -o jsonpath='{.metadata.annotations.k8s\.ovn\.org/node-subnets}'
{"default":["10.131.0.0/23"]}

So any pod with an IP in 10.131.0.0/23 lives on worker-1. That single fact lets you place a pod on a node from its IP alone, which is handy when reading router logs or firewall logs.

The pod-to-pod path, in words

Each pod has a virtual ethernet pair: one end is eth0 inside the pod, the other is plugged into the OVS bridge br-int on the node. OVN models each node as a logical switch, joins all node switches to one cluster-wide logical router (ovn_cluster_router), and gives each node a gateway router for traffic leaving the cluster. Now the two cases:

• Same node: pod A → br-int → pod B. Pure OVS switching, no encapsulation, no node NIC involved.
• Different nodes: pod A → br-int → the packet is wrapped in a Geneve tunnel header (UDP port 6081) addressed from node 1's IP to node 2's IP → out through the node's physical NIC, which OVN has attached to a second bridge called br-ex → across the machine network → node 2 unwraps it → br-int → pod B. This is the overlay: pod IPs ride inside packets whose outer addresses are node IPs, so the physical network never needs to know about 10.128.0.0/14.
• Pod to Service: OVN load balancers on the node's logical switch rewrite the ClusterIP to one of the endpoint pod IPs before the packet even leaves the node. There is no kube-proxy on OpenShift with OVN-Kubernetes; OVN does that job.

hostNetwork pods

A pod with spec.hostNetwork: true skips the overlay entirely: it shares the node's network namespace, gets the node's IP, and binds ports directly on the node. OVN-Kubernetes itself, the Machine Config Daemon, node exporters, and routers on bare metal (see the HostNetwork publishing strategy below) all run this way. The cost is that two hostNetwork pods on one node cannot listen on the same port, and NetworkPolicy sees their traffic as coming from the node, not from a pod. Only privileged SCCs allow it (Post 22), so an application team asking for hostNetwork is asking for a security exception, not a networking feature.

MTU and overlay overhead

MTU (maximum transmission unit) is the largest packet a link carries. Geneve adds roughly 100 bytes of headers, so on a standard 1500-byte machine network OVN sets the cluster network MTU to 1400 (you saw clusterNetworkMTU: 1400 above). The classic MTU incident is a bare-metal or VMware cluster where someone enabled jumbo frames (9000) on some switches but not all: small requests work, large responses (a file download, a big JSON payload, a TLS certificate chain) silently hang. The tell is "curl to the pod works for small pages and stalls on large ones". Changing the cluster MTU after installation is supported but is a multi-step migration driven by the Cluster Network Operator and the Machine Config Operator that reboots every node; read the current procedure rather than patching it live. Verify what a node believes with oc debug node/<node> -- chroot /host ip link show br-ex.

Ingress in OpenShift: the Ingress Operator and its routers

In vanilla Kubernetes you install an ingress controller yourself. OpenShift ships one and manages it with an Operator. The pieces, from the outside in:

• Ingress Operator, in namespace openshift-ingress-operator. It watches IngressController custom resources and, on cloud platforms, also creates the wildcard DNS record and the load balancer for each one.
• IngressController CR: the description of one router deployment: how many replicas, on which nodes, what domain it serves, how it is published to the outside, which routes it is allowed to admit. A fresh cluster has exactly one, named default.
• Router pods, in namespace openshift-ingress, named router-<ingresscontroller>-xxxx. Each is an HAProxy instance plus a small controller that watches Route objects and rewrites the HAProxy configuration when they change. The default is two replicas.

$ oc get ingresscontroller -n openshift-ingress-operator
NAME      AGE
default   120d

$ oc get pods -n openshift-ingress -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE
router-default-7c9f6d5b4-x2kqm    1/1     Running   0          12d   10.129.4.12   infra-0
router-default-7c9f6d5b4-zl8vn    1/1     Running   0          12d   10.130.6.9    infra-1

$ oc get svc -n openshift-ingress
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                                     PORT(S)
router-default            LoadBalancer   172.30.88.14    internal-a1b2c3.elb.ca-central-1.amazonaws.com  80:31245/TCP,443:30987/TCP
router-internal-default   ClusterIP      172.30.201.55   <none>                                          80/TCP,443/TCP,1936/TCP

The default wildcard domain is *.apps.<cluster-name>.<base-domain>; every Route without an explicit host gets a name under it. Read it from the cluster's Ingress config object rather than guessing:

$ oc get ingresses.config/cluster -o jsonpath='{.spec.domain}{"\n"}'
apps.prod-tor01.bank.internal

Replicas, placement and infra nodes

Banks run routers on dedicated infra nodes: worker machines labelled node-role.kubernetes.io/infra="" and usually tainted so ordinary workloads stay off them. The reasons are licensing (Red Hat does not count infra nodes against OpenShift subscriptions when they only run platform components), isolation (a runaway application cannot starve the routers of CPU), and predictability (the firewall team wants to know exactly which nodes receive inbound traffic). The IngressController carries a nodePlacement block with the selector and the tolerations:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  domain: apps.prod-tor01.bank.internal
  replicas: 3
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
    tolerations:
    - key: node-role.kubernetes.io/infra
      operator: Exists
      effect: NoSchedule
  endpointPublishingStrategy:
    type: LoadBalancerService
    loadBalancer:
      scope: Internal
  defaultCertificate:
    name: wildcard-apps-2026
  tuningOptions:
    clientTimeout: 30s
    serverTimeout: 30s
    threadCount: 4
  logging:
    access:
      destination:
        type: Container

Change replicas with oc patch ingresscontroller default -n openshift-ingress-operator --type=merge -p '{"spec":{"replicas":3}}', never by scaling the router Deployment directly (the Operator will scale it straight back). Everything in this CR is documented under oc explain ingresscontroller.spec.

Replacing the default certificate

The installer generates a self-signed wildcard certificate, which no bank browser trusts. Replacing it is a two-step, low-risk change: create a TLS secret in openshift-ingress containing the corporate wildcard certificate (full chain) and key, then point the IngressController at it.

$ oc create secret tls wildcard-apps-2026 -n openshift-ingress \
    --cert=wildcard-apps-fullchain.crt --key=wildcard-apps.key
secret/wildcard-apps-2026 created

$ oc patch ingresscontroller default -n openshift-ingress-operator --type=merge \
    -p '{"spec":{"defaultCertificate":{"name":"wildcard-apps-2026"}}}'
ingresscontroller.operator.openshift.io/default patched

$ oc get pods -n openshift-ingress -w      # routers roll one at a time

Two things people forget: the API server certificate is a separate object (apiserver/cluster, covered in Post 22), and if the wildcard was signed by an internal CA, that CA must also be added to the cluster's trusted bundle so platform components (the OAuth server, the console, image pulls) can trust it. Certificate expiry of the ingress wildcard is a classic bank incident; put the expiry date in the monitoring system on day one (Post 24).

Router sharding: why a bank runs more than one router

Router sharding means running several IngressControllers, each admitting only a subset of Routes, each published on a different load balancer and (usually) different nodes. A typical bank layout:

• default: internal applications, published on an internal load balancer only reachable from the corporate network.
• dmz: the handful of customer-facing applications, running on routers placed on nodes in a DMZ network segment behind the external firewall and web application firewall.
• pci: card-data workloads whose entry point must be physically and logically separate for PCI DSS scoping.

Each shard gets its own spec.domain and selects its routes with a routeSelector (match Route labels) and/or a namespaceSelector (match namespace labels):

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: dmz
  namespace: openshift-ingress-operator
spec:
  domain: dmz.prod-tor01.bank.com
  replicas: 2
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/dmz: ""
    tolerations:
    - key: node-role.kubernetes.io/dmz
      operator: Exists
      effect: NoSchedule
  routeSelector:
    matchLabels:
      exposure: dmz
  endpointPublishingStrategy:
    type: HostNetwork

A Route labelled exposure: dmz with a host under dmz.prod-tor01.bank.com will now be served by the DMZ routers. Verify which routers admitted a route with oc get route <name> -o jsonpath='{.status.ingress[*].routerName}'.

endpointPublishingStrategy: how traffic reaches the routers

This field answers "what sits in front of the router pods", and it differs by platform:

On a vSphere cluster with an F5 in front, "route works from inside but not outside" is very often the F5 pool pointing at a node that no longer runs a router. That is why you check the strategy first: oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.endpointPublishingStrategy.type}'.

Access logs, timeouts and tuning

By default routers do not log requests. Turn on access logging per IngressController with spec.logging.access.destination.type: Container (a sidecar named logs you read with oc logs -n openshift-ingress deploy/router-default -c logs) or Syslog with an address and port to ship to the SIEM. A bank generally does the latter for the DMZ shard because customer-facing access logs are an audit requirement. A line looks like this; the backend name tells you namespace, route, and which pod served it:

$ oc logs -n openshift-ingress deploy/router-default -c logs --tail=1
2026-09-08T13:41:02.118Z router-default-7c9f6d5b4-x2kqm router[15]: 10.0.14.22:53211 [08/Sep/2026:13:41:02.101] fe_sni~ be_secure:payments-prod:payments-api/pod:payments-api-6d8f9-k2xvb:payments-api:https:10.129.2.17:8443 0/0/1/12/13 200 1834 - - ---- 41/12/0/0/0 0/0 "GET /v1/accounts HTTP/1.1"

Timeouts have two levels. The IngressController's tuningOptions set defaults for every route on that router (clientTimeout, serverTimeout, tunnelTimeout for websockets, threadCount, maxConnections). A single Route can override with the annotation haproxy.router.openshift.io/timeout: 60s. The HAProxy default of 30 seconds is the reason a long-running report endpoint returns 504 from the router while the application log shows the request completing happily at 45 seconds. Also worth knowing: spec.tlsSecurityProfile controls minimum TLS version and ciphers (a bank sets Intermediate or a Custom profile to meet policy), and spec.httpErrorCodePages lets you replace the router's stock 503 and 404 pages with branded ones.

Routes: TLS termination, paths, weights and annotations

A Route is OpenShift's original way to expose a Service on a hostname; it predates the Kubernetes Ingress object and is still the native, feature-complete choice on OpenShift. A Route names a host, a target Service (optionally a port and path), and a TLS mode. The router watches Routes and builds HAProxy frontends and backends from them.

The three TLS termination modes

The question "edge, passthrough or re-encrypt?" is really "who holds the certificate, and is the hop from router to pod encrypted?" Keep this table in your head:

Creating each mode from the command line:

$ oc expose svc/payments-ui                       # plain HTTP route, no TLS
$ oc create route edge payments-ui-tls --service=payments-ui \
    --cert=ui.crt --key=ui.key --ca-cert=corp-ca.crt --insecure-policy=Redirect
$ oc create route passthrough payments-mtls --service=payments-mtls --port=8443
$ oc create route reencrypt payments-api --service=payments-api --port=https \
    --dest-ca-cert=backend-ca.crt --insecure-policy=Redirect

Two shortcuts save a lot of certificate handling. If you omit --cert on an edge or re-encrypt route, the router uses its default wildcard certificate, which is fine for hosts under *.apps. And if the backend pod uses an OpenShift service serving certificate (annotate the Service with service.beta.openshift.io/serving-cert-secret-name: <name> and the platform mints a cert signed by the internal service CA), a re-encrypt route needs no destinationCACertificate at all, because the router already trusts that CA. That is the cleanest re-encrypt pattern to describe in an interview.

A production Route, annotated

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments-api
  namespace: payments-prod
  labels:
    exposure: internal
  annotations:
    haproxy.router.openshift.io/timeout: 60s
    haproxy.router.openshift.io/ip_whitelist: 10.20.0.0/16 10.21.4.0/24
    haproxy.router.openshift.io/rate-limit-connections: "true"
    haproxy.router.openshift.io/rate-limit-connections.rate-http: "200"
    haproxy.router.openshift.io/hsts_header: max-age=31536000;includeSubDomains
spec:
  host: payments-api.apps.prod-tor01.bank.internal
  path: /v1
  to:
    kind: Service
    name: payments-api
    weight: 100
  port:
    targetPort: https
  tls:
    termination: reencrypt
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None

• path: /v1: path-based routing. Several Routes can share a host and split on path; the router picks the longest matching path. Passthrough routes cannot have paths, because the router never sees the HTTP request.
• insecureEdgeTerminationPolicy: Redirect: port 80 answers with a 301 to https. Allow serves both, None (the default) refuses plain HTTP. For passthrough only None and Redirect make sense.
• ip_whitelist: HAProxy drops connections from any source not in the list. Note that with a cloud load balancer in front, the source IP the router sees is the LB's unless proxy protocol is enabled on the IngressController, so test before you rely on it.
• rate-limit-connections: per-source-IP limits enforced by HAProxy stick tables; .rate-http, .rate-tcp and .concurrent-tcp are the knobs.
• wildcardPolicy: Subdomain would make this route answer for *.payments-api.apps.... Wildcard routes are disabled on the router unless spec.routeAdmission.wildcardPolicy: WildcardsAllowed is set on the IngressController.

$ oc get route -n payments-prod
NAME           HOST/PORT                                       PATH   SERVICES       PORT    TERMINATION          WILDCARD
payments-api   payments-api.apps.prod-tor01.bank.internal      /v1    payments-api   https   reencrypt/Redirect   None
payments-ui    payments-ui.apps.prod-tor01.bank.internal              payments-ui    8080    edge/Redirect        None
payments-mtls  payments-mtls.apps.prod-tor01.bank.internal            payments-mtls  8443    passthrough          None

Weighted backends and sticky sessions

A Route can point at up to four Services with weights, which gives you A/B testing or a canary without a mesh. Kubernetes-style traffic splitting only works if both Services have ready endpoints:

$ oc set route-backends payments-ui payments-ui-v1=90 payments-ui-v2=10
route.route.openshift.io/payments-ui backends updated

$ oc set route-backends payments-ui
NAME                 KIND     TO               WEIGHT
routes/payments-ui   Service  payments-ui-v1   90 (90%)
routes/payments-ui   Service  payments-ui-v2   10 (10%)

Under the hood this is spec.to.weight plus spec.alternateBackends[]. For sticky sessions: on edge and re-encrypt routes HAProxy inserts a cookie so a client keeps hitting the same pod; disable it with haproxy.router.openshift.io/disable_cookies: "true" or name it with router.openshift.io/cookie_name. Passthrough routes cannot use cookies, so the router balances by source IP (haproxy.router.openshift.io/balance: source is the default there; for other routes the default algorithm is random, with roundrobin and leastconn available).

Route versus Kubernetes Ingress on OpenShift

Application teams arriving with Helm charts will bring Ingress objects. OpenShift handles them through an ingress-to-route controller (part of openshift-route-controller-manager): for every Ingress that uses the openshift-default IngressClass (or has no class and the cluster default applies) and has a host in its rules, it creates one Route per host/path, owned by the Ingress. A TLS secret referenced by the Ingress is copied into the generated Route's edge configuration; the annotation route.openshift.io/termination: reencrypt requests re-encrypt instead. The generated Routes are named <ingress-name>-<random> and are deleted when the Ingress is. The limits: Ingress objects without a host are ignored, passthrough is not expressible, and most HAProxy annotations must go on the Ingress (they are copied over). The practical advice you would give a team: Ingress is fine for portability, but if you need passthrough, weights, or precise annotations, write a Route.

$ oc get ingress,route -n payments-dev
NAME                                    CLASS               HOSTS                                     ADDRESS   PORTS
ingress.networking.k8s.io/payments-ui   openshift-default   payments-ui.apps.dev-tor01.bank.internal            80, 443

NAME                                         HOST/PORT                                  PATH   SERVICES      PORT   TERMINATION   WILDCARD
route.route.openshift.io/payments-ui-h7k2x   payments-ui.apps.dev-tor01.bank.internal   /      payments-ui   8080   edge          None

Route admission: the same host in two namespaces

By default a router refuses to serve a hostname that is already claimed by a Route in another namespace; the older Route wins and the newer one shows Admitted: False with reason HostAlreadyClaimed. This is a security feature: it stops a team in payments-dev from hijacking payments-api.apps.... It is also the second most common cause of "my new route returns 503": the route exists, the app is fine, but no router admitted it.

$ oc get route payments-api -n payments-dev -o yaml | sed -n '/^status:/,$p'
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2026-08-14T13:02:11Z"
      message: a route in another namespace holds payments-api.apps.prod-tor01.bank.internal
        and is older than payments-api
      reason: HostAlreadyClaimed
      status: "False"
      type: Admitted
    host: payments-api.apps.prod-tor01.bank.internal
    routerName: default

Different paths on the same host across namespaces are also blocked under the default routeAdmission.namespaceOwnership: Strict; InterNamespaceAllowed relaxes it, and almost no bank turns that on.

Services and DNS inside the cluster

Services work exactly as in Post 5; OVN implements them without kube-proxy, and oc get endpoints (or the newer oc get endpointslices) still tells you whether a Service has anyone behind it. DNS is where OpenShift adds structure. The DNS Operator in openshift-dns-operator manages a DNS custom resource named default, which in turn runs CoreDNS as a DaemonSet called dns-default in openshift-dns, one pod per node, fronted by a ClusterIP that is always the tenth address of the service network: 172.30.0.10. A second DaemonSet, node-resolver, writes the internal image registry's hostname into every node's /etc/hosts so the container runtime can pull from it without cluster DNS.

$ oc get pods -n openshift-dns -o wide | head -4
NAME                  READY   STATUS    RESTARTS   AGE   IP            NODE
dns-default-4jp8x     2/2     Running   0          12d   10.128.0.7    master-0
dns-default-b2m7k     2/2     Running   0          9d    10.131.0.5    worker-1
node-resolver-5x9dc   1/1     Running   0          12d   10.0.12.41    master-0

$ oc rsh -n payments-prod deploy/payments-api cat /etc/resolv.conf
search payments-prod.svc.cluster.local svc.cluster.local cluster.local bank.internal
nameserver 172.30.0.10
options ndots:5

The name shape is the familiar <service>.<namespace>.svc.cluster.local. Thanks to the search list, a pod can say payments-db (same namespace), payments-db.payments-prod (any namespace) or the full name. That ndots:5 line matters: a name with fewer than five dots is first tried with every search suffix appended, so api.partner.com generates four failed cluster lookups before the real one. On a slow corporate upstream this alone turns a 20 ms call into 400 ms. Teams that see mysterious latency on external calls should use fully qualified names with a trailing dot or lower ndots in dnsConfig.

Forwarding to corporate DNS

CoreDNS answers for cluster.local and forwards everything else to whatever the node's own /etc/resolv.conf says (DHCP or the installer's static config). In a bank that default is rarely enough: internal zones such as bank.internal must resolve through specific corporate resolvers, sometimes over TLS, and audit wants to know exactly where queries go. That is configured on the DNS Operator's CR:

$ oc edit dns.operator/default
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
  - name: corporate
    zones:
    - bank.internal
    - corp.bank.com
    forwardPlugin:
      policy: Sequential
      upstreams:
      - 10.10.5.53
      - 10.10.6.53
  upstreamResolvers:
    policy: Sequential
    upstreams:
    - type: SystemResolvConf
  logLevel: Normal
  operatorLogLevel: Normal

spec.servers adds zone-specific forwarders; spec.upstreamResolvers replaces the catch-all default. Turn on query logging temporarily with logLevel: Debug (or Trace) and read oc logs -n openshift-dns ds/dns-default -c dns; turn it off afterwards, because the volume is enormous. The Operator also supports spec.nodePlacement (banks sometimes keep DNS pods off DMZ nodes) and a spec.cache block for positive and negative TTLs on newer releases.

Common DNS incidents

• Cluster names resolve, corporate names do not: the upstream is unreachable from the nodes (firewall change, resolver decommissioned). Test from a node with oc debug node/<n> -- chroot /host dig @10.10.5.53 payments-db.bank.internal, then from a pod against 172.30.0.10.
• NXDOMAIN for a Service that exists: nine times out of ten the pod is in a different namespace and used the short name; the tenth time the Service was created in the wrong namespace. oc get svc -A | grep <name> ends the argument.
• DNS works, then randomly times out: a NetworkPolicy with an egress section that forgot to allow DNS, or a dns-default pod on one node in CrashLoopBackOff so one in N queries fails. oc get pods -n openshift-dns and oc get co dns first.
• Slow external calls: the ndots:5 effect above, or a forwarder policy of RoundRobin including a dead upstream.

NetworkPolicy in practice: deny by default, then allow exactly what you need

A fresh OpenShift project has no NetworkPolicy, which means every pod can talk to every pod in the cluster. No bank leaves it that way. The baseline that auditors expect in every namespace is: deny all ingress, allow traffic from within the namespace, allow the routers in, allow the monitoring stack to scrape, and (if you also restrict egress) allow DNS. OVN-Kubernetes enforces NetworkPolicy as OVN ACLs, so there is nothing to install. Here is the set, written as you would ship it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-openshift-ingress
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          policy-group.network.openshift.io/ingress: ""

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-openshift-monitoring
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: monitoring
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-dns
    ports:
    - protocol: UDP
      port: 5353
    - protocol: TCP
      port: 5353

The two namespace labels are pre-applied by the platform: openshift-ingress carries policy-group.network.openshift.io/ingress: "" and openshift-monitoring carries network.openshift.io/policy-group: monitoring. Confirm with oc get ns openshift-ingress openshift-monitoring --show-labels before you write a policy that depends on them.

Make it automatic: the project request template

You do not want to rely on teams applying these five policies. OpenShift creates every new project from a project request template, and you can add objects to it. Export the default with oc adm create-bootstrap-project-template -o yaml > template.yaml, append the NetworkPolicies (plus a ResourceQuota and LimitRange while you are there), create the Template in openshift-config, and point the cluster at it with oc edit project.config.openshift.io/cluster and spec.projectRequestTemplate.name: project-request. From then on, every oc new-project is born with the baseline. Post 26 walks through the full template. Note that it only applies to projects created through the request flow; namespaces created directly by cluster-admins or by GitOps need the policies in the repository.

AdminNetworkPolicy: cluster-wide rules that teams cannot override

A NetworkPolicy lives in the namespace it protects, so a namespace admin can delete it. Regulated environments need rules that hold regardless. AdminNetworkPolicy (API group policy.networking.k8s.io, generally available in recent 4.16+ OVN-Kubernetes clusters) is cluster-scoped, evaluated before any NetworkPolicy, and ordered by an explicit priority where lower numbers win. Its actions are Allow, Deny and Pass (defer to namespace NetworkPolicies). Its sibling BaselineAdminNetworkPolicy (a singleton named default) is evaluated after NetworkPolicies, giving you a cluster-wide default when a namespace has no policy of its own. A PCI example:

apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
  name: pci-isolation
spec:
  priority: 10
  subject:
    namespaces:
      matchLabels:
        compliance: pci
  ingress:
  - name: allow-pci-routers
    action: Allow
    from:
    - namespaces:
        matchLabels:
          kubernetes.io/metadata.name: openshift-ingress
  - name: allow-monitoring
    action: Allow
    from:
    - namespaces:
        matchLabels:
          network.openshift.io/policy-group: monitoring
  - name: deny-everything-else
    action: Deny
    from:
    - namespaces: {}

One more OVN-specific tool for policy debugging: annotate a namespace with k8s.ovn.org/acl-logging: '{"deny":"alert","allow":"notice"}' and OVN writes every allowed or denied connection for that namespace to /var/log/ovn/acl-audit-log.log on the node (readable via oc debug node, or from the ovn-acl-logging container). When a team swears "our policy allows it", this log settles it.

Egress control: fixed IPs, firewalls and proxies

Everything so far was about traffic coming in or moving around. Banks care at least as much about traffic going out, for one practical reason: enterprise firewalls and databases allowlist by source IP. By default, a pod's outbound connection is source-NATed to the IP of whichever node it happens to be on. A namespace with pods spread across twenty workers presents twenty possible source IPs, and they change on every reschedule. The firewall team will not open twenty node IPs to the mainframe. OpenShift gives you four tools.

EgressIP: one predictable source address per namespace

An EgressIP object pins one or more IPs from the machine network to a set of namespaces (and optionally pods). OVN-Kubernetes assigns each IP to one node that carries the label k8s.ovn.org/egress-assignable: "", adds it as a secondary address on that node's interface, and routes matching pods' outbound traffic through it. If that node dies, the IP moves to another labelled node automatically; that failover is why you label at least two nodes.

$ oc label node worker-3 k8s.ovn.org/egress-assignable=""
$ oc label node worker-4 k8s.ovn.org/egress-assignable=""

$ cat egressip.yaml
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: payments-prod-egress
spec:
  egressIPs:
  - 10.0.14.200
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: payments-prod

$ oc apply -f egressip.yaml
$ oc get egressip
NAME                   EGRESSIPS     ASSIGNED NODE   ASSIGNED EGRESSIPS
payments-prod-egress   10.0.14.200   worker-3        10.0.14.200

If ASSIGNED NODE is empty, no labelled node is eligible: the IP is outside the node's subnet, already in use, or on a cloud the instance has hit its secondary-IP limit (the cluster records per-node capacity in the cloud.network.openshift.io/egress-ipconfig node annotation). Prove it works from inside the cluster with oc rsh and curl https://ifconfig.me or, better, by asking the database team what source address they see. A hostNetwork pod is never affected by EgressIP, and traffic to other pods or Services is not either; it only shapes traffic leaving the cluster network.

EgressFirewall: what a namespace may talk to

EgressFirewall is the outbound counterpart of NetworkPolicy, but with two features policies lack: rules by DNS name and an ordered allow/deny list. One object per namespace, and it must be named default. Rules are evaluated top to bottom; internal cluster traffic is never affected.

apiVersion: k8s.ovn.org/v1
kind: EgressFirewall
metadata:
  name: default
  namespace: payments-prod
spec:
  egress:
  - type: Allow
    to:
      cidrSelector: 10.50.0.0/24        # database subnet
    ports:
    - port: 5432
      protocol: TCP
  - type: Allow
    to:
      dnsName: api.payments-partner.com
  - type: Deny
    to:
      cidrSelector: 0.0.0.0/0

Check that it was accepted with oc get egressfirewall -n payments-prod; the STATUS column reads EgressFirewall Rules applied when OVN has programmed it. DNS-name rules depend on OVN resolving the name itself, so they follow the cluster DNS forwarding configuration above.

Egress router and the cluster-wide proxy

The egress router (an EgressRouter CR that runs a pod with a Multus macvlan interface holding a fixed IP and NATs a namespace's traffic through it) predates EgressIP and is now a niche answer for legacy layouts; mention it in one sentence and move on to EgressIP. The cluster-wide proxy, on the other hand, is everywhere in banks, because nodes are not allowed to reach the internet directly. It is a singleton object:

$ oc get proxy/cluster -o yaml | sed -n '/^spec:/,$p'
spec:
  httpProxy: http://proxy.bank.internal:3128
  httpsProxy: http://proxy.bank.internal:3128
  noProxy: .bank.internal,10.0.0.0/8
  trustedCA:
    name: user-ca-bundle
status:
  httpProxy: http://proxy.bank.internal:3128
  httpsProxy: http://proxy.bank.internal:3128
  noProxy: .bank.internal,.cluster.local,.svc,10.0.0.0/8,10.128.0.0/14,127.0.0.1,172.30.0.0/16,api-int.prod-tor01.bank.internal,localhost

Notice that status.noProxy is longer than what you typed: the Cluster Network Operator appends the cluster, service and machine networks and the internal API hostname automatically. trustedCA names a ConfigMap in openshift-config holding the proxy's CA bundle, and the bundle is injected into any ConfigMap labelled config.openshift.io/inject-trusted-cabundle: "true", which is how application namespaces obtain it. Changing the proxy object rolls the Machine Config Pools, because the proxy settings are written onto every node.

The rest of the networking toolbox

Multus and secondary networks

Multus is a meta-CNI that OpenShift always installs; it lets a pod have more than one network interface. The primary interface still comes from OVN-Kubernetes, and each additional one is described by a NetworkAttachmentDefinition (NAD). Common drivers are macvlan (a pod interface directly on a node VLAN, for example to reach a storage or market-data network without NAT), ipvlan, bridge, and SR-IOV via the SR-IOV Network Operator for low-latency workloads that need a slice of a physical NIC. IP addressing on secondary networks usually uses the Whereabouts IPAM plugin that ships with OpenShift.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: backend-vlan120
  namespace: payments-prod
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "ens192.120",
    "mode": "bridge",
    "ipam": { "type": "whereabouts", "range": "192.168.120.0/24" }
  }'

A pod opts in with the annotation k8s.v1.cni.cncf.io/networks: backend-vlan120 and gets a net1 interface; the result is recorded in the k8s.v1.cni.cncf.io/network-status annotation, which is where you look when a pod "has no IP on the second network". Remember NetworkPolicy applies only to the primary network; a secondary interface is a deliberate hole that security must sign off on.

MetalLB, Service Mesh and IPsec

MetalLB answers "how do I get a Service of type LoadBalancer on bare metal or vSphere". The MetalLB Operator runs speakers on nodes; you define an IPAddressPool and either an L2Advertisement (a node answers ARP for the IP, simple, no switch changes) or a BGPAdvertisement plus BGPPeer (the node announces the IP to the bank's routers, which is what network teams prefer at scale). It is also how you publish additional IngressController shards with LoadBalancerService on-prem.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: dmz-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.60.1.10-10.60.1.30
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: dmz-l2
  namespace: metallb-system
spec:
  ipAddressPools:
  - dmz-pool

OpenShift Service Mesh is Red Hat's Istio distribution: Envoy sidecars (or ambient mode in newer versions) provide mutual TLS between services, retries, timeouts, traffic splitting and distributed tracing without touching application code. Version 2.x used the ServiceMeshControlPlane and ServiceMeshMemberRoll objects; version 3.x, generally available since 2025, is built on the upstream Sail Operator with an Istio CR and behaves like community Istio. Banks adopt it mainly for mTLS-everywhere evidence and for canary releases; the platform team's job is to run the control plane and the ingress gateways, and to know that a mesh-enabled namespace has an Envoy in the path when you troubleshoot 503s.

Finally, IPsec: OVN-Kubernetes can encrypt all pod-to-pod overlay traffic between nodes. The setting is ipsecConfig.mode under spec.defaultNetwork.ovnKubernetesConfig on networks.operator.openshift.io/cluster, with values Disabled, External (only traffic to external hosts, via node IPsec configuration) and Full (overlay plus external). Enabling it costs CPU and a few percent of throughput, and it reduces the cluster MTU further (ESP headers), which is why it is decided at design time, not switched on during an audit.

The networking troubleshooting method

Here is the ordered method for a north-south ticket ("route returns 503"), followed by the variations. Say it in this order in the interview; each step either finds the fault or eliminates a layer.

1. Is the platform healthy? Three cluster operators own the path. If any is Degraded, stop and read its message before touching the application.

   $ oc get co network ingress dns
   NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   dns       4.18.12   True        False         False      41d
   ingress   4.18.12   True        False         False      41d
   network   4.18.12   True        False         False      41d
   
   $ oc get pods -n openshift-ingress -o wide
   $ oc get pods -n openshift-ovn-kubernetes -o wide | grep -v Running

2. Was the route admitted, and does it have endpoints? oc describe route answers both in one screen. A 503 with the router's "Application is not available" page means the router has no healthy backend for that host: not admitted (HostAlreadyClaimed, wrong shard, wrong domain), Service selector matching zero pods, or pods not Ready.

   $ oc describe route payments-api -n payments-prod
   Name:                   payments-api
   Namespace:              payments-prod
   Requested Host:         payments-api.apps.prod-tor01.bank.internal
                             exposed on router default (host router-default.apps.prod-tor01.bank.internal) 92 days ago
   Path:                   /v1
   TLS Termination:        reencrypt
   Insecure Policy:        Redirect
   Endpoint Port:          https
   
   Service:        payments-api
   Weight:         100 (100%)
   Endpoints:      10.129.2.17:8443, 10.131.0.44:8443
   
   $ oc get endpoints payments-api -n payments-prod
   NAME           ENDPOINTS                            AGE
   payments-api   10.129.2.17:8443,10.131.0.44:8443    92d

   If Endpoints shows <none>, compare oc get svc payments-api -o yaml selector against oc get pods --show-labels, then check readiness probes; that is Post 9 territory.
3. Bypass the router. From a pod in the same namespace, curl the Service directly. If this works and the route does not, the problem is in the router or a NetworkPolicy blocking openshift-ingress; if it fails, the router was never the problem.

   $ oc rsh -n payments-prod deploy/payments-ui
   sh-5.1$ curl -sk -o /dev/null -w '%{http_code}\n' https://payments-api:8443/v1/health
   200
   sh-5.1$ exit
   
   $ oc get networkpolicy -n payments-prod
   NAME                           POD-SELECTOR   AGE
   deny-all-ingress               <none>         92d
   allow-same-namespace           <none>         92d

   That output is missing allow-from-openshift-ingress, and that is your 503. If the namespace has no shell-capable image, use oc run curl --rm -it --image=registry.access.redhat.com/ubi9/ubi-minimal -- curl ... or oc debug deploy/payments-ui.
4. Read the router's view. Router logs show the exact backend HAProxy chose and the HAProxy termination flags; the HAProxy config confirms what the router generated for the route.

   $ oc logs -n openshift-ingress deploy/router-default -c logs | grep payments-api | tail -5
   $ oc rsh -n openshift-ingress deploy/router-default grep -A6 'be_secure:payments-prod:payments-api' haproxy.config
   $ oc rsh -n openshift-ingress deploy/router-default \
       bash -c 'echo "show stat" | socat stdio unix-connect:/var/lib/haproxy/run/haproxy.sock' | grep payments

5. Go to the node. When you need to see packets or test a port from a node's point of view, oc debug node gives you a privileged shell with the host filesystem at /host.

   $ oc debug node/worker-1
   Starting pod/worker-1-debug-8fj2k ...
   sh-5.1# chroot /host
   sh-5.1# ss -tlnp | grep -E ':(80|443|1936) '
   sh-5.1# timeout 3 bash -c '</dev/tcp/10.50.0.12/5432' && echo open || echo closed
   sh-5.1# tcpdump -nn -i br-ex host 10.50.0.12 and port 5432 -c 20
   sh-5.1# crictl ps --name payments-api -q | head -1 | xargs crictl inspect | grep '"pid"'
   sh-5.1# nsenter -n -t <pid> -- tcpdump -nn -i eth0 -c 20

   The last two lines capture inside the pod's network namespace, which is how you see traffic before OVN touches it. nc -zv host port works where nc is present; the bash /dev/tcp trick works everywhere.
6. Collect evidence for Red Hat or for the RCA. oc adm must-gather -- /usr/bin/gather_network_logs collects OVN databases, flows and node network state cluster-wide; attach it to the support case. For egress problems add oc get egressip, oc get egressfirewall -A and the node annotations from step one.

The same six steps compress into a lookup table for the tickets you will actually get:

Storage in OpenShift 4: CSI everywhere

Storage in OpenShift 4 is Kubernetes storage with the drivers pre-installed and operator-managed. Every backend is a CSI driver (Container Storage Interface: the standard plug-in API for storage; the in-tree cloud volume plugins are gone). The Cluster Storage Operator (oc get co storage) installs the right CSI driver operator for the platform into openshift-cluster-csi-drivers: AWS EBS, Azure Disk and Azure File, GCP PD, vSphere, IBM, and so on. Each driver operator creates a default StorageClass and keeps the driver's controller and node pods running. Everything else you learned in Post 7 applies unchanged.

$ oc get storageclass
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
thin-csi (default)            csi.vsphere.vmware.com                  Delete          WaitForFirstConsumer   true                   210d
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   198d
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   198d
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc         Delete          Immediate              false                  198d

$ oc get clustercsidrivers
NAME                     AGE
csi.vsphere.vmware.com   210d

On AWS you would see gp3-csi (default) and gp2-csi from ebs.csi.aws.com; on Azure managed-csi; on GCP standard-csi. The default is chosen by the annotation storageclass.kubernetes.io/is-default-class: "true"; a PVC with no storageClassName gets it. A bank often makes the ODF block class the default on-prem and leaves the cloud disk class for anything that explicitly asks.

Binding mode: why WaitForFirstConsumer is the default

volumeBindingMode: WaitForFirstConsumer means the PV is not provisioned when the PVC is created but when the first pod using it is scheduled. The reason is topology: an EBS volume lives in one availability zone and a vSphere datastore may be reachable from only some hosts. If the volume were created first, the scheduler might be unable to place a pod where the disk is. With WaitForFirstConsumer, the scheduler picks a node considering everything else (CPU, affinity, taints), then the CSI driver creates the disk in that node's zone. The visible side effect is that an unused PVC sits in Pending forever, and its events say so politely. Immediate (used by Ceph RBD and CephFS, which are reachable from every node) provisions on creation.

Access modes and which backend gives you which

Access modes describe how many nodes may mount a volume, not how many pods. RWO (ReadWriteOnce) allows one node at a time, so several pods on the same node can share it, which surprises people in both directions. RWX (ReadWriteMany) allows many nodes. ROX (ReadOnlyMany) is many nodes, read-only. RWOP (ReadWriteOncePod, generally available since OpenShift 4.16) restricts to exactly one pod cluster-wide, which is what a single-writer database actually wants. The backend decides what is possible:

Reclaim policy, expansion and snapshots

Dynamically provisioned PVs default to reclaimPolicy: Delete: delete the PVC and the disk is gone. Bank data classification often mandates Retain for production databases, which leaves the PV in Released state with its data until a human acts. Expansion works when the StorageClass has allowVolumeExpansion: true: edit spec.resources.requests.storage on the PVC, the CSI driver grows the disk, and the filesystem is resized online for most drivers or on the next mount for others; you can never shrink. Snapshots use the snapshot.storage.k8s.io/v1 API: a VolumeSnapshotClass per driver, a VolumeSnapshot pointing at a PVC, and a new PVC with dataSource pointing at the snapshot to restore. The commands you run daily:

$ oc get pv | grep -v Bound                       # Released and Available PVs
$ oc get pvc -A | grep -v Bound                   # anything Pending or Lost
NAMESPACE      NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
payments-dev   uploads-shared   Pending                                      thin-csi       14m

$ oc patch pvc payments-db-data -n payments-prod -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
$ oc get volumesnapshot -n payments-prod
NAME                    READYTOUSE   SOURCEPVC          RESTORESIZE   SNAPSHOTCLASS                 AGE
payments-db-pre-patch   true         payments-db-data   100Gi         ocs-storagecluster-rbdplugin-snapclass   3h

OpenShift Data Foundation, and the other on-prem options

OpenShift Data Foundation (ODF), formerly OpenShift Container Storage (OCS), is Red Hat's software-defined storage for OpenShift: Ceph run by the Rook operator inside the cluster, plus NooBaa for object storage. From one operator you get three things a bank needs and cloud disks alone cannot give: RBD block volumes (RWO, class ocs-storagecluster-ceph-rbd), CephFS shared filesystems (RWX, class ocs-storagecluster-cephfs), and S3-compatible object storage (NooBaa's multicloud gateway or Ceph RGW, consumed through an ObjectBucketClaim). Banks reach for it on-prem (vSphere or bare metal, where there is no EBS), whenever RWX is needed, and as the backend for platform services: the image registry, Prometheus, Loki's object storage, OADP backups.

ODF runs in the openshift-storage namespace, installed from OperatorHub as the odf-operator (Post 23); you then create a StorageCluster CR named ocs-storagecluster. In internal mode it consumes disks from at least three nodes labelled cluster.ocs.openshift.io/openshift-storage="", often dedicated storage/infra nodes tainted node.ocs.openshift.io/storage=true:NoSchedule so application pods stay off them; in external mode it connects to an existing Red Hat Ceph Storage cluster run by the storage team. Data is replicated three ways by default, so 1.5 TiB of raw disk yields about 500 GiB usable.

Checking ODF health

$ oc get storagecluster,cephcluster -n openshift-storage
NAME                                                 AGE    PHASE   EXTERNAL   CREATED AT             VERSION
storagecluster.ocs.openshift.io/ocs-storagecluster   198d   Ready              2026-02-22T09:14:03Z   4.18.0

NAME                                                      DATADIRHOSTPATH   MONCOUNT   AGE    PHASE   MESSAGE                        HEALTH      EXTERNAL
cephcluster.ceph.rook.io/ocs-storagecluster-cephcluster   /var/lib/rook     3          198d   Ready   Cluster created successfully   HEALTH_OK   false

When HEALTH is not HEALTH_OK, the Ceph toolbox pod gives you the real Ceph CLI. It is disabled by default; enable it once on the OCSInitialization object, then oc rsh in:

$ oc patch OCSInitialization ocsinit -n openshift-storage --type=json \
    -p '[{"op":"replace","path":"/spec/enableCephTools","value":true}]'
$ oc rsh -n openshift-storage $(oc get pod -n openshift-storage -l app=rook-ceph-tools -o name)
sh-5.1$ ceph status
  cluster:
    id:     6a1c0f2e-3b4d-4e5f-9a8b-7c6d5e4f3a2b
    health: HEALTH_WARN
            1 osds down
            Degraded data redundancy: 41821/125463 objects degraded (33.333%), 173 pgs degraded
  services:
    mon: 3 daemons, quorum a,b,c (age 3d)
    mgr: a(active, since 3d), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 2 up (since 4m), 3 in (since 198d)
    rgw: 1 daemon active
  data:
    pools:   12 pools, 353 pgs
    objects: 41.82k objects, 158 GiB
    usage:   470 GiB used, 1.0 TiB / 1.5 TiB avail
    pgs:     173 active+undersized+degraded
             180 active+clean
sh-5.1$ ceph osd status
sh-5.1$ ceph health detail
sh-5.1$ ceph df

Reading that: one OSD (Object Storage Daemon, one per disk) is down, so a third of the placement groups have only two copies. The cluster still serves I/O, but a second failure would lose data. The common ODF incidents and where to look:

• OSD down: oc get pods -n openshift-storage -l app=rook-ceph-osd; usually the node rebooted or its disk failed. If the node is back and the pod stays down, the OSD's disk may be gone and needs the documented OSD replacement procedure. Never ceph osd out on a whim: it triggers a full rebalance.
• Near-full: ODF alerts CephClusterNearFull at 75 percent, CephClusterCriticallyFull at 80 and switches the cluster read-only at 85. At that point every PVC on Ceph stops accepting writes, which takes the registry, Prometheus and every database with it. The fix is adding capacity (another storage node or bigger device set) or deleting orphaned PVs; the prevention is the alert wired to a pager.
• MDS slow requests: the MDS (metadata server) serves CephFS directory operations. A workload creating millions of small files, or an under-resourced MDS, produces "MDSs report slow requests" and every CephFS mount feels frozen. Check ceph fs status, give the MDS more memory in the StorageCluster, or split the workload.

NFS, Local Storage Operator and LVM Storage

NFS from an enterprise filer (NetApp, Dell, Pure) is still everywhere in banks. OpenShift has no built-in NFS provisioner; you either pre-create PVs with nfs: volume sources, use the storage vendor's CSI driver (Trident, PowerScale CSI) which is the supported path, or run the community nfs-subdir-external-provisioner, which works but carries no Red Hat support. The recurring pain is permissions: NFS does not honour fsGroup the way block volumes do (there is no chown on mount), SELinux relabelling does not apply to NFS exports, and root_squash on the export maps the container's root to nobody. The usual answers are a supplementalGroups entry matching the export's group ownership, an export that permits the namespace's UID range, or no_root_squash for a specific trusted export, all agreed with the storage team, and tied to the SCC rules in Post 22.

The Local Storage Operator turns disks physically attached to nodes into PVs: a LocalVolumeDiscovery finds devices, a LocalVolumeSet or LocalVolume claims them, and the resulting PVs have node affinity, so the pod is pinned to that node. It is mainly used under ODF on bare metal (ODF's OSDs consume the local PVs) and for databases that want raw NVMe. LVM Storage (LVMS, formerly the ODF LVM operator) is the answer for single-node OpenShift and small edge clusters: an LVMCluster CR builds a volume group from local disks and the class lvms-vg1 provisions RWO volumes with snapshot support, with no Ceph overhead. A branch-office or trading-floor edge cluster is where you would name it.

Platform components that need storage

Three platform services silently need persistent storage, and a cluster that "works" without them is a cluster that will lose data on the next restart.

The internal image registry. On cloud installs the registry operator configures itself (S3 on AWS, a PVC on vSphere). On bare metal it comes up with managementState: Removed because it has nowhere to put images, and the first task after install is to give it storage. The configuration lives in one object:

$ oc edit configs.imageregistry.operator.openshift.io/cluster
spec:
  managementState: Managed
  replicas: 2
  rolloutStrategy: RollingUpdate
  storage:
    pvc:
      claim: image-registry-storage      # must be RWX (CephFS/NFS) when replicas > 1
  # cloud alternative:
  # storage:
  #   s3:
  #     bucket: prod-tor01-registry
  #     region: ca-central-1

Two rules an interviewer may probe. First, storage.emptyDir: {} is allowed and tempting in a lab, and every image pushed to it disappears when the registry pod restarts; it must never be used outside a throwaway cluster. Second, running two registry replicas with RollingUpdate requires an RWX volume, because both pods mount the same PVC on different nodes; with an RWO volume you get one replica and rolloutStrategy: Recreate, which is a short outage on every registry update.

Prometheus and Alertmanager. Out of the box the monitoring stack writes to emptyDir, so a Prometheus pod restart erases your metrics history and every alert's context. Persistent storage is set in the cluster-monitoring-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        spec:
          storageClassName: ocs-storagecluster-ceph-rbd
          resources:
            requests:
              storage: 200Gi
    alertmanagerMain:
      volumeClaimTemplate:
        spec:
          storageClassName: ocs-storagecluster-ceph-rbd
          resources:
            requests:
              storage: 10Gi

Loki for logging stores chunks in object storage, never on a PVC: the LokiStack CR points at a Secret describing an S3 bucket, an ODF NooBaa bucket claim, or Azure/GCS. Sizing retention against object storage cost, and what happens to logs when the bucket is unreachable, are covered in Post 24.

Storage troubleshooting

Storage tickets have the same shape as networking tickets: a symptom, an ordered list of causes, and one command that distinguishes them. The list below is in the order you should check.

PVC Pending

oc describe pvc and read the last event; it names the cause almost every time.

If the events are empty, the provisioner never saw the claim: check the CSI controller pods in openshift-cluster-csi-drivers or openshift-storage and oc get co storage.

Multi-Attach error: the RWO volume that will not follow its pod

A node goes down. Its StatefulSet pod is rescheduled elsewhere and sits in ContainerCreating with this event:

$ oc describe pod payments-db-0 -n payments-prod | tail -5
  Warning  FailedAttachVolume  4m   attachdetach-controller  Multi-Attach error for volume "pvc-3c1e9a7f-2b4d-4c8e-9f1a-0d2e3f4a5b6c"
                                                             Volume is already used by pod(s) payments-db-0
  Warning  FailedMount         1m   kubelet                  Unable to attach or mount volumes: unmounted volumes=[data],
                                                             unattached volumes=[data kube-api-access-x7k2p]: timed out waiting for the condition

The volume is RWO, and as far as the control plane knows it is still attached to the dead node, because the kubelet there never reported an unmount. Kubernetes tracks this in a VolumeAttachment object. The ordered fix:

1. Confirm the old node is really gone or NotReady (oc get nodes). If it is merely NotReady and coming back, wait: on a healthy control plane the attach/detach controller force-detaches after about six minutes once the node is confirmed unhealthy.
2. If the old pod object still exists in Terminating on the dead node, force-delete it: oc delete pod payments-db-0 -n payments-prod --force --grace-period=0. That tells the controller the volume is no longer in use.
3. If the node is permanently gone (cloud instance terminated), delete the Node object; the Machine API usually does this for you, but check with oc get machines -n openshift-machine-api.
4. Last resort, only after confirming on the storage side that the disk is not mounted anywhere: oc get volumeattachment | grep pvc-3c1e and oc delete volumeattachment <name>. Deleting it while the old node is alive and writing corrupts the filesystem, which is why this is step four and why you say "after confirming" in the interview.

Volume stuck Terminating, and orphaned PVs

A PVC that will not delete has the finalizer kubernetes.io/pvc-protection, which means a pod still references it; oc describe pvc lists the pod under Used By. Delete or fix the pod and the PVC goes. PVs carry kubernetes.io/pv-protection for the same reason. Removing a finalizer by hand (oc patch pvc x -p '{"metadata":{"finalizers":null}}') is the very last option, because it orphans the backend disk. Orphaned PVs come from the opposite path: a Retain class whose PVC was deleted leaves the PV Released with a stale claimRef. To reuse it, clear the reference and it becomes Available:

$ oc get pv | grep Released
pvc-8a9b0c1d-…   100Gi   RWO   Retain   Released   payments-prod/payments-db-data   ocs-storagecluster-ceph-rbd   61d
$ oc patch pv pvc-8a9b0c1d-… -p '{"spec":{"claimRef":null}}'

Schedule a monthly job that lists Released PVs and the Ceph or cloud disks they map to; a bank's storage bill is where orphans show up first.

Permission denied inside the pod

The app starts, then fails writing to its volume. On OpenShift the container is running as a random UID from the namespace's range (the restricted-v2 SCC assigns it) and the volume was created with root ownership or with files from another UID. Check oc rsh then id and ls -ln /data. For block volumes the fix is securityContext.fsGroup: the SCC already sets one from the namespace annotation openshift.io/sa.scc.supplemental-groups, and the CSI driver chowns the volume to it on mount, so a broken case usually means the driver's fsGroupPolicy is None or the image's entrypoint runs chown and fails. For NFS, see the discussion above: supplementalGroups and export options. For hostPath or local volumes, SELinux is the usual culprit (avc: denied in oc debug node then ausearch -m avc -ts recent), and the answer is a correct seLinuxOptions level or a relabel, never privileged. Post 22 covers the SCC side in detail.

Disk full on a node and DiskPressure eviction

Nodes have one filesystem that matters, holding images, container writable layers, emptyDir volumes and container logs. When it crosses the kubelet's eviction thresholds (defaults: nodefs.available under 10 percent, imagefs.available under 15 percent, inodes under 5 percent), the node reports DiskPressure, the scheduler stops placing pods there, and the kubelet evicts pods, starting with those exceeding their ephemeral-storage requests, with the event The node was low on resource: ephemeral-storage.

$ oc get nodes -o custom-columns='NAME:.metadata.name,DISK:.status.conditions[?(@.type=="DiskPressure")].status'
NAME       DISK
worker-1   True
worker-2   False

$ oc get --raw /api/v1/nodes/worker-1/proxy/stats/summary | jq '.node.fs | {capacityBytes, usedBytes, availableBytes}'

$ oc debug node/worker-1
sh-5.1# chroot /host
sh-5.1# df -h /var/lib/containers /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       120G  109G   11G  91% /var
sh-5.1# du -sh /var/log/pods/* 2>/dev/null | sort -h | tail -5
sh-5.1# crictl images | wc -l
sh-5.1# crictl rmi --prune                        # unused images only

The three usual causes: a container logging gigabytes to stdout (the kubelet rotates each container's log at containerLogMaxSize, 50Mi on OpenShift, keeping containerLogMaxFiles of them, but a pod with thirty restarts multiplies that); an application writing into its container filesystem or an emptyDir without an ephemeral-storage limit; and image accumulation on nodes that run many different workloads. The kubelet's image garbage collector removes unused images when usage passes imageGCHighThresholdPercent (85) down to imageGCLowThresholdPercent (80). Tune all of this per pool with a KubeletConfig applied through the Machine Config Operator (Post 20), and enforce limits.ephemeral-storage through a namespace LimitRange so one team cannot fill a node:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-disk-hygiene
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    imageGCHighThresholdPercent: 80
    imageGCLowThresholdPercent: 70
    containerLogMaxSize: 50Mi
    containerLogMaxFiles: 3
    evictionHard:
      nodefs.available: "10%"
      imagefs.available: "15%"

oc adm top node shows CPU and memory only; for disk you need the conditions, the stats summary, or the node exporter metrics in Prometheus (node_filesystem_avail_bytes), which is where the alert should live.

Backup and restore of volume data

Kubernetes objects can be regenerated from Git; volume data cannot. The supported tool is the OADP Operator (OpenShift API for Data Protection), which packages Velero. You install it in openshift-adp, create a DataProtectionApplication pointing at an S3 bucket (AWS S3, or an ODF NooBaa bucket on-prem), and then Backup, Restore and Schedule objects do the work. Volume data is captured either through CSI snapshots or, with snapshotMoveData: true, through the Data Mover, which copies snapshot contents to the object store so a restore works even if the original storage cluster is gone. That last property is what a bank's disaster recovery test actually checks.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: payments-prod-nightly
  namespace: openshift-adp
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
    - payments-prod
    snapshotMoveData: true
    storageLocation: default
    ttl: 720h0m0s

Likely interview questions See Post 33

Next up: OpenShift security, where SCCs decide what your pod is allowed to do, RBAC decides who may ask, OAuth ties the cluster to the bank's identity provider, and hardening turns all three into evidence an auditor will accept.