OpenShift, EKS, Terraform and GitOps Interview Question Bank

What is OpenShift, and how is it different from Kubernetes?

OpenShift is Red Hat's Kubernetes distribution: upstream Kubernetes plus an opinionated, supported set of integrated pieces. The important additions are an immutable OS (RHCOS) managed by the cluster itself, cluster operators that install and upgrade every component as one unit, integrated OAuth, Routes with an HAProxy router, an internal registry, Security Context Constraints and a web console. Kubernetes is the engine; OpenShift is the whole car, with one vendor to call when it breaks.

What is RHCOS and why does OpenShift insist on it?

Red Hat Enterprise Linux CoreOS is an immutable, container-optimized OS built on rpm-ostree, configured by Ignition at first boot and then owned by the Machine Config Operator. You never SSH in and run dnf install; you declare a MachineConfig and the cluster rolls it out. Control plane nodes must run RHCOS, and workers should too, because that is what makes upgrades and drift control reliable.

What container runtime does OpenShift use?

CRI-O, a lightweight OCI runtime built for Kubernetes and versioned in lockstep with it. There is no Docker daemon on the nodes; when you need to inspect containers on a node you use crictl from an oc debug node session.

Cluster operators versus OLM operators: what is the difference?

Cluster operators ship inside the OpenShift release payload and are managed by the Cluster Version Operator; they are the platform itself (ingress, DNS, monitoring, machine-config, etcd) and you see them with oc get co. OLM operators are optional add-ons installed from OperatorHub through a Subscription and upgraded from catalogs on their own channels. Different lifecycle, different upgrade path, different troubleshooting.

What does oc get co tell you, and what does healthy look like?

It lists every ClusterOperator with its version and three conditions: AVAILABLE, PROGRESSING and DEGRADED. Healthy is True False False on every row with the same version. Anything Degraded or Progressing for a long time is the first thing to oc describe co <name> during an incident or before an upgrade.

What is a Project, and how does it relate to a namespace?

A Project is a Kubernetes namespace plus OpenShift metadata (display name, description, requester annotation) created through the projectrequests API rather than directly. That indirection is what lets the platform apply a project template with quotas, NetworkPolicies and RoleBindings, and lets you control who may create projects at all through the self-provisioner role. oc get projects shows only the ones you can see.

Route versus Ingress: which one do you use on OpenShift?

A Route is OpenShift's original object (route.openshift.io/v1) served by the HAProxy-based IngressController; it natively supports edge, passthrough and re-encrypt TLS, router sharding and weighted backends. A standard Ingress object also works because OpenShift translates it into Routes automatically. For portability with EKS you can write Ingress; for OpenShift-specific TLS and sharding you write Routes.

Why use oc instead of kubectl?

oc embeds kubectl, so every kubectl command works, and adds OpenShift verbs: oc login, oc project, oc new-project, oc new-app, oc expose for Routes, oc debug, oc rsh, and the whole oc adm family (must-gather, upgrade, policy, node-logs, top). Keep the oc version within one minor of the cluster.

IPI versus UPI?

Installer-Provisioned Infrastructure means openshift-install creates the VPC, load balancers, DNS and machines and the Machine API manages nodes afterward. User-Provisioned Infrastructure means you build all of that first and the installer only lays down the cluster, which is common on vSphere or bare metal in banks with strict network ownership. IPI gives you MachineSets and self-healing nodes for free; UPI gives you control at the cost of automation.

What are ROSA and ARO?

Red Hat OpenShift Service on AWS and Azure Red Hat OpenShift are managed OpenShift, jointly operated by Red Hat SRE and the cloud provider: they run the control plane, upgrades and infrastructure operators, and you own projects, workloads and configuration. ROSA with hosted control planes is the current default, where the control plane runs in a Red Hat-owned account and you only pay for workers. A bank chooses them for less operational load and still keeps responsibility for IAM, networking design and workload security.

What are infra nodes and how do you create them?

Worker nodes labeled node-role.kubernetes.io/infra="" and tainted so only platform components (router, registry, monitoring, logging) land on them, which keeps them off application capacity and, under Red Hat's subscription rules, outside workload core counts. Method: create a MachineSet with the label and taint, then move each component with its own knob: IngressController spec.nodePlacement, registry spec.nodeSelector, and nodeSelector stanzas in cluster-monitoring-config.

What are hosted control planes?

HyperShift runs a cluster's control plane (API server, etcd, controllers) as Pods on a management cluster instead of on three dedicated nodes, and workers join it as a hosted cluster. You get faster cluster creation, lower cost and stronger isolation between control plane and workloads. ROSA HCP and RHACM-managed fleets use it.

What happened to DeploymentConfig?

DeploymentConfig has been deprecated since 4.14 in favor of the standard Deployment; it still works but new workloads should not use it. It differed by having ImageChange and ConfigChange triggers, lifecycle hooks and oc rollout latest. If a team still depends on image triggers, point them to a CI pipeline or Argo CD Image Updater instead.

What is an ImageStream?

A named pointer to a set of image tags, each resolved to an immutable digest, living in a project; it is metadata, not a registry. It gives you oc import-image, stable references like image-registry.openshift-image-registry.svc:5000/ns/app:1.4, and triggers for builds and rollouts. Its value today is digest pinning and controlled imports from external registries.

What is Source-to-Image?

S2I builds a runnable image from source code using a builder image (Java, Python, Node) without a Dockerfile, driven by a BuildConfig with sourceStrategy. It is convenient for developers on OpenShift Local, but a bank usually builds in a CI pipeline outside the cluster so scanning, signing and provenance happen once for every platform.

What are the console perspectives?

Administrator (cluster settings, operators, nodes, monitoring) and Developer (topology, builds, pipelines). Which perspectives are visible is set on the Console operator's spec.customization.perspectives, and recent releases (4.19) fold the developer views into the Administrator perspective with the Developer perspective disabled by default. Know it exists; interviewers ask because app teams ask.

How many control plane nodes does a cluster have, and why?

Three, because etcd needs a Raft majority: three members tolerate one failure while keeping quorum, and an even number adds no tolerance. Five is possible but adds write latency for a second failure you rarely need. Single-node OpenShift and three-node compact clusters exist for edge sites, not for a bank's production.

What does the Cluster Version Operator do?

It reads the desired version from the ClusterVersion object, pulls the release payload, and applies its manifests in order so every cluster operator ends up at the same version. It is why an OpenShift upgrade is one object edit, and why oc get clusterversion and oc adm upgrade are your upgrade dashboards.

What does oc get mcp show, and what is a healthy pool?

Every MachineConfigPool with UPDATED, UPDATING, DEGRADED and machine counts (total, ready, updated, degraded). Healthy is Updated True, Updating False, Degraded False with all machines counted as updated. A pool that stays Updating with one degraded machine is one node that cannot drain or reboot.

Walk me through what happens when you apply a MachineConfig.

You create a MachineConfig labeled for a pool (machineconfiguration.openshift.io/role: worker). The MachineConfig Controller merges all configs for that pool into a new rendered-worker-<hash> and points the pool at it. The MachineConfig Daemon on each node then cordons, drains (respecting PodDisruptionBudgets), writes the files and units through rpm-ostree, reboots and uncordons, one node at a time by default (spec.maxUnavailable). You track progress with oc get mcp and the node annotations currentConfig, desiredConfig and state.

How and why would you pause a MachineConfigPool?

oc patch mcp worker --type merge -p '{"spec":{"paused":true}}' stops the pool from rolling new rendered configs, so no worker reboots. You use it during change freezes and for EUS-to-EUS upgrades, but never leave it paused for weeks: certificate rotation and the next upgrade need the pool to move.

How do you add a kernel argument to all workers?

A MachineConfig for the worker pool with spec.kernelArguments, for example - transparent_hugepage=never. The MCO renders it and reboots the workers one by one, so it is a scheduled change with a window. Verify on a node with oc debug node/<name> and cat /host/proc/cmdline.

What is a KubeletConfig and when do you use it?

A CR that changes kubelet settings for a pool (maxPods, systemReserved, image garbage collection thresholds, podPidsLimit) without hand-writing the kubelet config file. You label the pool (for example custom-kubelet: large-pods), reference it in machineConfigPoolSelector, and the MCO renders a 99-worker-generated-kubelet MachineConfig and rolls the nodes.

What are the upgrade channels?

candidate-4.y, fast-4.y, stable-4.y and eus-4.y for the even-numbered Extended Update Support releases. You set it with oc adm upgrade channel stable-4.18. A bank runs stable; fast is for lower environments where you want the release a few weeks earlier.

What is the update graph?

Red Hat's OpenShift Update Service publishes a graph of tested upgrade edges; oc adm upgrade shows only the recommended targets for your channel and version, and conditional edges with known risks appear under --include-not-recommended. Disconnected clusters run their own OpenShift Update Service or upgrade by digest with oc adm upgrade --to-image.

How does an EUS-to-EUS upgrade work?

You pause the worker pools, upgrade the control plane through the odd intermediate release to the next EUS (for example 4.16 to 4.17 to 4.18), then unpause so workers reboot once instead of twice. It halves the application disruption, which is why banks plan around EUS releases.

Give me your pre-upgrade checklist.

In order: approved change ticket with a window and backout plan; oc get co all healthy and oc get mcp all updated; oc get clusterversion -o yaml shows Upgradeable=True and the target is recommended in oc adm upgrade; deprecated API usage checked with oc get apirequestcounts and acknowledged in the admin-acks ConfigMap; every OLM operator supports the target version; fresh etcd backup; no PodDisruptionBudgets with zero allowed disruptions that would block drains; no critical alerts firing; capacity for surge; and the same upgrade already done in dev and UAT.

The upgrade has been stuck for two hours. What do you do?

Find which layer is stuck: oc adm upgrade and the ClusterVersion conditions say whether a cluster operator or the MCO is waiting. For an operator, oc describe co <name> and the operator's Pod logs in its openshift-* namespace name the blocker. For nodes, oc get mcp and the node with state: Degraded, then the MachineConfig Daemon logs on that node; the usual causes are a drain blocked by a PDB or a node that will not come back. Fix the cause, never force the pool, and open a Sev 2 with a must-gather if a control plane operator is the one stuck.

Can you roll back an OpenShift upgrade?

No. Red Hat does not support rolling back to a previous minor or z-stream once the upgrade has started. Your protection is testing in lower environments, a fresh etcd backup, and fixing forward with support. Restoring etcd to the pre-upgrade snapshot is a disaster-recovery action that loses every change since the snapshot, not a rollback button.

How do you back up etcd, and what files do you get?

oc debug node/<master> -- chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup on one control plane node. It writes snapshot_<timestamp>.db (the etcd data) and static_kuberesources_<timestamp>.tar.gz (the static Pod manifests and certificates). Copy both off the node and automate the whole thing as a daily job before every change.

When would you actually restore etcd?

Only for loss of quorum you cannot recover by replacing members, or catastrophic corruption or deletion of cluster state. It is a documented disaster-recovery procedure (cluster-restore.sh on one control plane node, then rejoin the others) that rewinds the whole cluster. A bad application deploy is never a reason; that is a Git revert.

What is etcd defragmentation and when is it needed?

etcd keeps deleted space inside its database file until you defragment, so the file grows even when data shrinks. The cluster-etcd-operator defragments automatically when fragmentation is high, but you can do it by hand with oc rsh -n openshift-etcd etcd-<node> and etcdctl defrag one member at a time, leader last, checking etcdctl endpoint status -w table before and after.

How do certificates rotate, and what are pending CSRs?

Cluster-managed certificates (API server, etcd, service serving certs) rotate automatically. Node kubelet client and serving certificates go through CertificateSigningRequests, which the machine approver signs automatically for Machine API nodes; on UPI or after a long shutdown you approve them yourself with oc get csr and oc adm certificate approve <name>. A new node stuck NotReady with pending CSRs is a classic.

How do you shut down and start a cluster gracefully?

Take an etcd backup, note the certificate expiry window, drain and cordon workers, then oc debug node/<n> -- chroot /host shutdown -h 1 on workers first and control plane last. Start control plane nodes first, wait for the API, approve any pending CSRs, watch oc get co settle, then start and uncordon workers. If the cluster was off long enough for certificates to expire, there is a documented recovery procedure before anything else works.

How do you get images into a disconnected cluster?

With the oc mirror plugin: an ImageSetConfiguration lists the release versions, operator catalogs and extra images; you mirror to a local registry such as Quay, and it generates the ImageDigestMirrorSet and CatalogSource manifests to apply. Upgrades then go by digest or through a local OpenShift Update Service.

What is RHACM?

Red Hat Advanced Cluster Management is the hub for a fleet: it creates, imports and upgrades clusters, enforces configuration with Policies that report compliance per cluster, deploys applications through integrated Argo CD, and aggregates metrics with Thanos. A bank uses it to push the same security baseline to every cluster and prove it.

How do you replace a broken worker on an IPI cluster?

Drain it if possible, then delete its Machine in openshift-machine-api (oc delete machine <name> -n openshift-machine-api); the MachineSet creates a replacement, the machine approver signs its CSRs, and it joins the pool. If the node is dead, force-delete the stuck Pods first so RWO volumes detach. On UPI you rebuild the VM and approve the CSRs by hand.

How do you check for deprecated APIs before an upgrade?

oc get apirequestcounts shows which removed-in-next-release APIs are still being called and by which users, and the APIRemovedInNextReleaseInUse alert fires for the same reason. The upgrade is blocked until you fix the callers and acknowledge with a patch to the admin-acks ConfigMap in openshift-config.

What CNI does OpenShift use?

OVN-Kubernetes, and since 4.17 it is the only in-tree option because OpenShift SDN was removed. It is a Geneve overlay built on Open Virtual Network, runs as the ovnkube-node DaemonSet in openshift-ovn-kubernetes with a per-node OVN database, and gives you NetworkPolicy, AdminNetworkPolicy, EgressIP, EgressFirewall and optional IPsec out of the box.

What is an IngressController?

The CR in openshift-ingress-operator that defines a router: the HAProxy Pods in openshift-ingress, their replica count, node placement, the domain they serve and how they are published (LoadBalancerService on cloud, HostNetwork or NodePortService on bare metal). The default one handles every Route unless you shard.

What is router sharding and when would a bank use it?

Running additional IngressControllers that admit only Routes matching a routeSelector or namespaces matching a namespaceSelector, each with its own domain and load balancer. Typical bank split: an internal router on the corporate network, an external router in the DMZ, and sometimes a dedicated router for PCI-scoped applications.

Edge, passthrough and re-encrypt: explain each.

Edge terminates TLS at the router with a certificate on the Route and sends plain HTTP to the Pod. Passthrough forwards the TLS stream untouched, so the Pod holds the certificate and the router can only route by SNI hostname, not path. Re-encrypt terminates at the router and opens a new TLS connection to the Pod, verified against destinationCACertificate or the service serving certificate. A bank's default is re-encrypt for end-to-end encryption with insecureEdgeTerminationPolicy: Redirect.

How do wildcard routes work?

A Route with wildcardPolicy: Subdomain serves *.apps-team.example.com, but only if the IngressController allows it through routeAdmission.wildcardPolicy: WildcardsAllowed; the default is disallowed because a wildcard can hijack hostnames other teams expect to own.

A Route returns 503. Walk me through it.

503 from the router means it has no healthy backend, so work backward from the Route: oc get route -o yaml and check the Admitted condition (false usually means another namespace already owns the hostname); confirm spec.to.name is a real Service and its targetPort matches the container port; run oc get endpoints <svc>, and if it is empty the selector does not match or the Pods are failing readiness. Only then look at the router logs with oc logs -n openshift-ingress deploy/router-default.

What NetworkPolicy baseline do you put in every project?

Four policies applied by the project template: a default deny on ingress, allow from the same namespace, allow from the ingress routers using the namespace label policy-group.network.openshift.io/ingress: "", and allow from openshift-monitoring so scrapes work. Without them every Pod in the cluster can reach every other Pod across namespaces, which fails a bank's segmentation requirement immediately.

What is EgressIP and why would you need it?

A fixed source IP for traffic leaving the cluster from selected namespaces or Pods, defined in an EgressIP CR with a namespaceSelector and hosted on nodes labeled k8s.ovn.org/egress-assignable="", with failover between them. You need it whenever a downstream firewall, database or mainframe allow-lists by source IP, which in a bank is most of them.

What is EgressFirewall?

A per-namespace EgressFirewall CR (always named default) with ordered Allow and Deny rules by cidrSelector or dnsName controlling what Pods may reach outside the cluster. It answers the audit question "can this workload call the internet?" with a declarative no.

How do you forward DNS for a corporate zone?

Edit the DNS operator with oc edit dns.operator/default and add a spec.servers entry for the zone (for example corp.bank.internal) with the forwardPlugin.upstreams pointing at the Active Directory DNS servers; spec.upstreamResolvers sets the default upstreams. CoreDNS in openshift-dns reloads without a restart.

What is Multus for?

It is the meta-CNI that lets a Pod have additional interfaces defined by a NetworkAttachmentDefinition (macvlan, ipvlan, bridge, SR-IOV) and requested with the k8s.v1.cni.cncf.io/networks annotation. Use cases are a separate storage or replication VLAN and legacy systems that must see a real network address.

How do you get LoadBalancer Services on-prem?

The MetalLB Operator: an IPAddressPool of addresses and either an L2Advertisement (ARP from one node at a time) or a BGPAdvertisement peering with the top-of-rack switches. Without it, on bare metal or vSphere a type: LoadBalancer Service sits Pending forever.

How does the cluster-wide proxy work?

The proxy/cluster object holds httpProxy, httpsProxy, noProxy and a trustedCA ConfigMap in openshift-config; the MCO pushes it to nodes and operators inject it into their Pods. The classic mistake is a noProxy that misses the machine, service or Pod CIDRs or the API endpoints, which makes operators call the proxy for cluster-internal traffic. Set it at install time when you can.

Why does MTU matter, and what does a mismatch look like?

Geneve encapsulation costs 100 bytes, so a 1500-byte host network gives a 1400-byte cluster MTU, and a jumbo-frame fabric needs the overlay MTU set to match. A mismatch looks like small requests working while large transfers or TLS handshakes hang. Changing it later is a multi-step migration on the network.operator object with node reboots, so get it right in install-config.yaml.

When is hostNetwork: true acceptable?

For platform components that must own node ports, such as the router on bare metal or node exporters; it requires the hostnetwork or privileged SCC. It is not acceptable for applications because it bypasses NetworkPolicy and causes port conflicts, and a request for it is an architecture conversation, not an SCC grant.

How do you collect network diagnostics for Red Hat?

oc adm must-gather -- gather_network_logs adds OVN databases and flows to the standard bundle. For a single path you use oc get network.operator -o yaml, the ovnkube-controller container logs on the source node, and the ovnkube-trace tool from the OVN Pods to simulate the packet.

How does storage get provisioned on OpenShift?

Through CSI drivers shipped as operators: AWS EBS and EFS, vSphere, Azure Disk and File, plus Red Hat's OpenShift Data Foundation and LVM Storage. In-tree cloud drivers are gone. oc get storageclass shows what is offered and oc get csidriver shows what is installed.

Explain the access modes and which storage gives you each.

ReadWriteOnce is one node at a time (block storage: EBS, vSphere disks, Ceph RBD); ReadWriteMany is many nodes at once (file storage: EFS, CephFS, NFS, Azure Files); ReadOnlyMany is many readers; ReadWriteOncePod pins a volume to a single Pod. Interviewers listen for whether you know block cannot do RWX.

What does WaitForFirstConsumer mean?

A StorageClass volumeBindingMode that delays creating the volume until a Pod is scheduled, so the disk is created in the same availability zone as the node. A PVC sitting Pending with "waiting for first consumer to be created" is normal, not a fault, until a Pod claims it.

What is inside OpenShift Data Foundation?

Ceph deployed by the Rook operator: RBD for block (ocs-storagecluster-ceph-rbd, RWO), CephFS for shared file (ocs-storagecluster-cephfs, RWX) and RGW for S3 object storage, plus NooBaa as the multicloud object gateway. It needs at least three nodes with local disks and is defined by a StorageCluster CR in openshift-storage.

A team needs ReadWriteMany. What are the options?

CephFS from ODF on-prem, EFS through the EFS CSI driver on AWS, Azure Files on ARO, or an NFS appliance with a CSI driver. EBS and vSphere disks cannot do it, and the honest follow-up is whether they need shared files at all or an object bucket would serve them better.

How is the internal registry's storage configured?

Through configs.imageregistry.operator.openshift.io/cluster: on AWS IPI it gets an S3 bucket automatically, on-prem it needs an RWX PVC or ODF object storage under spec.storage, and managementState: Managed turns it on. With only an RWO volume you must set one replica and rolloutStrategy: Recreate.

What causes a Multi-Attach error?

An RWO volume is still attached to a node (typically a NotReady node or a Pod stuck Terminating) when a new Pod on another node tries to mount it. Confirm with oc get volumeattachment, make sure the old Pod is really gone (force delete if the node is dead), and the attachment clears. It is why a StatefulSet on a failed node does not just move on its own.

A PVC is stuck Pending. Method?

oc describe pvc and read the event: no default StorageClass or a misspelled one; the CSI controller Pod failing (check its logs in the driver namespace); a ResourceQuota on requests.storage or PVC count; WaitForFirstConsumer with no Pod yet; or on static PVs, no PV with matching size, mode and class.

How do you grow a volume?

The StorageClass must have allowVolumeExpansion: true; then edit the PVC's spec.resources.requests.storage upward and the CSI driver resizes the backing disk and, for most drivers, the filesystem online. You cannot shrink, and oc describe pvc shows a FileSystemResizePending condition if the Pod must restart to finish.

How do you back up applications and volumes?

CSI snapshots through a VolumeSnapshotClass and VolumeSnapshot for point-in-time copies, and OADP (OpenShift API for Data Protection, Red Hat's Velero operator) for real backups: a DataProtectionApplication pointing at an S3 bucket, then Backup, Restore and Schedule CRs that capture namespace objects plus volume data via snapshots or the Kopia data mover. Test the restore, not just the backup.

What is a Security Context Constraint?

An OpenShift admission control that decides what a Pod may request: which UIDs and SELinux labels, which capabilities, host namespaces and volume types, and whether privileged containers are allowed. It predates Pod Security Admission and is evaluated against the SCCs granted to the user and the Pod's ServiceAccount.

What does restricted-v2 enforce?

The default since 4.11: run as a UID from the namespace's range (never root), SELinux label assigned, all capabilities dropped except NET_BIND_SERVICE, allowPrivilegeEscalation: false, the runtime default seccomp profile, and no host network, PID, IPC or hostPath. It maps to the Pod Security "restricted" level, which is the point of the -v2.

How does OpenShift choose which SCC applies to a Pod?

Admission collects every SCC the requesting user and the Pod's ServiceAccount can use, sorts by priority (highest first), then by restrictiveness (most restrictive first), then by name, and picks the first one that admits the Pod, recording it in the openshift.io/scc annotation. The trap: a ServiceAccount granted anyuid (priority 10) gets it even for Pods that did not need it.

anyuid versus privileged?

anyuid only lifts the UID restriction, so a container may run as root or a fixed UID, while capabilities stay dropped and host access stays blocked. privileged allows everything: host namespaces, all capabilities, hostPath and privileged containers. Application teams may occasionally justify anyuid; they never get privileged.

How do you grant an SCC to a ServiceAccount, and how do you not?

oc adm policy add-scc-to-user anyuid -z app-sa -n payments creates a RoleBinding to the system:openshift:scc:anyuid ClusterRole for that one ServiceAccount, which is the auditable way. Never grant to a group like system:authenticated or edit the SCC's users list by hand. Check with oc adm policy who-can use scc anyuid.

How do you make an image run under restricted-v2?

Listen on a port above 1024, do not hardcode a UID the process depends on, and make writable paths group-owned by GID 0 with group write (chgrp -R 0 /app && chmod -R g=u /app), because OpenShift runs the container as an arbitrary UID with GID 0. Use emptyDir for scratch space instead of writing into the image filesystem.

Where do the arbitrary UIDs come from?

Each project carries an openshift.io/sa.scc.uid-range annotation such as 1000680000/10000; restricted-v2 assigns the first UID of that range, with supplemental and fs groups from matching annotations. Inside the container id shows something like uid=1000680000 gid=0(root).

Pod Security Admission versus SCCs on OpenShift?

Both run. SCCs enforce; PSA runs globally in audit and warn mode with the namespace labels synced from the SCC permissions of ServiceAccounts in that namespace. The PodSecurityViolation alert tells you which workloads would break if enforcement were tightened, which is exactly what you fix before that day arrives.

Name the default cluster roles you hand out.

admin (manage a project including RoleBindings), edit (create and change workloads, no RBAC), view (read-only, no secrets), cluster-reader (read everything cluster-wide), cluster-admin (everything), plus self-provisioner for creating projects and basic-user for logging in. Inspect any of them with oc describe clusterrole admin.

How do you give a team access to their project?

oc adm policy add-role-to-group edit AD-payments-devs -n payments, where the group already exists from LDAP sync; this creates a RoleBinding you can see with oc get rolebinding -n payments. Bind roles to groups, never to individual users, so an access review is a group membership check.

How do you stop developers from creating their own projects?

Remove the subjects from the self-provisioners ClusterRoleBinding (oc patch clusterrolebinding.rbac self-provisioners -p '{"subjects": null}') and annotate it with rbac.authorization.kubernetes.io/autoupdate: "false" so an upgrade does not restore it. Projects then come only from the platform team's onboarding automation.

How do you connect OpenShift to Active Directory?

Add an LDAP identity provider to oauth/cluster with the bind DN, a bind password Secret and CA ConfigMap in openshift-config, and an ldaps:// URL with the user search base and filter. Groups do not come along automatically: you write an LDAPSyncConfig (augmentedActiveDirectory schema) and run oc adm groups sync --sync-config=ldap-sync.yaml --confirm on a CronJob.

What do you do with the kubeadmin user?

Once an identity provider works and a real group holds cluster-admin, delete it with oc delete secret kubeadmin -n kube-system; it is irreversible. Keep a documented break-glass path (a sealed cluster-admin credential in the vault with an alert on use) because auditors ask for both the removal and the fallback.

How do ServiceAccount tokens work now?

Since 4.11 no long-lived token Secret is created automatically; Pods get a short-lived projected token, and for tooling you mint one with oc create token app-sa --duration=1h. Long-lived tokens for CI are a finding; use OIDC federation or short tokens issued per run.

How do you encrypt etcd?

Set spec.encryption.type to aesgcm (or aescbc) on apiserver/cluster; Secrets, ConfigMaps, Routes and OAuth tokens are then encrypted at rest and keys rotate automatically. Verify with the Encrypted condition on the openshiftapiserver and kubeapiserver objects, and pair it with RHCOS disk encryption for defense in depth.

What are the API audit profiles?

Default (metadata for everything, bodies for OAuth and login), WriteRequestBodies, AllRequestBodies and None, set on apiserver/cluster under spec.audit.profile with optional per-group rules. Logs land in /var/log/kube-apiserver/audit.log on control plane nodes, readable with oc adm node-logs --role=master --path=kube-apiserver/, and a bank forwards them to the SIEM.

How do you restrict where images can be pulled from?

image.config.openshift.io/cluster with registrySources.allowedRegistries (or blockedRegistries) and allowedRegistriesForImport for ImageStreams; the MCO writes it into every node's container registry configuration. Add the internal mirror, Quay and the registry hosting OpenShift's own images, or you break the platform.

What does FIPS mode mean for OpenShift?

Set fips: true in install-config.yaml and the nodes boot with FIPS-validated crypto modules and RHEL crypto policies; it cannot be turned on after installation, and the installer itself must run on a FIPS-enabled RHEL host. Government and some bank workloads mandate it.

What does the Compliance Operator do?

Runs OpenSCAP scans against profiles like ocp4-cis, ocp4-cis-node and ocp4-pci-dss through a ScanSettingBinding, producing ComplianceCheckResult objects (PASS, FAIL, MANUAL) and ComplianceRemediation objects that can apply fixing MachineConfigs. oc get compliancecheckresults -n openshift-compliance is your audit evidence.

What is Red Hat Advanced Cluster Security?

The StackRox platform: Central with a scanner, plus Sensor, Collector and an admission controller on each secured cluster. It scans images for CVEs, enforces policies at admission (no privileged containers, no critical fixable CVEs), detects runtime anomalies, and reports CIS, PCI and NIST compliance; roxctl image check puts the same policies in CI.

Explain Subscription, CSV, InstallPlan and OperatorGroup.

A CatalogSource is an index image of operator bundles. An OperatorGroup says which namespaces an operator in this namespace may watch (one per namespace). A Subscription is your intent: package, channel, source and approval mode. OLM resolves it into an InstallPlan (the concrete resources to create) and the result is a ClusterServiceVersion, the operator's Deployment, RBAC and CRDs, which should reach phase Succeeded.

How does manual approval work, and why use it in production?

installPlanApproval: Manual on the Subscription makes OLM create InstallPlans with approved: false; you approve with oc patch installplan install-xxxxx -n ns --type merge -p '{"spec":{"approved":true}}'. Production uses it so an operator upgrade happens inside a change window with a ticket, not whenever the catalog updates.

An operator install is stuck. Method?

oc get sub -n ns -o yaml and read status.conditions (a ResolutionFailed message names the problem); confirm the CatalogSource is READY in openshift-marketplace; check for a missing or duplicate OperatorGroup (TooManyOperatorGroups); then oc get csv and oc describe csv for a Pending or Failed phase, and finally the catalog-operator and olm-operator logs in openshift-operator-lifecycle-manager.

How do you upgrade an operator safely?

Read its release notes for CRD or behavior changes and confirm it supports your OpenShift version; back up its CRs; do it in dev and UAT first; in production switch the channel or approve the pending InstallPlan inside a window, watch the new CSV replace the old, and verify the workloads it manages. Some operators need a CRD storage version migration afterward.

How do operator catalogs work disconnected?

Mirror the catalogs you need with oc mirror (the operators section of ImageSetConfiguration), apply the generated CatalogSource and ImageDigestMirrorSet, and disable the default sources with oc patch OperatorHub cluster --type json -p '[{"op":"add","path":"/spec/disableAllDefaultSources","value":true}]'.

How do you uninstall an operator cleanly?

Delete the Subscription, then the CSV (oc delete csv <name> -n ns), which removes the Deployment and RBAC. OLM deliberately leaves CRDs and CRs; deleting a CRD deletes every CR of that kind, so decide that consciously. Sweep leftover webhooks, ClusterRoles and the OperatorGroup if the namespace was dedicated.

Where does a cluster operator's upgrade come from versus an OLM operator's?

A cluster operator upgrades only with the OpenShift release, driven by the CVO. An OLM operator upgrades when its catalog publishes a new bundle on the channel you subscribed to, gated by your approval mode. That is why "we upgraded the cluster" and "we upgraded the logging operator" are two separate changes.

What is OLM v1?

The rewrite (GA in 4.18) built on ClusterCatalog and ClusterExtension: cluster-scoped, no OperatorGroup, installs run under a ServiceAccount you must grant explicit RBAC, with version ranges, pinning and CRD upgrade safety checks. OLM v0 and OperatorHub still serve most operators, so both coexist for now.

What is the difference between an operator and a Helm chart?

A chart installs resources once; an operator is a running controller that keeps reconciling its CRs, handling day-2 tasks such as failover, backups and version upgrades. You use an operator for stateful or lifecycle-heavy software and a chart for a stateless application.

What is in the built-in monitoring stack?

The Cluster Monitoring Operator in openshift-monitoring runs two Prometheus replicas (prometheus-k8s), Alertmanager (alertmanager-main), Thanos Querier, node-exporter, kube-state-metrics and the metrics server, with dashboards in the console. It monitors the platform only; application metrics need user workload monitoring.

What goes in cluster-monitoring-config?

A ConfigMap in openshift-monitoring whose config.yaml sets enableUserWorkload: true, Prometheus retention and a volumeClaimTemplate for persistent storage, nodeSelector and tolerations for infra nodes, and resource limits. Without a PVC, Prometheus data lives in emptyDir and disappears on restart.

How does user workload monitoring work?

Enabling it creates openshift-user-workload-monitoring with its own Prometheus and Thanos Ruler. Teams create ServiceMonitor, PodMonitor and PrometheusRule objects in their own namespaces using the monitoring-edit or monitoring-rules-edit roles, and the results appear in the same console and Alertmanager.

Write a ServiceMonitor from memory.

monitoring.coreos.com/v1, kind ServiceMonitor, a selector.matchLabels matching the Service's labels, and endpoints naming the Service port name (not number), path and interval. The two mistakes I check first are a port name mismatch and a selector matching the Pod labels instead of the Service labels.

And a PrometheusRule?

Groups of rules with alert, expr, for, labels (severity: critical|warning|info) and annotations (summary, description, runbook_url). Severity labels drive Alertmanager routing, and a rule without a runbook link fails our platform standard.

How do you route alerts to PagerDuty and Slack?

Edit the alertmanager.yaml inside the alertmanager-main Secret in openshift-monitoring (or the console's Alertmanager configuration page): receivers for PagerDuty, Slack and a ServiceNow webhook, and a route tree matching on severity and namespace. Send Watchdog to a dead-man's-switch so you notice when alerting itself breaks.

Which platform alerts do you never silence?

etcdMembersDown, etcdHighNumberOfLeaderChanges, KubeAPIErrorBudgetBurn, ClusterOperatorDegraded, KubeNodeNotReady, MCDDrainError, KubePersistentVolumeFillingUp, NodeFilesystemAlmostOutOfSpace, ClusterNotUpgradeable and the absence of Watchdog.

PromQL for memory per Pod and for restart loops?

Memory: sum by (pod) (container_memory_working_set_bytes{namespace="payments", container!=""}), and divide by kube_pod_container_resource_limits{resource="memory"} for percent of limit. Restarts: increase(kube_pod_container_status_restarts_total{namespace="payments"}[1h]) > 3.

What is the current logging stack?

The Cluster Logging Operator with Vector as the collector DaemonSet and the Loki Operator running a LokiStack on object storage (S3 or ODF), viewed through the console plugin. Elasticsearch, Fluentd and Kibana are retired; Logging 6 uses the observability.openshift.io/v1 ClusterLogForwarder with a collector ServiceAccount bound to the collect-*-logs ClusterRoles.

How do you forward logs to Splunk?

A ClusterLogForwarder in openshift-logging with an output of type: splunk pointing at the HEC URL, the HEC token in a Secret referenced under authentication, and an index; then a pipeline mapping inputRefs (application, infrastructure, audit) to that output. Keep Loki as a second output for on-cluster search.

Are audit logs collected by default?

No. The audit input (kube-apiserver, openshift-apiserver, OAuth, OVN ACL and node auditd logs) must be named explicitly in a pipeline, and it is the one regulators care about most, so it goes to the SIEM with a retention of a year or more.

Prometheus keeps getting OOMKilled. Why, and what do you do?

Almost always cardinality: application metrics with IDs or request paths as labels. Find the offenders with topk(10, count by (__name__)({__name__=~".+"})) and prometheus_tsdb_head_series, drop labels with metricRelabelings, set enforcedSampleLimit for user workload monitoring, and only then raise resources or move Prometheus to infra nodes.

How is retention handled?

retention (default 15 days) and retentionSize in the monitoring ConfigMaps; long-term metrics go out with remoteWrite to Thanos, Amazon Managed Prometheus or Mimir. Loki retention is limits.global.retention.days on the LokiStack, and audit logs live in the SIEM, not in Loki.

What is the Insights Operator?

It uploads anonymized configuration to console.redhat.com, where Red Hat's rules flag known issues and misconfigurations, surfaced as the InsightsRecommendationActive alert and in the portal. Banks route it through the proxy or run disconnected; check it before every upgrade when it is available.

What is your method for any OpenShift problem?

Four layers, checked in order: control plane and cluster operators; nodes and machines; platform services (ingress, DNS, network, storage, registry, OLM); then the workload (Pods, events, logs, SCC, quota, NetworkPolicy). If many teams are affected I go top-down; if one application is, bottom-up. Saying the order out loud is half the answer.

First five commands on a sick cluster?

oc get co, oc get nodes, oc get mcp, oc get clusterversion, and oc get pods -A | grep -v -E 'Running|Completed', followed by oc get events -A --sort-by=.lastTimestamp | tail -30.

A node is NotReady. Method?

oc describe node for conditions and taints, oc adm node-logs <node> -u kubelet for the kubelet's story, then oc debug node/<node> and chroot /host for systemctl status kubelet crio, disk under /var/lib/containers and the MachineConfig Daemon state. On IPI also check oc get machine -n openshift-machine-api and the cloud console; if the instance is gone, delete the Machine and let the MachineSet replace it.

What does oc debug node actually do?

It starts a privileged Pod on that node with the host filesystem mounted at /host; after chroot /host you have crictl, journalctl, rpm-ostree status and nmcli. It needs cluster-admin and is recorded in the audit log, which is why it replaces SSH at a bank.

A cluster operator is Degraded. Method?

oc get co <name> -o yaml and read the Degraded condition message and status.relatedObjects, which name the namespace and objects to inspect; then the operator's Pods and logs there. Typical mappings: authentication degraded means the OAuth Route is unreachable (ingress or DNS), ingress means a load balancer or node placement problem, monitoring means a PVC, image-registry means storage.

The API is slow cluster-wide. Where do you look?

etcd first: etcdctl endpoint status -w table from an etcd Pod and the fsync and commit duration alerts, which point at disk latency on the control plane. Then KubeAPIErrorBudgetBurn, a noisy client in apiserver_request_total by user agent, API Priority and Fairness rejections, and admission webhook latency, because a slow or dead webhook makes every write crawl.

CrashLoopBackOff on OpenShift specifically: what is different?

After the usual describe and logs --previous, check the SCC angle: oc get pod -o yaml | grep openshift.io/scc, and look for "permission denied" on a file or a bind to a port below 1024, which means the image assumes root and is running as an arbitrary UID. Fix the image before touching SCCs.

The Deployment creates no Pods and says "unable to validate against any security context constraint". Method?

The failure is on the ReplicaSet, so oc describe rs or oc get events shows which field failed (runAsUser: 0, a capability, hostPath). Decide whether the requirement is legitimate, remove it from the manifest if not, and if it is, grant the narrowest SCC (nonroot-v2 before anyuid) to that ServiceAccount only. oc adm policy scc-subject-review -f deploy.yaml predicts the result before you apply.

ImagePullBackOff behind a mirror registry. Method?

Read the exact error in oc describe pod: "unauthorized" means the global pull secret in openshift-config lacks the registry; "x509" means the mirror's CA is not in the additionalTrustedCA ConfigMap; "manifest unknown" means the image was never mirrored or the reference is by tag while only an ImageDigestMirrorSet exists (tags need an ImageTagMirrorSet). Test from a node with crictl pull.

What is must-gather and how do you scope it?

oc adm must-gather collects cluster state into a tarball for a Red Hat case; --image adds a product-specific gatherer (ODF, logging, ACS), --since limits the window, and oc adm inspect ns/<ns> grabs one namespace. Check the bundle for secrets before uploading.

Red Hat support severities?

Severity 1 is production down or data at risk and gets round-the-clock engagement on a Premium subscription; 2 is major function degraded; 3 is moderate; 4 is a question. Open with the cluster ID from oc get clusterversion, version, timeline and a must-gather, and keep someone on the line for a Sev 1.

Who is on a major incident bridge?

An incident commander (decisions, priorities), a technical lead doing the hands-on work, a communications lead posting updates on a fixed cadence (every 30 minutes at most banks), a scribe keeping the timeline, and a vendor liaison for the Red Hat or AWS case. One person should never hold two of those roles on a Sev 1.

What goes in an RCA?

Summary; customer and business impact with duration; a timestamped timeline of detection, escalation, mitigation and resolution; root cause found by five-whys, with contributing factors; what went well and badly; and corrective and preventive actions with owners and due dates, plus links to evidence.

Incident, problem, change: define each in ITIL terms.

An incident is an unplanned interruption whose goal is restoring service fast. A problem is the underlying cause of one or more incidents, worked to a permanent fix. A change is any addition, modification or removal that could affect a service, done under approval. Incidents open problems; problems close through changes.

What is an emergency change?

A change raised to resolve an incident or prevent imminent harm, approved by the on-call approver or an emergency CAB rather than the weekly board, still ticketed, executed with four eyes and evidence, and reviewed afterward. It is a faster path through the process, not a way around it.

What makes a postmortem blameless, and why does a bank care?

It examines systems, signals and decisions rather than individuals, which is the only way to get an honest timeline. The bank cares because regulators may need to be told: OSFI expects material technology incidents reported within 24 hours, and the report is only as good as the postmortem behind it.

How do you make every new project start safe?

A project request template: oc adm create-bootstrap-project-template -o yaml, add the ResourceQuota, LimitRange, the four baseline NetworkPolicies and standard RoleBindings, create it in openshift-config, and point project.config.openshift.io/cluster at it under spec.projectRequestTemplate.name. It applies to new projects only, so existing ones need a one-time sweep.

What default quotas and limits do you set?

A ResourceQuota on requests.cpu, requests.memory, limits.*, pods, persistentvolumeclaims and requests.storage (and zero services.loadbalancers), sized by environment, plus a LimitRange giving every container a default request and limit so nothing schedules unbounded. Teams grow it through a ticket with a reason.

How is RBAC wired to Active Directory groups?

Group sync creates <app>-admins, <app>-devs and <app>-readers groups; the onboarding automation binds them to admin, edit and view in the project. No user-level bindings, so the quarterly access review is an AD membership export.

What is in your golden Deployment?

Requests and limits, readiness and liveness probes, a securityContext with runAsNonRoot, allowPrivilegeEscalation: false, capabilities dropped and seccompProfile: RuntimeDefault, two or more replicas with topology spread and a PodDisruptionBudget, standard app.kubernetes.io/* labels, a ServiceMonitor, a re-encrypt Route, images by digest, and secrets from the vault via External Secrets.

Helm, Kustomize or OpenShift Templates?

Helm for packaging and reusable internal charts with release management; Kustomize for per-environment overlays without templating, built into oc apply -k and ideal for GitOps; OpenShift Templates (oc process) only for legacy. Our pattern is a platform Helm chart with Kustomize overlays synced by Argo CD.

A team says their app needs root. What do you do?

Ask why: a port below 1024, file ownership or installing packages at runtime are all fixable in the image. If a fixed non-zero UID is genuinely required, grant nonroot-v2; anyuid needs a written justification, a risk acceptance with an expiry, and never privileged.

What is a ClusterResourceQuota?

A quota that spans multiple projects selected by label or the openshift.io/requester annotation, created with oc create clusterresourcequota; a line of business gets one envelope across its dev, UAT and feature projects, viewable per project through AppliedClusterResourceQuota.

What documentation does onboarding need?

A request form (owner, AD groups, environment, quota size, egress needs, data classification), the golden path repository, a "your first deployment" guide, the runbook for the ten most common tickets, and a RACI saying what the platform team owns versus the app team. Links live in the project's annotations.

How do you measure whether onboarding works?

Lead time from request to first successful Route, tickets per onboarding, percentage of workloads on the golden path, count of SCC exceptions, and repeat questions in the support channel. If the form feeds a pipeline, most of those numbers come from the pipeline itself.

How do you handle exceptions to a standard?

A written exception with a business owner, a risk acceptance, compensating controls and an expiry date, tracked in a register and reviewed quarterly; technically it becomes a labeled namespace or a policy exclusion in ACS or Kyverno tied to the record number.

What does AWS manage in EKS, and what is still yours?

AWS runs the control plane: API servers and etcd across three availability zones, their patching, scaling and backups, under a 99.95% SLA. You own the data plane (unless you use Fargate or Auto Mode), add-on versions, VPC design, IAM mappings, upgrades (you trigger them), and every workload.

How is API endpoint access configured?

Public, private or both, via endpointPublicAccess and endpointPrivateAccess, with a CIDR allow-list on the public side. A bank runs private-only and reaches the API through VPN or Direct Connect; the private endpoint resolves through a Route 53 private zone that EKS manages.

How do you design the VPC for a production cluster?

At least two AZs; small dedicated subnets for the control plane ENIs; private subnets for nodes tagged kubernetes.io/role/internal-elb=1; public subnets only for internet-facing load balancers tagged kubernetes.io/role/elb=1; a NAT gateway per AZ; and VPC endpoints for ECR, S3, STS and EC2 so image pulls and IAM never leave the VPC.

Why do EKS clusters run out of IP addresses?

The VPC CNI gives every Pod a real VPC address, so Pod density is bounded by subnet size and by the ENI and IP limits of each instance type. The symptom is Pods stuck in ContainerCreating with "failed to assign an IP address". Fixes: a secondary CIDR from 100.64.0.0/10 with custom networking, prefix delegation, tuned WARM_IP_TARGET, or IPv6.

VPC CNI versus OVN-Kubernetes?

VPC CNI has no overlay: Pods are routable in the VPC, security groups can attach to Pods, and AWS tooling sees them, at the cost of consuming VPC IPs and needing the network policy agent or Calico for NetworkPolicy. OVN-Kubernetes is a Geneve overlay with its own Pod CIDR and policy built in. Different trade-off, same Kubernetes on top.

What is prefix delegation?

ENABLE_PREFIX_DELEGATION=true makes the CNI attach /28 prefixes to ENIs instead of single addresses, which multiplies Pods per node (with max-pods raised to match). It needs Nitro instances and contiguous free /28 blocks, so a fragmented subnet defeats it.

What is CNI custom networking?

AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true plus an ENIConfig per AZ places Pods in a secondary CIDR separate from the nodes. It relieves pressure on the routable CIDR; the trade-offs are SNAT to the node IP for external traffic and losing the primary ENI's Pod capacity.

Managed node groups, Fargate, Karpenter or Auto Mode?

Managed node groups are AWS-managed Auto Scaling groups with rolling AMI updates; Fargate runs each Pod serverless with no DaemonSets or privileged access; Karpenter provisions right-sized nodes just in time from NodePool and EC2NodeClass and consolidates them; EKS Auto Mode has AWS run Karpenter, the CNI, storage and load balancer controllers and rotate Bottlerocket nodes within 21 days, for a per-instance premium. A bank typically picks managed node groups plus Karpenter, and Auto Mode for teams without platform staff.

Access entries versus the aws-auth ConfigMap?

The aws-auth ConfigMap mapped IAM roles to Kubernetes users and was easy to corrupt and lock yourself out with. Access entries are an EKS API (aws eks create-access-entry plus associate-access-policy with policies like AmazonEKSClusterAdminPolicy or AmazonEKSViewPolicy, scoped to the cluster or namespaces), managed by IaC and logged in CloudTrail. Set the authentication mode to API on new clusters.

IRSA versus EKS Pod Identity?

IRSA uses the cluster's OIDC provider: a role trust policy naming system:serviceaccount:ns:sa and an eks.amazonaws.com/role-arn annotation on the ServiceAccount, configured per cluster. Pod Identity uses the eks-pod-identity-agent add-on and a create-pod-identity-association that maps a ServiceAccount to a role whose trust policy names pods.eks.amazonaws.com once, reusable across clusters. Pod Identity is the simpler default for EKS-only workloads; IRSA still covers older SDKs and non-EKS clusters.

Why does IMDSv2 matter on a node?

Any Pod that can reach 169.254.169.254 can steal the node's instance role credentials. Requiring IMDSv2 tokens with a hop limit of 1 in the launch template stops that, and a bank enforces it with an SCP. Workloads then get credentials only through IRSA or Pod Identity.

What are EKS managed add-ons?

AWS-curated versions of vpc-cni, coredns, kube-proxy, eks-pod-identity-agent, the EBS and EFS CSI drivers, the CloudWatch observability agent and GuardDuty agent, installed and upgraded through aws eks create-addon and update-addon with a configurationValues JSON. They must be upgraded alongside each cluster version.

What does the AWS Load Balancer Controller do?

A controller (Helm installed, with IAM through IRSA or Pod Identity) that creates an ALB from an Ingress with ingressClassName: alb and an NLB from a Service with the aws-load-balancer-type: external annotation. Target type ip sends traffic straight to Pod IPs; subnets must carry the role tags or nothing gets created.

ALB or NLB?

ALB is layer 7: host and path routing, WAF, OIDC authentication, gRPC; use it for HTTP Ingress. NLB is layer 4: TCP and TLS passthrough, static IPs, PrivateLink, very high throughput; use it for non-HTTP Services and for anything a partner must allow-list by IP.

Walk me through an EKS upgrade.

One minor at a time. Pre-checks: Cluster Insights (aws eks list-insights) for deprecated APIs, at least five free IPs in each control plane subnet, add-on compatibility. Then aws eks update-cluster-version (about ten minutes, no downtime), upgrade coredns, kube-proxy and vpc-cni to compatible versions, roll the managed node groups (new AMI, respecting PDBs and maxUnavailable) or let Karpenter drift them, then verify. Dev first, production last, under a change ticket.

What is EKS extended support?

Each version gets 14 months of standard support, then 12 months of extended support at roughly six times the control plane price (about $0.60 per hour instead of $0.10). With upgradePolicy: STANDARD, EKS auto-upgrades the control plane when standard support ends, so the upgrade calendar is a budget item.

How do you get control plane and audit logs?

Enable the api, audit, authenticator, controllerManager and scheduler log types with aws eks update-cluster-config --logging; they land in CloudWatch Logs, and a subscription filter through Kinesis Data Firehose ships the audit stream to Splunk. Audit and authenticator are the ones compliance needs; ingestion cost is by volume.

How are Secrets encrypted in EKS?

Envelope encryption with KMS; EKS now applies it by default, and for a regulated cluster you supply a customer-managed key so rotation and access are governed by your key policy and visible in CloudTrail. The key cannot be swapped out afterward, so choose it at creation.

What does GuardDuty do for EKS?

EKS Protection analyzes the Kubernetes audit log for anomalies (anonymous access, privileged containers, credential misuse), and Runtime Monitoring deploys the GuardDuty agent add-on to catch crypto-mining, reverse shells and container escapes. Findings flow to Security Hub and the SIEM.

Container Insights, AMP and AMG: what is each?

Container Insights is CloudWatch's metrics and logs for the cluster, installed through the amazon-cloudwatch-observability add-on. Amazon Managed Service for Prometheus is a Prometheus-compatible store you remote-write to; Amazon Managed Grafana is the dashboarding front end with SSO. A bank that already owns Splunk or Dynatrace usually feeds those instead.

EBS or EFS for a workload?

EBS CSI for block, RWO, single-AZ volumes with snapshots (use WaitForFirstConsumer so the disk lands in the Pod's AZ); EFS CSI for NFS, RWX, multi-AZ file sharing with access points, at higher latency and cost per gigabyte. Databases go on EBS; shared content goes on EFS.

How do you do backup and disaster recovery for EKS?

Velero with the AWS plugin backs up objects to S3 and volumes through EBS snapshots on a schedule, with snapshots copied to the DR region. Because the control plane is regional, DR is a second cluster built from Terraform, populated by Argo CD, and restored from Velero, with RTO and RPO deciding whether it stays warm. Test the restore quarterly.

What drives EKS cost, and what do you do about it?

Nodes first, then NAT gateway data processing, load balancers, EBS, CloudWatch log ingestion (control plane logs included), cross-AZ traffic and extended support. Levers: honest requests, Karpenter consolidation with Spot and Graviton, VPC endpoints for ECR and S3, retention on logs, and cost allocation tags with Kubecost or OpenCost for showback.

What is Bottlerocket?

AWS's minimal immutable container OS, the EKS counterpart of RHCOS: API-driven configuration, no package manager or SSH by default (an admin container when you need one), image-based updates with rollback, and SELinux enforcing. It is the default in Auto Mode and a good choice for hardened node groups.

What is Terraform state, and how do you protect it?

The file mapping your configuration to real resource IDs; without it Terraform cannot plan. Keep it in an S3 backend with versioning, KMS encryption, a tight bucket policy and locking (DynamoDB, or S3 native locking with use_lockfile = true in Terraform 1.10 and later). Never commit it to Git.

One state per environment: how?

Separate root modules (directories) per environment, each with its own backend key and ideally its own AWS account and pipeline role, rather than workspaces, because a prod apply should not be one variable away from dev. Cross-stack values come from outputs read through terraform_remote_state or SSM parameters.

count versus for_each?

count indexes by position, so removing the middle item shifts every later index and destroys and recreates resources. for_each keys by a map or set value, so changes are surgical. Use for_each for anything with identity (node groups, IAM roles, subnets) and count only as an on/off switch.

How do you organize and version modules?

Small, single-purpose modules in a private registry or Git, referenced with a pinned tag (source = "git::https://...//modules/eks?ref=v2.3.0"), semver released, tested with terraform test, and composed by thin root modules. Provider constraints use ~> in required_providers and the lock file is committed.

How do you detect and handle drift?

A scheduled terraform plan -detailed-exitcode per state (exit code 2 means drift), with the plan posted as a report. Then either reapply to enforce or update the code to accept the change. Prevent it with SCPs and read-only console access; the drift report itself is compliance evidence.

When do you use import and moved?

An import block (Terraform 1.5 and later) adopts an existing resource declaratively, visible in the plan. A moved block renames or relocates a resource in code without destroying it, and removed (1.7 and later) forgets one without deleting it. They replace the older terraform import and state mv commands in a pipeline.

Which lifecycle settings do you actually use?

prevent_destroy on the cluster, KMS keys and state buckets; ignore_changes on desired_size when an autoscaler owns it; create_before_destroy for launch templates and certificates; and preconditions to fail a plan when an input is unsafe.

What is the pitfall with the kubernetes and helm providers?

Creating the cluster and in-cluster resources in the same root module: the provider's endpoint and token are unknown at plan time, so you get plans that fail or default to localhost. Split into a cluster module and an add-ons module, authenticate with an exec block running aws eks get-token, and hand everything past the bootstrap (Argo CD itself) to GitOps.

Describe your Terraform pipeline.

fmt and validate; tflint; a security scan (Checkov or Trivy); plan saved as an artifact and posted to the pull request; policy checks on the plan JSON; a manual approval tied to the change ticket for production; apply from that saved plan on merge; post-apply smoke tests; and a nightly drift run. One apply at a time per state.

How does the pipeline authenticate to AWS?

OIDC federation: GitHub Actions or GitLab presents its identity token and assumes an IAM role whose trust policy restricts the repository and branch through the sub claim. No long-lived keys, and separate roles for plan (read-only) and apply.

What is policy as code in Terraform?

Rules evaluated against the plan before apply: OPA and Conftest on terraform show -json, Sentinel on Terraform Cloud, or Checkov. Typical bank rules: no public S3, EKS endpoint private, encryption on every volume and bucket, mandatory cost and owner tags, no 0.0.0.0/0 ingress.

Why are secrets in state a problem?

State stores every attribute in plaintext, including database passwords and generated keys; sensitive = true only hides output. Mitigate with encrypted backends and tight access, keep secrets in Secrets Manager or Vault and reference them by ARN, and use ephemeral resources and write-only arguments (1.10 and 1.11) so values never land in state.

Terraform versus Ansible?

Terraform is declarative provisioning with state, for cloud and cluster infrastructure. Ansible is agentless, idempotent task execution, for OS configuration, orchestration and day-2 operations; on OpenShift the kubernetes.core collection makes it useful for fleet tasks. Terraform builds the EKS cluster; Ansible or GitOps runs what lives inside.

How would you create ROSA with Terraform?

The terraform-redhat/rhcs provider and the rosa-hcp module create the cluster, machine pools, identity providers and operator roles, with the AWS provider handling VPC and IAM prerequisites. OpenTofu runs the same code if licensing is a concern.

Describe a container build pipeline end to end.

Checkout, unit tests, dependency and static analysis, a multi-stage build with Buildah or Kaniko, an image scan that fails on critical fixable CVEs, an SBOM, a Cosign signature, push with an immutable tag, then a pull request to the config repository that bumps the image reference. Argo CD does the deployment; the pipeline never runs kubectl apply.

Why immutable tags?

Tag by git SHA or semver and deploy by digest, never latest, so what ran on Tuesday is exactly reproducible and rollback means pointing at a known artifact. Turn on tag immutability in ECR or Quay so nobody can overwrite a tag underneath a running deployment.

Where do scanning and signing happen?

Scan in CI (Trivy, Grype or roxctl image check) and again in the registry (Clair in Quay, Inspector for ECR) because new CVEs appear after the build. Sign with Cosign, and verify at admission with Kyverno verifyImages or an ACS policy so unsigned images cannot start in production.

Why separate the application repo from the config repo?

The app repo holds source, Dockerfile and CI; the config repo holds manifests and per-environment values that Argo CD watches. Separation prevents a manifest commit from triggering a rebuild, gives platform and developers different approvers, and makes the config history a clean audit trail of what ran where.

Define GitOps in one breath.

Git is the single declarative source of truth for desired state; an agent in the cluster pulls from it and continuously reconciles, so every change is a reviewed commit and drift is detected and corrected. Pull-based reconciliation, not CI pushing with kubectl.

Name the Argo CD components.

argocd-server (API and UI), argocd-repo-server (clones repositories and renders Helm and Kustomize), argocd-application-controller (compares live to desired and syncs), Dex for SSO, Redis as cache, plus the ApplicationSet and notifications controllers. On OpenShift the GitOps operator installs all of this from an ArgoCD CR in openshift-gitops.

What sync policy do you set on an Application?

syncPolicy.automated with prune: true to delete what Git no longer declares and selfHeal: true to revert manual edits, plus syncOptions such as CreateNamespace=true and ServerSideApply=true. Production often keeps automation on but restricts when it may run through AppProject sync windows.

Sync waves versus hooks?

Waves order resources inside one sync with argocd.argoproj.io/sync-wave (namespaces and CRDs at -1, databases before apps). Hooks run Jobs at phases: PreSync for a schema migration, PostSync for a smoke test, SyncFail for cleanup, with a hook-delete-policy.

App of Apps versus ApplicationSet?

App of Apps is a root Application whose manifests are other Applications, a simple bootstrap pattern with a hand-maintained list. ApplicationSet is a controller that generates Applications from generators (git directories, cluster list, pull requests, matrix), so one template covers every team or every cluster. Fleets use ApplicationSets.

What does an AppProject control?

Which source repositories, destination clusters and namespaces an Application may use, which cluster-scoped kinds it may create, project roles with tokens, and sync windows for change freezes. The default project allows everything, so every team gets its own.

How do you handle secrets in GitOps?

Git holds no secret values. External Secrets Operator pulls from Vault, CyberArk or AWS Secrets Manager into Kubernetes Secrets from an ExternalSecret that is safe to commit; Sealed Secrets or SOPS with KMS are the alternatives when the vault is the problem. Rotation then happens in the vault, not in a pull request.

How do you promote between environments?

One directory or overlay per environment in the config repository; dev is updated automatically by the pipeline, UAT and production by a pull request that bumps the same digest, reviewed and tied to a change ticket. Environment branches drift; directories do not.

How do you roll back in GitOps?

git revert the commit and let Argo CD sync, which keeps the audit trail intact. argocd app rollback is for emergencies only, and with auto-sync on it will be overwritten by Git, so you revert in Git anyway. Database migrations do not roll back on their own, which is why they are forward-compatible.

Why is an Application permanently OutOfSync?

Something else is writing to the live object: an HPA changing replicas, a mutating webhook or API defaults adding fields, an operator managing annotations, or a rotated Secret. Fix with ignoreDifferences, RespectIgnoreDifferences and server-side apply; argocd app diff shows the exact field.

What are Argo Rollouts and Tekton?

Argo Rollouts replaces Deployment with a Rollout that does canary steps (setWeight, pause, analysis against Prometheus) or blue-green with automatic promotion. Tekton is Kubernetes-native CI (Task, Pipeline, PipelineRun, triggers) shipped as OpenShift Pipelines, with Tekton Chains for signed provenance.

What are the DORA metrics?

Deployment frequency, lead time for changes, change failure rate and time to restore service; Argo CD and the incident tool give you all four. A bank tracks them next to change success rate to prove that GitOps made releases safer, not just faster.

Describe the CVE lifecycle and your remediation SLAs.

Discover (registry and ACS scans, Red Hat Security Advisories, the cluster's errata), triage by severity, exploitability and exposure, remediate (rebuild on a patched base image, upgrade the operator, apply the z-stream), verify with a rescan, and file the evidence. Typical bank SLAs are critical within 7 to 15 days, high within 30, medium within 90, with a documented risk acceptance when no fix exists.

What are your image standards?

Approved bases only (Red Hat UBI 9 minimal or distroless), rebuilt on a schedule, non-root, no secrets baked in, pinned by digest, labeled with owner and source, with an SBOM and a signature, scanned in CI and in the registry, and pulled only from allow-listed registries. Anything else is stopped at admission.

Kyverno, Gatekeeper or ACS?

Kyverno writes policies in YAML and can validate, mutate, generate and verify image signatures. Gatekeeper uses OPA Rego through ConstraintTemplates, more powerful and harder to read. ACS is Red Hat's full platform (scanning, admission, runtime, compliance) with support. On OpenShift at a bank you lead with ACS and add Kyverno for mutation and generation.

Which secrets management tools do you know?

HashiCorp Vault (Kubernetes auth, dynamic database credentials, PKI), CyberArk Conjur (common in banks), AWS Secrets Manager on EKS, delivered through External Secrets Operator or the Secrets Store CSI driver. etcd encryption protects the copy at rest; it is not a secrets manager.

How do audit logs reach the SIEM, and what do you alert on?

OpenShift through the ClusterLogForwarder audit input, EKS through CloudWatch and Firehose, both to Splunk or QRadar with retention of a year or more. Alerts: any cluster-admin action, Secret reads outside the owning namespace, SCC and RBAC changes, exec into production Pods, oc debug node, and break-glass logins.

How do you use the CIS benchmarks?

The CIS Red Hat OpenShift benchmark runs through the Compliance Operator's ocp4-cis and ocp4-cis-node profiles; the CIS Amazon EKS benchmark runs with kube-bench, with controls for the managed control plane marked not applicable. Results are scored, tracked over time and presented with justifications for exceptions.

OSFI B-13, PCI DSS and SOC 2 in one line each.

OSFI B-13 is the Canadian regulator's guideline on technology and cyber risk management (governance, resilience, cyber security, third parties), in force since January 2024. PCI DSS 4.0 protects card data with segmentation, logging, quarterly scans and strong access control. SOC 2 is an attestation that your controls for security and availability operate as described, based on evidence you hand over.

How do access reviews work?

Quarterly recertification of cluster-admin, project admin and AD group membership: a script exports bindings and group members, owners attest or remove, break-glass usage is reviewed, and the signed report is retained. Group-based RBAC makes it a membership export instead of a hunt.

What evidence do you give auditors?

Timestamped exports: ClusterRoleBindings and SCC grants, Compliance Operator and ACS results, upgrade records tied to change tickets, etcd backup logs and restore tests, vulnerability trend reports, proof of audit log retention, and the exception register. Generated on a schedule (see Post 32), stored immutably with a hash.

Security incident versus availability incident?

An availability incident is about restoring service fast. A security incident suspects unauthorized access or tampering, so you involve the SOC, preserve evidence (isolate with NetworkPolicy and revoke credentials instead of deleting Pods), maintain chain of custody, and trigger regulatory reporting. Same bridge, different first move.

What is defense in depth on this platform?

Layers that each catch what the previous missed: private networks and endpoints, private-only API access, RBAC and SCCs, NetworkPolicy and EgressFirewall, signed and scanned images enforced at admission, runtime detection with ACS or GuardDuty, encrypted etcd and disks, and audit logs in a SIEM someone reads.

What have you automated on a platform?

The daily health check, the pre-upgrade gate, LDAP group sync, project onboarding from a form, etcd backups, CSR approval for known machines, compliance and access reports, and alert-to-ticket creation. Each one replaced a manual step that was done inconsistently before.

What does your health check script check?

Cluster operators not True/False/False, nodes NotReady, MachineConfigPools degraded or updating, pending CSRs, Pods not Running or Completed in openshift-* namespaces, etcd member health and database size, PVCs Pending, TLS Secrets expiring within 30 days, firing critical alerts from the Alertmanager API, and node disk usage. Output is JSON plus a Markdown summary posted to the team channel.

And the pre-upgrade gate?

The same checks plus: target version listed as recommended in oc adm upgrade, Upgradeable=True, no rows in oc get apirequestcounts for removed APIs, every OLM operator compatible with the target, an etcd backup under 24 hours old, no PDB with zero allowed disruptions, and an approved change ticket number. Any failure exits non-zero and blocks the pipeline.

How do you produce a compliance report automatically?

Aggregate Compliance Operator ComplianceCheckResult objects, the ACS compliance API, SCC grants, cluster-admin bindings, namespaces without NetworkPolicy, images from non-allowed registries and Pods without limits into a CSV and HTML report, run monthly and stored in the evidence bucket with a checksum.

How do you run a script as a CronJob with least privilege?

A dedicated ServiceAccount, a ClusterRole with only get and list on the resources it reads (clusteroperators, nodes, machineconfigpools, pods), a ClusterRoleBinding, and a CronJob using the ose-cli image with concurrencyPolicy: Forbid, restartPolicy: OnFailure, resource limits and history limits. It runs happily under restricted-v2.

How do you use Ansible with OpenShift?

The kubernetes.core collection: k8s to apply objects idempotently, k8s_info to query, k8s_drain for nodes, helm for charts, and redhat.openshift for group sync and Routes. It shines for fleet tasks that mix cluster steps with non-cluster ones (load balancers, ServiceNow), run from Ansible Automation Platform for RBAC and audit of every run.

What does idempotent mean for your scripts?

Running it twice gives the same end state with no extra side effects: declare desired state with oc apply or state: present, check before changing, and fail loudly with a non-zero exit instead of half-applying. It is what makes automation safe to rerun during an incident.

Where does platform automation live?

In a versioned repository with CI and review, executed either as in-cluster CronJobs for cluster-local checks or from AAP or a pipeline runner for fleet-wide tasks, each with its own ServiceAccount, secrets from the vault, logs to the SIEM and a README next to the code.

How does change management work for a platform change?

Standard changes are pre-approved and repeatable; normal changes go to the CAB with an implementation plan, test evidence, backout plan and a window; emergency changes get expedited approval and a retrospective review. A cluster upgrade is a normal change, and freeze periods (month-end, holidays) are planned around.

How do you approach on-call?

A rotation with a clear escalation path, runbooks for every paging alert, response targets by severity (Sev 1 within 15 minutes), a handover note at each shift change, and relentless alert hygiene so pages mean something. If an alert has no action, it is not a page.

How do you work with Red Hat support?

Open the case at the right severity with cluster ID, version, timeline and must-gather; keep the case as the record; use the TAM for Sev 1 escalation and roadmap questions; search the Knowledgebase and Insights first, because half the answers are already there.

How do you collaborate with cloud and architecture teams?

They own the landing zone (accounts, VPCs, transit gateway, IAM boundaries); I bring EKS requirements (subnets, IP ranges, endpoints, roles) to their architecture review, record decisions as ADRs, and agree a RACI so nobody discovers ownership during an incident.

What documentation do you insist on?

Runbooks written as symptom, steps and escalation; architecture diagrams that match reality; the standards and golden path; decision records; and the onboarding guide, all in Git and updated in the same pull request as the change. The test is whether an on-call engineer can follow it at three in the morning.

What do you say when you do not know?

"I don't know that one; here is how I would find out" followed by the method: oc explain or --help, the Red Hat docs and Knowledgebase, a test on OpenShift Local, then what I do know that borders the question. A bank panel would rather hear that than a confident guess.

How do you prioritize competing incidents?

By customer impact and regulatory exposure: a production customer-facing outage beats a degraded production service, which beats a non-production issue blocking a release, which beats a single developer's problem. Say the order, communicate it, and hand off what you cannot hold.

What would your first 90 days look like?

Thirty days learning the clusters, runbooks, people and access while shadowing on-call; sixty days owning a runbook, fixing a recurring alert and automating one check; ninety days leading a non-production upgrade and proposing one standard based on what I have seen hurt the team.