Security, Compliance and Hardening for a Regulated Platform

In a bank, the platform team is audited. Internal audit, external auditors and the regulator all ask the same three questions about every control on the cluster: who owns it, where is the evidence it works, and how fast is it fixed when it fails. Post 22 gave you the OpenShift mechanisms (SCCs, RBAC, OAuth) and Post 28 the EKS controls. This post is the program-level view above both: threat model, frameworks, the hardening checklist, a vulnerability lifecycle you can run, image standards, admission policy, secrets, audit logging, compliance automation, governance and security incident response. Interviewers listen for whether you speak the language of controls, evidence and SLAs. After this post, you will.

Why a bank talks about controls, not tools

Start with vocabulary, because vocabulary separates candidates. A control is a safeguard with a stated purpose: "only images from the internal registry can run in production" is a control; "we use Kyverno" is a tool. Every control has a control owner (a named person or team accountable for it), evidence (proof it operated during the audit period, not merely that it exists) and a remediation SLA (how long a failure may stay open before it is escalated as a risk). When a control cannot be met you file an exception with a justification, an expiry date, a compensating control (something else that reduces the same risk) and a risk-management sign-off.

Banks organise this as three lines of defence. First line: you, the platform team, owning and operating the controls. Second line: risk and compliance, who set policy, review exceptions and challenge your evidence. Third line: internal audit, independent and reporting to the board, testing whether the first two lines do what they claim. Outside all three sit external auditors (SOC 2, financial audits) and the regulator (in Canada, OSFI). Saying "I need to get this control through second line" in an interview marks you as someone who has worked there. Everything below is organised so you can answer, for each control: what it is, how to implement it on OpenShift and EKS, and what you hand the auditor.

The threat model in plain words

Threat model = a written-down answer to "who could hurt us, how would they get in, and what would they reach". Kubernetes security is usually framed with the 4Cs: Cloud (or datacenter: the accounts, VPCs, IAM, hypervisors), Cluster (API server, etcd, nodes, RBAC, network), Container (images, runtime settings, what the process may do) and Code (your application and its dependencies). Each layer can only be as secure as the layer outside it: perfect code in a privileged container on a node with an open cloud role is still a breach waiting to happen.

The attacker paths you should be able to narrate, and the controls that break each chain:

• Compromised image → container escape → node → cloud credentials. A vulnerable library gives remote code execution in a pod; the pod runs as root with extra capabilities, so the attacker escapes to the node and reads the instance metadata service for the node's cloud role. Controls: image scanning and signing, non-root capability-dropped containers (restricted-v2 SCC, restricted PSA), immutable node OS, IMDSv2 with hop limit 1 on EKS so pods cannot reach the node role, least-privilege node IAM.
• Leaked kubeconfig or token in a Git repo or a CI log. Controls: short-lived OAuth tokens through the IdP with MFA, no static tokens, bound service account tokens, secret scanning in CI, alerting on logins from unusual sources, fast revocation.
• Exposed API server or console. Controls: private API endpoint, IdP-only login, kubeadmin removed on day one, network ACLs, WAF on any public route.
• Supply chain: a malicious dependency, poisoned base image, tampered Helm chart, operator or build server. Controls: pinned digests, SBOMs, signed images and provenance verified at admission, allow-listed registries, internal mirrors, curated operator catalogs.
• Insider or over-privileged automation: an admin or a pipeline with cluster-admin doing something it should not. Controls: RBAC least privilege, segregation of duties, audit logs forwarded off-cluster and reviewed, periodic access recertification.

Benchmarks and frameworks: what the auditor is reading from

You will not be asked to recite standards, but you will be asked which ones you have worked with and how they touched the platform. The honest, useful answer is that a handful of frameworks generate almost every control you will implement:

• CIS Benchmarks (Center for Internet Security): numbered configuration checklists. There is a CIS Kubernetes Benchmark, a separate CIS Red Hat OpenShift Container Platform Benchmark (OpenShift's defaults and file paths differ) and a CIS Amazon EKS Benchmark (drops control-plane items you cannot see, adds IAM and node items). The Compliance Operator and kube-bench implement them as automated checks.
• NIST SP 800-190, the Application Container Security Guide: a risk catalogue for images, registries, orchestrators, containers and hosts, and the reference most bank container standards are written from. NIST SP 800-53 is the large control catalogue (access control, audit, configuration management and so on) that the Compliance Operator's ocp4-moderate profile targets.
• PCI DSS 4.x applies wherever cardholder data is handled. Container-relevant requirements: segmentation of the cardholder data environment (Req 1), no vendor defaults (Req 2), critical and high patches typically within one month (Req 6), least privilege (Req 7), MFA (Req 8), audit history retained at least 12 months with three months immediately available (Req 10), regular scans and penetration tests (Req 11).
• SOC 2 is an attestation on the Trust Services Criteria; the platform mostly feeds the Security criteria for logical access (CC6), monitoring and operations (CC7) and change management (CC8). A Type II report samples evidence across a period, so a control that was on in March and off in July fails.
• ISO/IEC 27001 is a certifiable management system whose Annex A controls cover vulnerability, configuration, logging, change and supplier management. If the bank or the vendor is certified, the platform inherits those obligations.
• OSFI Guideline B-13 (Technology and Cyber Risk Management), in effect since 2024, is the Canadian regulator's expectation for federally regulated banks: governance, technology operations and resilience (asset inventory, patching, change, incident management, DR) and cyber security. OSFI also expects reportable technology and cyber incidents to be notified quickly; its advisory speaks of 24 hours.
• OSFI Guideline B-10 (Third-Party Risk Management) governs vendor oversight. When part of the platform is run by a services vendor, B-10 is why your access is scoped, logged and revocable.
• PIPEDA, Canada's federal privacy law: personal information must be safeguarded in proportion to its sensitivity, breaches that create a real risk of significant harm must be reported, and you must be able to say where personal data lives (namespaces, volumes, logs). Provincial laws such as Quebec's Law 25 add requirements.

What an auditor actually asks for

An auditor never asks "is the cluster secure". They ask for four things per control: the policy that requires it (a document with an owner and review date); the design (how it is implemented, ideally a config export or diagram); operating evidence that it worked throughout the period (dated exports, screenshots, scan reports, ticket numbers, and for periodic controls such as access reviews or restore tests the sign-off from each occurrence); and a population and sample: "give me every production change this quarter; I will pick 25 and you show me each approval". If you cannot produce the population, the auditor assumes the control did not operate.

The hardening checklist

Here is the checklist a bank's platform standard usually boils down to, with the OpenShift and EKS implementation side by side. Each row is a control; the tool is only how you meet it. Post 22 and Post 28 have the detailed commands for most rows.

Vulnerability management as a process you can run

Scanning is the easy part. Vulnerability management is the loop around it: know what you have, scan it, decide what matters, fix within SLA, prove it is fixed, and report. Interviewers want to hear the loop, because a scanner with no loop just produces a growing PDF nobody reads.

1. Inventory

You cannot scan what you do not know about. The inventory has five layers: application images by digest (oc get pods -A -o jsonpath over .status.containerStatuses[].imageID), the base images they were built from, operators and add-ons (oc get csv -A, the EKS add-on API), node OS (RHCOS version or AMI ID per node) and cluster version. It doubles as your OSFI B-13 asset register, so keep it as data written by a script (Post 32), not as a wiki page.

2. Scan

Scan at every stage, because each stage finds different things: in CI with Trivy or Grype, failing the pipeline on policy; in the registry with Quay and Clair (the Red Hat pairing) or ECR enhanced scanning backed by Amazon Inspector, which rescans stored images as new CVEs publish and catches yesterday's clean image becoming today's critical; in the cluster with Red Hat Advanced Cluster Security (ACS) or the Quay Container Security Operator (which surfaces ImageManifestVuln objects per namespace), showing what actually runs; and at cluster and node level, where OpenShift Insights and Red Hat CVE data say which errata apply to your version, ACS can scan RHCOS nodes, and on EKS Inspector covers the EC2 nodes.

$ trivy image --severity HIGH,CRITICAL --ignore-unfixed \
    registry.bank.internal/payments/ledger-api:1.14.2

registry.bank.internal/payments/ledger-api:1.14.2 (redhat 9.4)
==============================================================
Total: 3 (HIGH: 2, CRITICAL: 1)

Library         Vulnerability     Severity   Installed        Fixed            Title
openssl-libs    CVE-2024-5535     CRITICAL   3.0.7-27.el9     3.0.7-28.el9_4   openssl: SSL_select_next_proto buffer overread
glibc           CVE-2024-33599    HIGH       2.34-100.el9     2.34-100.el9_4.2 glibc: stack-based buffer overflow in netgroup cache
curl-minimal    CVE-2024-7264     HIGH       7.76.1-29.el9    7.76.1-29.el9_4.1 curl: libcurl ASN.1 date parser overread

usr/local/bin/ledger-api (gobinary)
===================================
Total: 1 (HIGH: 1, CRITICAL: 0)

Library                Vulnerability     Severity   Installed   Fixed     Title
golang.org/x/net       CVE-2023-44487    HIGH       v0.14.0     v0.17.0   HTTP/2 rapid reset

Read it in two halves. The OS section says the UBI base is stale: every fix is already in a newer el9_4 package, so a rebuild on the current base clears it. The binary section says the application's Go dependency is old, which only the app team can fix in go.mod. That split, platform fixes the base and the team fixes the dependency, is how remediation gets assigned. --ignore-unfixed hides CVEs with no vendor fix yet; keep those visible in reports but do not fail builds on them.

3. Triage

CVSS alone is a poor prioritiser: a 9.8 in a library your process never calls is less urgent than an exploited 7.5 on an internet-facing service. Triage on four axes: severity (CVSS, and Red Hat's own rating for RHEL packages, often lower than NVD because of how the package is built); exploitability (is it in CISA's KEV catalogue of known exploited vulnerabilities, and what is its EPSS probability of exploitation in the next 30 days); exposure (internet-facing, handles card or personal data, runs privileged); and fix availability. Be precise about "vulnerable package present" versus "reachable": the package is in the image, but is the vulnerable function on any code path? Reachability analysis and vendor VEX statements ("not affected by CVE-X because...") let you downgrade findings with evidence rather than opinion.

4. Remediate within SLA

Bank standards typically set SLAs by severity, commonly along the lines of critical within 7 to 15 days, high within 30, medium within 90, low on the next cycle, with internet-facing systems at the tight end. Say "typically" in the interview; the exact numbers are the bank's policy. What matters is knowing the levers:

• Rebuild on the updated base. Most OS-level CVEs vanish when the base image (UBI 9 minimal, say) is bumped and the image rebuilt. A platform-owned base image rebuilt weekly, with a pipeline that rebuilds every downstream image when the base changes, clears most findings without touching application code.
• Upgrade operators and add-ons through OLM channels (Post 23) or the EKS add-on API.
• z-stream cluster upgrades, which carry RHCOS and component CVE fixes; this is why "within 30 days of the latest z-stream" is a control (Post 20).
• MachineConfig or launch template changes for kernel parameters and mitigations that need a host setting.
• Compensating controls when no fix exists: a NetworkPolicy that removes exposure, a WAF rule, a disabled feature, with an exception on file.

5. Verify and report

Closing a ticket means re-scanning the new digest and attaching the clean result. Reporting is per team (their images, their SLA clock, their exceptions) and per control owner (platform-wide aging: how many criticals are past SLA). A build-failing policy stops new debt entering; the report shows existing debt shrinking.

# .trivyignore.yaml - exceptions with an expiry, reviewed by security
vulnerabilities:
  - id: CVE-2023-44487
    paths:
      - "usr/local/bin/ledger-api"
    expired_at: 2026-10-31
    statement: "Not reachable: HTTP/2 disabled in server config. Exception EXC-2026-0142, compensating control: router-level rate limit."

# GitHub Actions / Jenkins step: block the build on fixable HIGH and CRITICAL
$ trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed \
    --ignorefile .trivyignore.yaml \
    --format json --output trivy-report.json \
    registry.bank.internal/payments/ledger-api:${GIT_SHA}
$ echo "exit code: $?"
exit code: 1

Exit code 1 fails the job. The JSON report is archived as a build artifact, which doubles as audit evidence that the scan ran on that commit. A weekly report across all images is a small script over the same JSON (Post 32 builds the full version); the core of it fits in one jq line:

$ for f in reports/*.json; do
    jq -r --arg img "$f" '.Results[] | .Vulnerabilities[]? |
      [$img, .VulnerabilityID, .Severity, .PkgName, .InstalledVersion, (.FixedVersion // "none")] | @csv' "$f"
  done > weekly-cves.csv
$ head -3 weekly-cves.csv
"reports/ledger-api.json","CVE-2024-5535","CRITICAL","openssl-libs","3.0.7-27.el9","3.0.7-28.el9_4"
"reports/ledger-api.json","CVE-2024-33599","HIGH","glibc","2.34-100.el9","2.34-100.el9_4.2"
"reports/card-gateway.json","CVE-2024-5535","CRITICAL","openssl-libs","3.0.7-27.el9","3.0.7-28.el9_4"

Container image standards

An image standard is the document that says what every image on the platform must look like, so scanning and admission have something to enforce. The rules in almost every bank standard:

• Minimal, supported base: UBI minimal or UBI micro (supported, with Red Hat CVE data) or distroless. Fewer packages, fewer CVEs.
• Non-root, read-only root filesystem, no privilege escalation, all capabilities dropped. restricted-v2 enforces most of this and assigns an arbitrary UID, so images must not depend on a fixed UID or write to unwritable paths.
• No secrets baked in: not in layers, build args or ENV. A secret in a deleted layer is still in the image history.
• Pinned by digest, both the base in the Dockerfile and the image in the manifest. Tags move; digests do not.
• SBOM per image. A Software Bill of Materials lists every package and dependency with versions, in CycloneDX or SPDX format, generated by syft or Trivy at build time and stored with the image. When the next big CVE lands, "which of our 900 images contain package X" becomes a query over SBOMs.
• Signed and attested. cosign (Sigstore) signs the digest; attestations attach the SBOM and build provenance. SLSA grades how trustworthy that provenance is: hardened hosted builder, signed provenance, no single person able to tamper.
• Dependency scanning of go.mod, pom.xml or package-lock.json in CI, and hadolint on the Dockerfile so latest tags and a final USER root never reach a build.

# Dockerfile - multi-stage, UBI, non-root, digest-pinned
FROM registry.access.redhat.com/ubi9/go-toolset@sha256:4f1b2c...e9a0 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/ledger-api ./cmd/ledger-api

FROM registry.access.redhat.com/ubi9/ubi-micro@sha256:7d3e91...c2b4
LABEL org.opencontainers.image.source="https://git.bank.internal/payments/ledger-api" \
      com.bank.owner="payments-platform" \
      com.bank.data-classification="confidential"
COPY --from=build /out/ledger-api /usr/local/bin/ledger-api
USER 1001
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/ledger-api"]

Notice what is missing: no package manager in the final stage (ubi-micro has none), no shell needed, no ENV DB_PASSWORD, and USER 1001 so hadolint's "last user should not be root" rule passes and the process still works under OpenShift's random UID because 1001 is only a hint. The labels are how ownership shows up in scan reports later.

$ hadolint Dockerfile
Dockerfile:1 DL3007 warning: Using latest is prone to errors ... (only if you had used :latest)
$ syft registry.bank.internal/payments/ledger-api:1.14.2 -o cyclonedx-json > sbom.cdx.json
$ cosign sign --key cosign.key registry.bank.internal/payments/ledger-api@sha256:9c1f...a7e2
$ cosign attest --key cosign.key --type cyclonedx --predicate sbom.cdx.json \
    registry.bank.internal/payments/ledger-api@sha256:9c1f...a7e2
$ cosign verify --key cosign.pub registry.bank.internal/payments/ledger-api@sha256:9c1f...a7e2
Verification for registry.bank.internal/payments/ledger-api@sha256:9c1f...a7e2 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Signing is pointless unless something refuses unsigned images. That verification happens at admission, which brings us to policy engines. Here is the Kyverno rule that makes "signed images only" a control rather than a habit:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-bank-image-signatures
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: require-cosign-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaceSelector:
                matchLabels:
                  bank.internal/tier: prod
      verifyImages:
        - imageReferences:
            - "registry.bank.internal/*"
          required: true
          mutateDigest: true
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----

mutateDigest: true rewrites the tag to the verified digest, so what was verified is exactly what runs. Red Hat ACS can enforce the same with its signature integration, and OpenShift itself has been adding native sigstore verification through ClusterImagePolicy and ImagePolicy objects (configured through CRI-O's policy rather than a webhook); it was Tech Preview through several 4.1x releases, so check the release notes and oc explain clusterimagepolicy for your version before promising it to an interviewer.

Admission control and policy as code

Admission control is the API server's last checkpoint: after authentication and RBAC, before the object is written to etcd, admission logic can reject (validate), change (mutate) or create related objects (generate). Policy as code means those rules live in Git, are reviewed like any change, and are applied by Argo CD (Post 30), so the auditor can see who changed a rule and when. Four engines matter here:

• Kyverno: policies are plain Kubernetes YAML. Validate rules reject or flag; mutate rules fix (add a default seccompProfile, add the team label); generate rules create objects (the default-deny NetworkPolicy in every new namespace). A typical bank set: require owner and cost-centre labels, disallow latest, require probes, require limits, restrict registries, disallow hostPath, host networking and privileged containers, verify signatures.
• OPA Gatekeeper: rules in Rego, packaged as a ConstraintTemplate (logic) plus a Constraint (parameters and scope), with enforcementAction of deny, warn or dryrun.
• ValidatingAdmissionPolicy: built into Kubernetes (GA since 1.30, so OpenShift 4.17 onward), no webhook to run, rules in CEL (Common Expression Language). Fast and always-on; cannot mutate or generate.
• Red Hat ACS: policies span build (roxctl image check fails the pipeline), deploy (admission controller) and runtime (process and network behaviour), with a large default set ("Fixable Severity at least Important", "Latest tag", "Privileged Container", "Kubernetes Actions: Exec into Pod") mapped to CIS, PCI and NIST 800-190 in its compliance view.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: bank-baseline
spec:
  validationFailureAction: Audit      # flip to Enforce per namespace tier after rollout
  background: true
  rules:
    - name: disallow-latest-tag
      match: { any: [ { resources: { kinds: ["Pod"] } } ] }
      validate:
        message: "Image tags must be pinned; ':latest' or untagged images are not allowed."
        pattern:
          spec:
            containers:
              - image: "!*:latest & *:*"
    - name: require-limits
      match: { any: [ { resources: { kinds: ["Pod"] } } ] }
      validate:
        message: "CPU and memory limits are required (Post 09)."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"
    - name: restrict-registries
      match: { any: [ { resources: { kinds: ["Pod"] } } ] }
      validate:
        message: "Only registry.bank.internal images may run."
        pattern:
          spec:
            containers:
              - image: "registry.bank.internal/*"

And the same "no privileged containers" idea as a built-in ValidatingAdmissionPolicy, which is worth knowing because it needs no operator and survives a webhook outage:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: no-privileged-containers
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "object.spec.containers.all(c, !has(c.securityContext) || !has(c.securityContext.privileged) || !c.securityContext.privileged)"
      message: "Privileged containers are not permitted on this platform."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: no-privileged-containers-prod
spec:
  policyName: no-privileged-containers
  validationActions: ["Audit"]          # then ["Warn"], then ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        bank.internal/tier: prod

Rolling out policy without breaking teams

The rollout strategy is what interviewers listen for, because everyone has seen a well-meant "enforce" take down a deployment on a Friday. Ship the policy in audit mode and collect violations for two to four weeks; send each team its report with a deadline; exempt namespaces that need it through an exception with expiry, never by weakening the rule; switch to warn so developers see the message in their CLI and pipeline; then enforce on new namespaces first, then dev, non-prod and prod, each announced through change management. Exclude the platform's own namespaces (openshift-*, kube-*) from mutation and from any rule not tested against operator-managed pods, and choose the webhook failurePolicy deliberately: Ignore keeps the cluster deployable if the engine is down, Fail is stricter but means a Kyverno outage blocks every pod.

Secrets management

The rules are short: secrets never in Git, never in images, never in logs or env dumps, never in a ticket. Kubernetes Secrets (Post 6) are base64-encoded, not encrypted; anyone with get secrets in the namespace reads them. So a bank adds a central secrets manager as the source of truth, an integration that delivers secrets into pods without a human copying them, and encryption at rest in etcd (Post 22). The central store is usually HashiCorp Vault, CyberArk (very common in banks for privileged credentials), AWS Secrets Manager or Azure Key Vault. The integration patterns:

• External Secrets Operator (ESO) syncs from the vault into a normal Kubernetes Secret on a refresh interval. Apps need no change, which is why it is the most common pattern.
• Secrets Store CSI driver mounts secrets as files straight from the vault into the pod; nothing lands in etcd unless you ask.
• Vault Agent injector adds a sidecar that authenticates, renders secrets to a shared volume and renews them; good for dynamic database credentials.
• Vault Kubernetes auth underlies all three on OpenShift: the pod's service account token proves identity, and a Vault role binds that service account and namespace to a policy. On EKS, ESO or the CSI driver use Pod Identity or IRSA to reach Secrets Manager.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-payments
  namespace: payments-prod
spec:
  provider:
    vault:
      server: "https://vault.bank.internal:8200"
      path: "kv"
      version: "v2"
      caProvider:
        type: ConfigMap
        name: bank-root-ca
        key: ca.crt
      auth:
        kubernetes:
          mountPath: "ocp-prod-east"
          role: "payments-prod-reader"
          serviceAccountRef:
            name: "eso-vault-auth"

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ledger-db
  namespace: payments-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-payments
    kind: SecretStore
  target:
    name: ledger-db            # the Kubernetes Secret ESO creates and owns
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: payments/ledger/db
        property: password
    - secretKey: DB_USER
      remoteRef:
        key: payments/ledger/db
        property: username

Read the two objects together. The SecretStore is namespaced, so the payments-prod team's store can only authenticate as the eso-vault-auth service account, and the Vault role payments-prod-reader is bound to exactly that service account in exactly that namespace, with a read-only policy on kv/payments/*. That is least privilege per namespace: a compromised pod in another project cannot borrow this store. Older ESO installs serve external-secrets.io/v1beta1; oc api-resources | grep external-secrets tells you which. On the Vault side, the role looks like vault write auth/kubernetes/role/payments-prod-reader bound_service_account_names=eso-vault-auth bound_service_account_namespaces=payments-prod policies=payments-read ttl=1h.

The rest is hygiene you should be able to list: rotation on a schedule and immediately on staff departure or suspected leak, with ESO's refresh propagating the value and a rolling restart picking it up; leak detection with gitleaks or trufflehog (gitleaks protect --staged as a pre-commit hook, gitleaks detect over full history in CI, and a server-side push rule where the Git platform supports it); break-glass credentials held in the privileged access manager, checked out with approval, used only in a declared incident, rotated after every use and alerted on every login; and service account token hygiene: automountServiceAccountToken: false for pods that never call the API, bound projected tokens with short expiry instead of legacy long-lived Secret tokens, and a periodic sweep for stale kubernetes.io/service-account-token Secrets.

Audit logging and security monitoring

An audit log answers "who did what, to which object, from where, and did it succeed" for every API request. It is the most requested artifact in a platform audit, because access evidence, change evidence and incident forensics all come from it. Three things make it a control: it is on with enough detail, it is forwarded off-cluster to the SIEM within minutes so an attacker with node access cannot erase it, and someone alerts and reviews on it.

On OpenShift the audit policy is set on the cluster APIServer object. Default logs metadata for all requests plus bodies for login events; WriteRequestBodies adds the body of every write (what a bank usually wants, so a RoleBinding change shows the new subjects); AllRequestBodies adds reads and is very noisy; None is never acceptable. Forwarding is a ClusterLogForwarder pipeline with the audit input (Post 24).

apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  name: cluster
spec:
  audit:
    profile: WriteRequestBodies
---
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: audit-to-siem
  namespace: openshift-logging
spec:
  serviceAccount:
    name: audit-collector
  outputs:
    - name: splunk-hec
      type: splunk
      splunk:
        url: https://hec.splunk.bank.internal:8088
        authentication:
          token:
            secretName: splunk-hec-token
            key: hecToken
  pipelines:
    - name: audit
      inputRefs: ["audit"]
      outputRefs: ["splunk-hec"]

Field names differ between Logging 5 (logging.openshift.io/v1) and Logging 6 (observability.openshift.io/v1), so oc explain clusterlogforwarder.spec.outputs before you type this on a real cluster. On EKS the equivalent is control plane logging to CloudWatch, then a subscription filter or Kinesis Firehose into Splunk, QRadar or Sentinel:

$ aws eks update-cluster-config --name prod-eks-east --logging \
    '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
$ aws logs describe-log-groups --log-group-name-prefix /aws/eks/prod-eks-east/cluster \
    --query 'logGroups[].{name:logGroupName,retentionDays:retentionInDays}'
[ { "name": "/aws/eks/prod-eks-east/cluster", "retentionDays": 400 } ]

The alerts a bank SOC expects the platform team to define, because only you know what "unusual" means on a cluster: a cluster-admin ClusterRoleBinding created or changed; a Secret read by a human in production or by an identity that has never read it; exec or attach into a production pod; a privileged or hostPath pod admitted; an SCC granted (a RoleBinding to a system:openshift:scc:* role); repeated failed logins; API calls from outside the admin ranges; audit logging itself being reconfigured. Each is a filter on the audit stream. Here is the shape, run against a master's audit file when the SIEM is not in front of you:

$ oc adm node-logs --role=master --path=kube-apiserver/audit.log \
    | jq -c 'select(.objectRef.resource=="clusterrolebindings" and .verb=="create")
             | {t:.requestReceivedTimestamp, user:.user.username, ip:.sourceIPs[0],
                name:.objectRef.name, role:.requestObject.roleRef.name}'
{"t":"2026-09-04T14:02:11Z","user":"jdoe@bank.internal","ip":"10.40.7.21",
 "name":"tmp-admin-jdoe","role":"cluster-admin"}

That line, correlated with the change system, is either an approved change or an incident. The correlation is itself a control: "cluster-admin binding created AND no approved change window for that user" pages the SOC; the same event inside an approved window is logged and reviewed, not paged.

Beyond the API: node logs (SSH should be disabled, so any sshd login is an alert) and runtime security, which watches process and network behaviour inside containers: ACS runtime policies, Falco (rules such as "Terminal shell in container"), or Amazon GuardDuty EKS Protection with findings like PrivilegeEscalation:Kubernetes/PrivilegedContainer and its Runtime Monitoring agent. Retention is commonly 12 months with the latest three months searchable online, the PCI DSS wording many bank standards adopt. Log integrity means the SIEM is append-only for platform staff, storage is object-locked, and forwarding lag is monitored so a gap is itself an alert.

Compliance automation and reporting

The JD's "automate compliance reporting" means turning evidence collection into scheduled jobs. On OpenShift the engine is the Compliance Operator. It installs from OperatorHub into openshift-compliance and ships ProfileBundles that unpack into Profiles: ocp4-cis and ocp4-cis-node (the CIS OpenShift Benchmark, split into platform checks against the API and node checks that run on each node), ocp4-pci-dss and ocp4-pci-dss-node, ocp4-moderate and ocp4-moderate-node (NIST 800-53 moderate), rhcos4-* for the host OS, and others such as ocp4-stig. A TailoredProfile disables or adjusts rules with a documented reason, which is how approved exceptions are encoded. A ScanSetting holds schedule, node roles and result storage; the built-in default runs nightly and default-auto-apply also applies remediations. A ScanSettingBinding ties profiles to a setting and creates the ComplianceSuite and ComplianceScan objects that do the work.

apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: bank-cis-pci
  namespace: openshift-compliance
profiles:
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-pci-dss
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1

$ oc get compliancesuites -n openshift-compliance
NAME           PHASE   RESULT
bank-cis-pci   DONE    NON-COMPLIANT

$ oc get compliancecheckresults -n openshift-compliance \
    -l compliance.openshift.io/check-status=FAIL
NAME                                                          STATUS   SEVERITY
ocp4-cis-audit-log-forwarding-enabled                         FAIL     medium
ocp4-cis-api-server-encryption-provider-cipher                FAIL     medium
ocp4-cis-kubeadmin-removed                                    FAIL     medium
ocp4-cis-scc-limit-privileged-containers                      FAIL     medium
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults   FAIL     medium

$ oc get complianceremediations -n openshift-compliance
NAME                                                          STATE
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults   NotApplied
ocp4-cis-api-server-encryption-provider-cipher                NotApplied

Each ComplianceCheckResult carries a description and, often, manual check instructions; oc describe compliancecheckresult ocp4-cis-kubeadmin-removed says exactly what to fix. Statuses are PASS, FAIL, MANUAL (needs a human, such as "an IdP with MFA is configured"), INFO, NOT-APPLICABLE, ERROR or INCONSISTENT (nodes disagree, a finding in itself). Where the operator knows the fix it creates a ComplianceRemediation; node remediations become MachineConfigs, so applying them means a rolling reboot through the MachineConfigPool (Post 20). That is why production usually keeps autoApplyRemediations off and applies through a change ticket in a window, while default-auto-apply is fine in dev. Apply one with oc patch complianceremediation <name> -n openshift-compliance -p '{"spec":{"apply":true}}' --type=merge, wait for the pool, then oc compliance rerun-now scansettingbindings bank-cis-pci.

The oc compliance plugin (the oc-compliance package or krew) is the reporting tool: oc compliance fetch-raw scansettingbindings bank-cis-pci -o ./results pulls the raw ARF/XCCDF files auditors like, oc compliance controls profile ocp4-cis maps each rule to its NIST or CIS control ID, and oc compliance view-result prints one check with rationale and fix. Post 32's weekly job does oc get compliancecheckresults -o json, groups by severity and status, diffs against last week, writes CSV and HTML and attaches both to the compliance ticket.

On EKS the equivalent is AWS Config managed rules and conformance packs (endpoint not public, secrets encrypted with KMS, logging enabled, supported version), Security Hub with its EKS controls, and kube-bench run as a Job against the CIS EKS Benchmark for node checks Config cannot see. Kyverno and Gatekeeper write PolicyReport and ClusterPolicyReport objects (the wgpolicyk8s.io CRDs), so oc get policyreport -A gives pass/fail per namespace for the per-team view; ACS does the same in its Compliance and Vulnerability dashboards with scheduled report emails.

The finding lifecycle is the same whichever tool produced it: detect (scan result or policy report) → ticket (auto-created with check ID, severity and evidence) → owner (from the control owner register or the namespace's owner label) → fix (change ticket if production) → verify (re-scan shows PASS) → close with evidence (PASS result and change record linked). A CronJob in a platform namespace runs the exports weekly into the evidence bucket, so "show me March" is a folder listing. Control owners get a dashboard with one number per control: days since last PASS.

Network and data protection

Segmentation is the control auditors draw on a whiteboard. Inside the cluster it is NetworkPolicy (default-deny per namespace, explicit allows between tiers) and, on OpenShift, router sharding: separate IngressController instances selected by namespace or route labels and pinned to their own nodes, so a DMZ shard serves internet-facing routes from the DMZ subnet and a PCI shard serves only cardholder-data namespaces. Outbound, EgressFirewall restricts what a project may reach and EgressIP gives a namespace a fixed source address the corporate firewall can allow-list (Post 21). Where service-to-service encryption and identity are required, a service mesh (OpenShift Service Mesh, Istio-based) provides mTLS with PeerAuthentication set to STRICT.

Data classification makes those policies systematic: every namespace carries a label such as bank.internal/data-class: restricted, set at onboarding (Post 26), and Kyverno generate rules, router shard selectors and quota tiers key off it. "Which namespaces hold cardholder data" is a label query, not a spreadsheet.

Encryption in transit: TLS at the router for ordinary apps, re-encrypt routes carrying TLS to the pod for restricted data, cert-manager issuing internal certificates. Encryption at rest: EBS with a customer-managed KMS key in the StorageClass on EKS, OpenShift Data Foundation cluster-wide encryption with keys in Vault, and etcd encryption from Post 22. DLP for logs is yours too: no PII or card numbers in application logs, redaction or drop rules in the forwarder, retention by data class, and log access RBAC-restricted and audited like any other sensitive read. Backups are encrypted (OADP or Velero to object storage with SSE-KMS, etcd snapshots to an encrypted bucket) and restore-tested quarterly with a signed report, a standing audit item under OSFI B-13 and SOC 2 Availability.

Governance: the part of the job that is not YAML

Governance is the set of processes that make the technical controls provable. You are expected to operate inside them and to talk about them without sounding bored.

• Segregation of duties. Developers reach production only through the pipeline; the pipeline identity can deploy but not change RBAC or policy; platform admins can change the cluster but every action is logged and reviewed, with a second approver for high-risk changes. Argo CD with Git as the only path to production implements this without slowing teams (Post 30).
• Change management. Standard changes are pre-approved and repeatable (a routine z-stream on a non-critical cluster); normal changes go to the CAB (Change Advisory Board) with plan, test evidence, backout and window; emergency changes are approved after the fact during an incident. The auditor samples the ticket, the approval and the audit log entries inside the window.
• Access reviews. Quarterly recertification of cluster-admin and project admin roles: an export of who holds what, confirmed or revoked by the owner, sign-off stored. The most common evidence request, so keep the export scripted.
• Exception register with owner, risk rating, compensating control, approver and expiry; expired exceptions are re-approved or the finding reopens.
• Vendor controls: named accounts for vendor staff, access scoped to the engagement, activity logged, offboarding within a set number of days (B-10 territory).
• DR/BCP tests, penetration test readiness (scope, current inventory, a contact for the testers, findings triaged through the CVE lifecycle), security champions in application teams who own their findings, and a platform security roadmap that turns "we should sign images" into a funded quarter with a control owner.

$ oc get clusterrolebindings -o json | jq -r '
    .items[] | select(.roleRef.name=="cluster-admin") | .metadata.name as $b
    | .subjects[]? | [$b, .kind, .name] | @tsv'
cluster-admin                      Group   system:masters
cluster-admins                     Group   platform-cluster-admins
argocd-platform-cluster-admin      ServiceAccount   openshift-gitops/argocd-application-controller
tmp-admin-jdoe                     User    jdoe@bank.internal

That export is the quarterly recertification population. The last line is what the review exists to catch: a temporary binding that outlived its change window. The same query on EKS is the union of aws eks list-access-entries (with their access policies) and the cluster's RoleBindings.

Incident response for security events

Post 25 covered availability incidents, where the goal is to restore service fast. A security incident reorders the priorities: preserve evidence before you fix (do not delete the pod or rebuild the node; snapshot, export logs, capture the pod YAML and process list), isolate rather than destroy (a NetworkPolicy that blocks all traffic to the pod, cordon and taint the node, revoke the identity), rotate anything the attacker could have reached (tokens, secrets in the namespace, node role credentials), and escalate to the security operations centre or CSIRT immediately, because they own the investigation, the regulatory clock (OSFI's advisory expects reportable incidents notified within 24 hours; PIPEDA has its own breach rules) and communications. Your job is to be their hands on the platform and keep a timestamped log of everything you do.

Playbook: a secret was committed to Git

1. Treat it as compromised the moment it reached the remote; rewriting history does not un-leak it.
2. Identify what it unlocks and rotate first: new value in Vault, ESO refresh, rolling restart of consumers, confirm health on the new credential.
3. Revoke the old value at the provider, then check the audit and provider logs for any use between commit and revocation; hand that to the SOC.
4. Purge it from history (git filter-repo) under a change ticket, refresh every clone, and confirm the CI scanner and pre-commit hook would have caught it; fix the gap if not.
5. Record the incident and close with evidence: rotation timestamp, revocation confirmation, log review outcome, preventive change.

Playbook: a critical CVE in a base image used by 200 services

1. Scope in minutes: query the SBOM store (or the registry scanner, or ImageManifestVuln objects) for every image containing the package, joined to the running-image inventory for namespaces, owners and exposure.
2. Triage with KEV, EPSS and reachability to set the SLA; an exploited, internet-facing critical is "days" and may be declared an emergency change.
3. Fix the base once: rebuild the platform base on the patched UBI, sign it, publish it, trigger the downstream rebuild pipeline; teams off the managed base get a direct ticket with the one-line Dockerfile change.
4. Prioritise by exposure; services that cannot rebuild in time get a compensating control (NetworkPolicy, WAF rule, feature disabled) and a short-expiry exception.
5. Verify by re-scanning running digests, report daily to the control owner and the SOC until zero, then review why 200 services shared a base that was not rebuilt weekly.

Likely interview questions

Next up: the scripts behind all of this: Python, shell and Ansible against the Kubernetes API to automate health checks, evidence exports, the weekly CVE and compliance reports, and the operational chores that eat a platform team's week.