Automation for Platform Ops: Python, Shell, Ansible and the Kubernetes API

Every platform JD now carries the phrase "automation-first mindset", and every interviewer has a version of the same question: "Tell me about something you automated." A vague answer ("I wrote some scripts") loses the round; five specific answers with a measurable result each win it. You already write Python and shell. This post turns that into a portfolio of platform automations you can describe from memory: a cluster health check, a pre-upgrade gate, a compliance report, a quota report, a certificate-expiry alert, plus the scheduling, RBAC and engineering habits that make an interviewer believe you have run these in a regulated production environment. All the code is real enough to put in a repo tonight.

What "automation-first" means when a bank writes it

In a JD, automation-first is shorthand for a specific habit: if you did a task by hand twice, the third time it is a script in Git, run by a scheduler, producing a report that someone reads. Not "I could script that", but "it is scripted, it runs at 06:00, and here is last Tuesday's output". The JD lists the targets explicitly: operational tasks, health checks, validation, compliance reporting and platform deployments. Each of those is a section below.

The interview version of this bullet is the question "what have you automated?" followed by "how did you make it safe to run in production?" You need five crisp stories, each with the manual pain, the thing you built, the number it changed, and the incident it prevented. The last section gives you the template; the middle sections give you the material.

The automation ladder

Not every task deserves an operator. The mistake juniors make is building too much; the mistake seniors make is stopping at a shell script forever. Think of it as a ladder, and be able to say which rung a given task belongs on and why.

Rules that apply on every rung

• Idempotent = running it twice leaves the same end state as running it once. oc apply is idempotent, oc create is not, and anything that changes state must be safe to re-run after a half-failure.
• Dry-run by default. Destructive actions need an explicit --apply; without it the script prints what it would do. oc delete --dry-run=client and oc apply --dry-run=server give you this for free.
• Exit codes mean something. 0 = healthy, non-zero = attention. Schedulers, CI and alerts key off the exit code; a check that prints "ERROR" and exits 0 is invisible.
• Logs to stdout with timestamp and level. In a pod, stdout is what Vector ships to Loki (Post 24); a log file inside the container is lost.
• Timeouts everywhere: oc --request-timeout=30s, curl --max-time, activeDeadlineSeconds on Jobs. Hanging on a sick API server is worse than failing fast.
• Secrets never live in the script. In-cluster, the pod's ServiceAccount token is the credential; outside, tokens come from Vault, the CI secret store, or oc create token --duration=10m. A webhook URL in Git is an audit finding.
• Least-privilege identity: a dedicated ServiceAccount per automation, bound to a ClusterRole with exactly the verbs it needs, which for a report is get and list. Never cluster-admin, never the default SA.
• Packaged as a container image with pinned base and dependencies, run by a CronJob with that SA. That is what "cloud-native automation tooling" means: the automation is a workload, deployed and observed like any other.

Shell, oc, jq and yq: the working patterns

Most platform automation is 70 percent "get JSON from the API server, filter it, print it". The tools are oc (or kubectl), jq (JSON query language) and yq (the same idea for YAML). Learn these six shapes and you can write almost any check.

$ oc get pods -A -o json | jq -r '.items[] | select(.status.containerStatuses[]?.state.waiting.reason=="CrashLoopBackOff") | "\(.metadata.namespace)/\(.metadata.name)"'
$ oc get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'
$ oc get pods -A --field-selector=status.phase=Pending -l app.kubernetes.io/part-of=payments
$ oc get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName --sort-by=.metadata.creationTimestamp | tail -5
$ oc get ns -o name | cut -d/ -f2 | xargs -n1 -P4 -I{} sh -c 'oc get quota -n {} --no-headers 2>/dev/null | sed "s|^|{} |"'
$ oc get deploy payments-api -o yaml | yq '.spec.template.spec.containers[].image'

-o json | jq is the general tool; -o jsonpath is faster for one field with no jq dependency; --field-selector filters server-side, but only on a few fields such as status.phase, spec.nodeName and metadata.name; -l filters on labels; custom-columns gives a table with exactly your report's columns; xargs -P parallelises across namespaces. Every script below starts with set -euo pipefail (exit on error, on unset variable, and on any failure inside a pipeline) and an ERR trap that prints the failing line.

Script (a): the cluster health check

This is the one you will describe first in the interview. It looks at the same things you would look at by hand at the start of an incident (Post 25) and prints one table.

#!/usr/bin/env bash
# cluster-health.sh: prints a table, exits 1 if anything needs attention
set -euo pipefail
trap 'echo "ERROR at line $LINENO" >&2' ERR
oc() { command oc --request-timeout=30s "$@"; }
RC=0
report() {                      # report AREA "<problem lines or empty>"
  if [[ -z "$2" ]]; then printf '%-12s %-52s OK\n' "$1" "-"
  else RC=1; while IFS= read -r l; do printf '%-12s %-52s ATTENTION\n' "$1" "$l"; done <<< "$2"; fi
}
report ClusterOps "$(oc get co -o json | jq -r '.items[] | select(.status.conditions[]
  | (.type=="Available" and .status!="True") or (.type=="Degraded" and .status=="True"))
  | .metadata.name' | sort -u)"
report Nodes "$(oc get nodes -o json | jq -r '.items[] | select(.status.conditions[]
  | .type=="Ready" and .status!="True") | .metadata.name')"
report MCPs "$(oc get mcp -o json | jq -r '.items[] | select(.status.conditions[]
  | .type=="Updated" and .status!="True") | .metadata.name')"
report CSRs "$(oc get csr -o json | jq -r '.items[] | select(.status.conditions==null) | .metadata.name')"
report Pods "$(oc get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  -o custom-columns=A:.metadata.namespace,B:.metadata.name,C:.status.phase --no-headers | awk '{print $1"/"$2" "$3}')"
report PVCs "$(oc get pvc -A -o json | jq -r '.items[] | select(.status.phase!="Bound")
  | "\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"')"
HOST=$(oc get route console -n openshift-console -o jsonpath='{.spec.host}')
CERT=""; echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null \
  | openssl x509 -noout -checkend $((30*86400)) >/dev/null || CERT="$HOST ingress cert expires in <30d"
report IngressCert "$CERT"
exit $RC

It checks ClusterOperators not Available or Degraded (the first thing Red Hat support asks for), nodes not Ready, MachineConfigPools not Updated (a stuck rollout, Post 20), pending CSRs (nodes that cannot join until approved), pods not Running or Completed, PVCs not Bound, and the ingress certificate via openssl -checkend, which exits non-zero if the cert expires within the given seconds. The phase filter misses CrashLoopBackOff (phase is still Running); the first one-liner above is the extension. Run it every 15 minutes as a CronJob, and by hand at the start of every incident.

Script (b): the pre-upgrade validation gate

This gate is what a change advisory board wants attached to the upgrade ticket (Post 20 covers the upgrade itself). It is the health check plus four upgrade killers: no recent etcd backup, a PodDisruptionBudget that blocks a drain, workloads still calling an API the target release removes, and a target that is not on the recommended path. APIRequestCount (an OpenShift object counting who called which API version in the last 24 hours) reports removedInRelease as a Kubernetes minor, so 4.18 means 1.31.

#!/usr/bin/env bash
# preupgrade-gate.sh <ocp-version> <kube-minor>   e.g. ./preupgrade-gate.sh 4.18.20 1.31
set -euo pipefail
TARGET=${1:?ocp version}; KUBE=${2:?kubernetes minor of the target}
GATE=0
say() { printf '[%s] %s\n' "$1" "$2"; [[ $1 == FAIL ]] && GATE=1 || true; }

if ./cluster-health.sh >/tmp/health.txt; then say PASS "cluster health"
else say FAIL "cluster health (see /tmp/health.txt)"; fi

LAST=$(oc get jobs -n platform-etcd-backup -o json | jq -r '[.items[]
        | select((.status.succeeded // 0) > 0) | .status.completionTime] | sort | last // empty')
if [[ -n "$LAST" && $(( $(date +%s) - $(date -d "$LAST" +%s) )) -lt 86400 ]]; then say PASS "etcd backup at $LAST"
else say FAIL "no successful etcd backup in the last 24h"; fi

BLOCKERS=$(oc get pdb -A -o json | jq -r '.items[] | select(.status.disruptionsAllowed==0)
           | "\(.metadata.namespace)/\(.metadata.name)"' | paste -sd, -)
[[ -z "$BLOCKERS" ]] && say PASS "no PDB blocks a drain" || say FAIL "PDBs allowing 0 disruptions: $BLOCKERS"

REMOVED=$(oc get apirequestcounts -o json | jq -r --arg k "$KUBE" '.items[]
   | select(.status.removedInRelease == $k and .status.requestCount > 0)
   | "\(.metadata.name)=\(.status.requestCount)"' | paste -sd, -)
[[ -z "$REMOVED" ]] && say PASS "no traffic to APIs removed in $KUBE" || say FAIL "still called: $REMOVED"

oc adm upgrade | grep -q "$TARGET" && say PASS "$TARGET is in the recommended updates" \
  || say FAIL "$TARGET is not offered by the update graph"
exit $GATE

Attach the output to the change record. If the removed-API check fails, the fix is with the app team, not with the admin-acks ConfigMap; acknowledging the gate without fixing the caller just moves the outage to upgrade day.

Script (c): quota utilisation per namespace, to CSV

Post 26 set quotas on every project; finance and the capacity planners want to know who is near the ceiling. Quota values are strings like 1500m or 10Gi, so the script normalises them before dividing.

#!/usr/bin/env bash
# quota-report.sh > quota-$(date +%F).csv
set -euo pipefail
echo "date,namespace,quota,resource,used,hard,pct"
oc get resourcequota -A -o json | jq -r --arg d "$(date -u +%F)" '
  def norm: if type=="number" then . else
    (capture("^(?<n>[0-9.]+)(?<u>[a-zA-Z]*)$")
     | (.n|tonumber) * ({"m":0.001,"k":1e3,"M":1e6,"G":1e9,"Ki":1024,"Mi":1048576,
                          "Gi":1073741824,"Ti":1099511627776,"":1}[.u] // 1)) end;
  .items[] | .metadata.namespace as $ns | .metadata.name as $q | .status.hard as $h
  | (.status.used // {}) | to_entries[]
  | [$d, $ns, $q, .key, .value, $h[.key],
     (if ($h[.key]|norm) > 0 then ((.value|norm) / ($h[.key]|norm) * 100 | floor) else 0 end)]
  | @csv'

Pipe it to a dated file, upload to the reports bucket, and let a Grafana or spreadsheet view show the trend. The interview line: "app teams stopped opening 'my pod is Pending' tickets after we started sending them their own quota trend every Monday".

Script (d): orphaned-resource cleanup, dry-run by default

#!/usr/bin/env bash
# orphan-cleanup.sh [--apply]   default is dry-run: prints what it WOULD delete
set -euo pipefail
DAYS=${DAYS:-7}
DEL=(oc delete --dry-run=client); [[ ${1:-} == "--apply" ]] && DEL=(oc delete)
CUTOFF=$(date -u -d "-${DAYS} days" +%s)         # GNU date (gdate on macOS)
echo "running: ${DEL[*]}   completed jobs older than ${DAYS}d"

# 1. Released PVs: the claim is gone, reclaimPolicy=Retain left the volume behind
oc get pv -o json | jq -r '.items[] | select(.status.phase=="Released") | .metadata.name' \
  | while read -r pv; do "${DEL[@]}" pv "$pv"; done

# 2. Completed Jobs nobody owns (CronJob history limits already prune their own)
oc get jobs -A -o json | jq -r --argjson c "$CUTOFF" '.items[]
  | select((.status.succeeded // 0) > 0 and (.metadata.ownerReferences // []) == []
           and (.status.completionTime | fromdateiso8601) < $c)
  | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do "${DEL[@]}" job -n "$ns" "$name"; done

# 3. Failed and Evicted pods: logs are already in Loki, the objects are clutter
oc get pods -A --field-selector=status.phase=Failed -o json \
  | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do "${DEL[@]}" pod -n "$ns" "$name"; done

Released PVs deserve a human decision before --apply: with Retain the data is still on the backing volume, which is why the policy was Retain. The dry-run default is what lets you schedule this daily and run --apply only from a reviewed pipeline.

Script (e): the etcd backup CronJob

Red Hat's supported backup is /usr/local/bin/cluster-backup.sh run on a control-plane node, which snapshots etcd and the static pod resources. Running it by hand through oc debug node/... before every change does not scale, so the common pattern is a CronJob that runs the script from a privileged pod pinned to a control-plane node and ships the result off-cluster. Newer releases add a Tech Preview automated backup CR; check its status for your version. Until it is GA, this is what you will find in most banks.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: platform-etcd-backup
spec:
  schedule: "0 2 * * *"
  timeZone: America/Toronto
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 1800
      template:
        spec:
          serviceAccountName: etcd-backup      # oc adm policy add-scc-to-user privileged -z etcd-backup
          restartPolicy: Never
          nodeSelector: { node-role.kubernetes.io/master: "" }
          tolerations:
          - { key: node-role.kubernetes.io/master, operator: Exists, effect: NoSchedule }
          containers:
          - name: backup
            image: registry.bank.example/platform/etcd-backup:1.2.0   # ubi9 + aws-cli, pinned
            securityContext: { privileged: true }
            command: ["/bin/bash", "-c"]
            args:
            - |
              set -euo pipefail
              chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup
              aws s3 cp --recursive --sse aws:kms /host/home/core/assets/backup/ \
                "s3://bank-etcd-backups/$CLUSTER/$(date -u +%F)/"
              find /host/home/core/assets/backup -type f -mtime +2 -delete
            env:
            - { name: CLUSTER, value: prod-east }
            envFrom:
            - secretRef: { name: backup-bucket-credentials }   # or an IAM role / Vault Agent
            volumeMounts:
            - { name: host, mountPath: /host }
          volumes:
          - { name: host, hostPath: { path: /, type: Directory } }

Be able to say why this is the most dangerous automation on the cluster and what you do about it: it is privileged with the host root mounted, so the SA lives in its own namespace and is bound to nothing else; the image is built internally, signed and pinned; the bucket is KMS-encrypted with a retention policy, because an etcd snapshot contains every Secret in the cluster; and the backup is worthless until you have restored from it in non-prod (Post 20 covers the restore). Script (b) checks that this Job succeeded in the last 24 hours before any upgrade proceeds.

Python and the Kubernetes API

Shell stops being the right tool the moment you need pagination over 20,000 pods, retries on a throttled API server, a watch stream, or a unit test. The official kubernetes Python client (pip install kubernetes) gives you typed API groups: CoreV1Api for pods, secrets and nodes, AppsV1Api for deployments, RbacAuthorizationV1Api for bindings, and CustomObjectsApi for anything defined by a CRD, which on OpenShift means ClusterOperators (config.openshift.io/v1), MachineConfigPools (machineconfiguration.openshift.io/v1), Routes and everything an operator owns. Authentication is one call: config.load_kube_config() on a laptop or bastion, config.load_incluster_config() inside a pod (it reads the mounted ServiceAccount token). The patterns you must show an interviewer: retries with backoff, pagination with the _continue token, and a watch instead of a polling loop.

import random, time
from kubernetes import client, config, watch
from kubernetes.client.exceptions import ApiException

def with_retry(call, tries=5):
    """Retry throttling (429) and transient API-server errors with jittered backoff."""
    for i in range(tries):
        try:
            return call()
        except ApiException as e:
            if e.status not in (429, 500, 502, 503, 504) or i == tries - 1:
                raise
            time.sleep(min(2 ** i, 30) + random.random())

config.load_kube_config(context="prod-east")
custom = client.CustomObjectsApi()
mcps = with_retry(lambda: custom.list_cluster_custom_object(
    "machineconfiguration.openshift.io", "v1", "machineconfigpools"))
for mcp in mcps["items"]:
    conds = {c["type"]: c["status"] for c in mcp["status"].get("conditions", [])}
    print(mcp["metadata"]["name"], "Updated" if conds.get("Updated") == "True" else "NOT UPDATED")

w = watch.Watch()                     # react to events instead of polling every minute
for ev in w.stream(client.CoreV1Api().list_pod_for_all_namespaces,
                   field_selector="status.phase=Failed", timeout_seconds=600):
    print(ev["type"], ev["object"].metadata.namespace, ev["object"].metadata.name)

Three fallbacks worth naming: kubernetes.dynamic.DynamicClient addresses any resource by group, version and kind without typed classes; the openshift-client package (import openshift_client as oc) is Red Hat's Python wrapper around the oc binary, useful when you want oc semantics such as projects and logins; and plain subprocess.run(["oc", "get", "co", "-o", "json"], check=True, capture_output=True, text=True) plus json.loads is a respectable bridge when the CLI already does the job. The important part is check=True, so a failed oc raises instead of handing you an empty string.

Script (a): the pod compliance report

This finds workloads that break the baseline from Post 31: privileged containers, containers that may run as root, hostPath volumes, missing limits, and floating latest tags. It writes a CSV, logs a summary, posts it to a Slack or Teams webhook, and exits non-zero if anything was found. The root check is spec-level: restricted-v2 assigns a random non-root UID on OpenShift anyway, but a spec that does not say runAsNonRoot is still a finding, because the same manifest may land on EKS.

#!/usr/bin/env python3
"""pod_compliance.py: flag pods that break the platform baseline. CSV + summary + webhook."""
import argparse, csv, logging, os, sys, requests
from kubernetes import client, config

log = logging.getLogger("pod-compliance")
SKIP = ("openshift-", "kube-")     # platform namespaces are covered by the Compliance Operator

def api():
    try: config.load_incluster_config()
    except config.ConfigException: config.load_kube_config()
    return client.CoreV1Api()

def pods(v1):                      # paginate: never pull 20k pods in one response
    token = None
    while True:
        page = v1.list_pod_for_all_namespaces(limit=500, _continue=token)
        yield from page.items
        token = page.metadata._continue
        if not token: break

def findings(p):
    psc = p.spec.security_context
    for c in p.spec.containers:
        sc = c.security_context or client.V1SecurityContext()
        if sc.privileged: yield "privileged"
        nonroot = sc.run_as_non_root if sc.run_as_non_root is not None else (psc and psc.run_as_non_root)
        uid = sc.run_as_user if sc.run_as_user is not None else (psc and psc.run_as_user)
        if not nonroot and not uid: yield "may-run-as-root"
        if not (c.resources and c.resources.limits): yield "no-limits"
        if ":" not in c.image.rsplit("/", 1)[-1] or c.image.endswith(":latest"): yield "latest-tag"
    if any(v.host_path for v in (p.spec.volumes or [])): yield "hostPath"

def main():
    ap = argparse.ArgumentParser(); ap.add_argument("--out", default="pod-compliance.csv")
    ap.add_argument("--dry-run", action="store_true", help="write the CSV but do not notify")
    a = ap.parse_args()
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    rows, counts = [], {}
    for p in pods(api()):
        if p.metadata.namespace.startswith(SKIP): continue
        for f in sorted(set(findings(p))):
            rows.append([p.metadata.namespace, p.metadata.name, f]); counts[f] = counts.get(f, 0) + 1
    with open(a.out, "w", newline="") as fh:
        w = csv.writer(fh); w.writerow(["namespace", "pod", "finding"]); w.writerows(rows)
    summary = "Pod compliance: " + (", ".join(f"{k}={v}" for k, v in sorted(counts.items())) or "no findings")
    log.info("%s (%d rows written to %s)", summary, len(rows), a.out)
    if (hook := os.environ.get("SLACK_WEBHOOK")) and not a.dry_run:
        requests.post(hook, json={"text": summary}, timeout=10).raise_for_status()
    sys.exit(1 if rows else 0)

if __name__ == "__main__": main()

Because findings() is a pure function over a pod object, it is trivially testable with fake objects built from the client's own model classes. No cluster, no mocking of HTTP:

# tests/test_pod_compliance.py   (run: pytest -q)
from kubernetes import client
from pod_compliance import findings

def test_flags_latest_tag_missing_limits_and_root():
    pod = client.V1Pod(
        metadata=client.V1ObjectMeta(name="x", namespace="team-a"),
        spec=client.V1PodSpec(containers=[client.V1Container(name="c", image="nginx:latest")]))
    assert set(findings(pod)) == {"latest-tag", "no-limits", "may-run-as-root"}

def test_clean_pod_has_no_findings():
    c = client.V1Container(name="c", image="reg.bank.example/app:1.4.2",
        resources=client.V1ResourceRequirements(limits={"cpu": "500m", "memory": "256Mi"}),
        security_context=client.V1SecurityContext(run_as_non_root=True))
    assert list(findings(client.V1Pod(spec=client.V1PodSpec(containers=[c])))) == []

Script (b): certificate and token expiry checker

Expired certificates remain a top cause of self-inflicted outages, and they are entirely predictable. This reads every kubernetes.io/tls Secret, parses the leaf certificate with the cryptography library, and warns below 30 days. Two OpenShift details: Secrets generated by the service-CA operator carry the annotation service.beta.openshift.io/originating-service-name and rotate themselves, so skip them or drown in noise; and the platform's own certificates (API server, ingress, etcd, kubelet) are rotated by their operators, so the ones that bite are the ones humans uploaded: the custom ingress wildcard, the API server cert from the corporate CA, and app-team certs on Routes.

#!/usr/bin/env python3
"""tls_expiry.py: warn on TLS secrets expiring soon; exit 1 and/or push a gauge to Pushgateway."""
import base64, logging, os, sys
from datetime import datetime, timezone
from cryptography import x509
from kubernetes import client, config
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

WARN_DAYS = int(os.environ.get("WARN_DAYS", "30"))
AUTO = "service.beta.openshift.io/originating-service-name"   # service-CA certs rotate themselves
log = logging.getLogger("tls-expiry")

def days_left(secret):
    pem = base64.b64decode(secret.data["tls.crt"])
    leaf = x509.load_pem_x509_certificates(pem)[0]                # first cert in the bundle
    return (leaf.not_valid_after_utc - datetime.now(timezone.utc)).days

def main():
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    try: config.load_incluster_config()
    except config.ConfigException: config.load_kube_config()
    v1 = client.CoreV1Api()
    reg = CollectorRegistry()
    gauge = Gauge("tls_secret_days_left", "Days until notAfter", ["namespace", "secret"], registry=reg)
    expiring, checked = [], 0
    for s in v1.list_secret_for_all_namespaces(field_selector="type=kubernetes.io/tls").items:
        if (s.metadata.annotations or {}).get(AUTO) or not (s.data or {}).get("tls.crt"):
            continue
        try:
            d = days_left(s)
        except (ValueError, IndexError) as e:
            log.warning("%s/%s unparsable: %s", s.metadata.namespace, s.metadata.name, e); continue
        checked += 1
        gauge.labels(s.metadata.namespace, s.metadata.name).set(d)
        if d < WARN_DAYS:
            expiring.append((s.metadata.namespace, s.metadata.name, d))
            log.warning("%s/%s expires in %d days", s.metadata.namespace, s.metadata.name, d)
    if gw := os.environ.get("PUSHGATEWAY"):
        push_to_gateway(gw, job="tls-expiry", registry=reg)
    log.info("checked %d secrets, %d expiring within %d days", checked, len(expiring), WARN_DAYS)
    sys.exit(1 if expiring else 0)

if __name__ == "__main__": main()

Reading Secrets cluster-wide is the most sensitive permission in this post, so this SA gets get/list on secrets and nothing else, and the logs never print a certificate. With the gauge in Prometheus, alerting is one rule: tls_secret_days_left < 30. Tokens are the other half: since Kubernetes 1.24 (OpenShift 4.11) ServiceAccount tokens are bound and short-lived, so any long-lived token Secret you find (type=kubernetes.io/service-account-token) is a removal candidate, and the same script shape lists them.

Prometheus as an automation data source

The cluster already has the numbers you need for capacity and cost reports: the in-cluster monitoring stack (Post 24) exposes everything through the Thanos Querier, a component that answers PromQL across the platform and user-workload Prometheus instances, behind a Route with RBAC. A ServiceAccount bound to the cluster-monitoring-view ClusterRole can query it with a bearer token, which means any script can turn PromQL into a report.

#!/usr/bin/env bash
# capacity-report.sh: weekly headroom from the in-cluster Prometheus via Thanos Querier
set -euo pipefail
HOST=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}')
TOKEN=$(oc create token capacity-report -n platform-automation --duration=10m)
q() { curl -sS --fail --max-time 30 -H "Authorization: Bearer $TOKEN" "https://$HOST/api/v1/query" \
        --data-urlencode "query=$1" | jq -r '.data.result[0].value[1] // "0"'; }
pct() { jq -n "$1 * 100 / $2"; }; gib() { jq -n "$1 / 1073741824"; }
W='* on(node) group_left() kube_node_role{role="worker"}'        # workers only
CPU_ALLOC=$(q "sum(kube_node_status_allocatable{resource=\"cpu\"} $W)")
CPU_REQ=$(q "sum(kube_pod_container_resource_requests{resource=\"cpu\"} $W)")
CPU_USED=$(q 'sum(rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[1h]))')
MEM_ALLOC=$(q "sum(kube_node_status_allocatable{resource=\"memory\"} $W)")
MEM_REQ=$(q "sum(kube_pod_container_resource_requests{resource=\"memory\"} $W)")
printf '# Capacity report %s\n\n' "$(date -u +%F)"
printf 'CPU: allocatable %.0f cores, requested %.0f (%.0f%%), actually used %.0f\n' \
  "$CPU_ALLOC" "$CPU_REQ" "$(pct "$CPU_REQ" "$CPU_ALLOC")" "$CPU_USED"
printf 'Memory: allocatable %.0f GiB, requested %.0f GiB (%.0f%%)\n\n' \
  "$(gib "$MEM_ALLOC")" "$(gib "$MEM_REQ")" "$(pct "$MEM_REQ" "$MEM_ALLOC")"
printf '## Top namespaces by CPU (1h average, cores)\n'
curl -sS --fail -H "Authorization: Bearer $TOKEN" "https://$HOST/api/v1/query" --data-urlencode \
  'query=topk(10, sum by (namespace) (rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[1h])))' \
  | jq -r '.data.result[] | "- \(.metric.namespace): \(.value[1] | tonumber * 100 | round / 100)"'

The gap between "requested" and "actually used" is the sentence that gets a capacity meeting's attention: if teams request 400 cores and use 90, the cluster is full on paper and idle in reality, and the fix is right-sizing requests (Post 9), not buying nodes. For trends, use /api/v1/query_range with start, end and step. The metrics used here come from kube-state-metrics and the kubelet's cAdvisor endpoint and exist on EKS with Prometheus installed too, so one script produces the same report on both platforms.

Ansible for platform day-2

Ansible = an agentless automation tool that runs YAML playbooks over SSH or, for Kubernetes, over the API. On a platform team its job is everything a single cluster's GitOps cannot see: the same baseline on twelve clusters, bastion hosts, DNS and load-balancer entries, the registry VM, and day-2 procedures that need ordering across nodes. The kubernetes.core collection provides k8s (apply or delete any object, idempotently), k8s_info (read objects) and helm; redhat.openshift adds modules such as openshift_auth. Clusters are addressed by kubeconfig context, so the inventory is a list of context names, with credentials in Ansible Vault or the automation platform's credential store, never in the repo.

---
# baseline.yml: apply the platform baseline to every cluster, idempotently
- name: Platform baseline on every cluster
  hosts: localhost
  gather_facts: false
  vars_files:
    - vars/clusters.yml      # clusters: [{name: prod-east, context: prod-east, env: prod}, ...]
  tasks:
    - name: Apply baseline objects (namespaces, quotas, network policies)
      kubernetes.core.k8s:
        context: "{{ item.context }}"
        state: present
        apply: true
        src: "{{ playbook_dir }}/baseline/{{ item.env }}.yaml"
      loop: "{{ clusters }}"
      loop_control:
        label: "{{ item.name }}"

    - name: Verify the default-deny policy exists in the template namespace
      kubernetes.core.k8s_info:
        context: "{{ item.context }}"
        api_version: networking.k8s.io/v1
        kind: NetworkPolicy
        namespace: platform-baseline
        name: default-deny-ingress
      register: np
      loop: "{{ clusters }}"
      loop_control:
        label: "{{ item.name }}"
      failed_when: np.resources | length == 0

    - name: Refresh the bastion's oc binary to match the newest cluster
      ansible.builtin.get_url:
        url: "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/{{ oc_version }}/openshift-client-linux.tar.gz"
        dest: /opt/oc/openshift-client-{{ oc_version }}.tar.gz
        mode: "0644"
      delegate_to: bastion

Run it with ansible-playbook baseline.yml --check --diff first: --check is Ansible's dry-run, --diff shows what would change. apply: true makes the k8s module behave like oc apply (a three-way merge), which is what makes re-running safe. Keep ansible-lint in CI, encrypt variable files with ansible-vault, and pin collection versions in requirements.yml.

At a bank you rarely run ansible-playbook from a shell in production. Ansible Automation Platform (AAP) is Red Hat's enterprise wrapper: playbooks run in execution environments (container images with pinned collections), credentials are injected from its store at run time, and every job has an owner, a schedule, an approver and an audit log that maps to a change ticket. "How do you run automation in a regulated environment?" expects this answer: AAP with a job template per procedure, a survey for inputs, and an approval node in the workflow.

When to reach for Argo CD or RHACM instead

Ansible applies state once, when it runs. Argo CD (Post 30) and Red Hat Advanced Cluster Management policies reconcile state continuously and flag drift. The rule: if the object should exist forever and someone editing it by hand is a problem, it belongs in Git behind Argo or an RHACM Policy with remediationAction: enforce. If it is a procedure (rotate this credential, migrate these PVs, patch these bastions), it is a playbook. Many teams use Ansible exactly once per cluster, to bootstrap Argo, and then let Argo carry the baseline.

Scheduling, sinks and notifications

An automation nobody runs is a document. Inside the cluster the scheduler is the CronJob (Post 13). The fields that matter for platform automation are concurrencyPolicy: Forbid (never start a second run while one is still going), history limits (so completed Jobs do not pile up), startingDeadlineSeconds (how late a run may start before it is skipped), and on the Job itself backoffLimit and activeDeadlineSeconds. Here is the health check from script (a) as a deployable unit, with its identity and exactly the permissions it needs.

apiVersion: v1
kind: ServiceAccount
metadata: { name: health-check, namespace: platform-automation }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: platform-health-reader }
rules:
- apiGroups: [""]
  resources: [nodes, pods, persistentvolumeclaims, namespaces]
  verbs: [get, list]
- apiGroups: [config.openshift.io]
  resources: [clusteroperators, clusterversions]
  verbs: [get, list]
- apiGroups: [machineconfiguration.openshift.io]
  resources: [machineconfigpools]
  verbs: [get, list]
- apiGroups: [certificates.k8s.io]
  resources: [certificatesigningrequests]
  verbs: [get, list]
- apiGroups: [route.openshift.io]
  resources: [routes]
  verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: platform-health-reader }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: platform-health-reader }
subjects:
- { kind: ServiceAccount, name: health-check, namespace: platform-automation }

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cluster-health
  namespace: platform-automation
spec:
  schedule: "*/15 * * * *"
  timeZone: America/Toronto
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 600
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          serviceAccountName: health-check
          restartPolicy: Never
          containers:
          - name: check
            image: registry.bank.example/platform/ops-tools:1.4.0   # oc + jq + python, pinned
            command: ["/opt/scripts/cluster-health.sh"]
            env:
            - name: SLACK_WEBHOOK
              valueFrom: { secretKeyRef: { name: notify, key: slack-webhook } }
            resources:
              requests: { cpu: 100m, memory: 128Mi }
              limits: { cpu: 500m, memory: 256Mi }
            securityContext:
              allowPrivilegeEscalation: false
              runAsNonRoot: true
              capabilities: { drop: [ALL] }
              seccompProfile: { type: RuntimeDefault }

Inside the pod, oc finds the SA token automatically, so no kubeconfig is needed. Outside the cluster the same script runs from a GitHub Actions workflow (on: schedule: - cron: "0 6 * * 1"), a Jenkins job with a cron trigger, or, on the AWS side, an EventBridge Scheduler rule starting a Lambda or CodeBuild run; a nightly terraform plan -detailed-exitcode against every EKS cluster is the classic use (Post 29). Choose by where the credentials already live: in-cluster automation uses the SA, AWS-side automation uses an IAM role, and neither needs a human's kubeconfig.

Every automation needs a sink and a signal. Sinks: an S3 or GCS bucket with dated prefixes for anything auditors will ask for later, a Confluence or SharePoint page for reports humans read, a ServiceNow ticket via its REST API (POST /api/now/table/incident) for anything needing an owner and an SLA. Signals: a Slack or Teams webhook for the summary, email for the weekly report, and, most important, an alert on absence. A CronJob that silently stops is the failure nobody notices, so a PrometheusRule with time() - kube_cronjob_status_last_successful_time{cronjob="cluster-health"} > 3600 pages you when the check itself has not succeeded in an hour.

Automating compliance reporting

The JD says "compliance reporting" on purpose: at a bank the platform team owes a monthly evidence pack to risk and audit, and building it by hand from four consoles takes days. The sources are already machine-readable. The Compliance Operator (Post 31) scans nodes and platform against profiles such as CIS and stores each check as a ComplianceCheckResult with a status (PASS, FAIL, MANUAL, NOT-APPLICABLE) and a severity. Kyverno writes admission results to PolicyReport objects with per-namespace pass/fail summaries. The Trivy Operator writes VulnerabilityReport objects with critical and high counts per image; Red Hat Advanced Cluster Security exposes the same through roxctl and its REST API. An access review is a question to RBAC: who is cluster-admin, who is admin where, who holds a privileged SCC. The deliverable is one dated report with pass/fail counts and an owner per failing item, kept in Git or a bucket for the bank's retention period.

$ oc get compliancecheckresults -A -o json | jq -r '.items[]
    | [.metadata.labels["compliance.openshift.io/suite"], .metadata.name, .severity, .status] | @csv' \
  | sort | uniq -c | awk '$NF ~ /FAIL/' | head
$ oc get policyreports -A -o json | jq -r '.items[] | "\(.metadata.namespace),\(.summary.pass),\(.summary.fail)"'
$ oc get vulnerabilityreports -A -o json | jq '[.items[].report.summary.criticalCount] | add'

The access review is the piece auditors read most closely and the one most often done by hand. This script builds it from ClusterRoleBindings, namespaced RoleBindings granting admin or edit, and SecurityContextConstraints, which grant access two ways: directly through the SCC's users and groups lists, or through RBAC with the use verb on securitycontextconstraints (Post 22).

#!/usr/bin/env python3
"""access_review.py: who holds power on this cluster, as a dated CSV for the evidence pack."""
import csv
from datetime import date
from kubernetes import client, config

config.load_kube_config()
rbac, custom = client.RbacAuthorizationV1Api(), client.CustomObjectsApi()
POWER = ("cluster-admin", "admin", "edit")
rows = []                                   # scope, binding, grant, subject_kind, subject

for crb in rbac.list_cluster_role_binding().items:
    if crb.role_ref.name in POWER:
        for s in crb.subjects or []:
            rows.append(["cluster", crb.metadata.name, f"clusterrole/{crb.role_ref.name}", s.kind, s.name])

for rb in rbac.list_role_binding_for_all_namespaces().items:
    ns = rb.metadata.namespace
    if ns.startswith(("openshift-", "kube-")) or rb.role_ref.name not in POWER:
        continue
    for s in rb.subjects or []:
        rows.append([ns, rb.metadata.name, f"{rb.role_ref.kind.lower()}/{rb.role_ref.name}", s.kind, s.name])

sccs = custom.list_cluster_custom_object("security.openshift.io", "v1", "securitycontextconstraints")
for scc in sccs["items"]:
    name = scc["metadata"]["name"]
    if name in ("restricted", "restricted-v2"):
        continue
    for u in scc.get("users") or []:  rows.append(["cluster", name, "scc-direct", "User/SA", u])
    for g in scc.get("groups") or []: rows.append(["cluster", name, "scc-direct", "Group", g])

for cr in rbac.list_cluster_role().items:
    for rule in cr.rules or []:
        if "securitycontextconstraints" in (rule.resources or []) and "use" in (rule.verbs or []):
            rows.append(["cluster", cr.metadata.name, "scc-via-rbac", "ClusterRole",
                         ",".join(rule.resource_names or ["*"])])

out = f"access-review-{date.today()}.csv"
with open(out, "w", newline="") as fh:
    w = csv.writer(fh); w.writerow(["scope", "binding", "grant", "subject_kind", "subject"]); w.writerows(rows)
admins = {r[4] for r in rows if r[2] == "clusterrole/cluster-admin"}
print(f"{out}: {len(rows)} grants, {len(admins)} cluster-admin subjects: {sorted(admins)}")

Two refinements make it audit-grade. Groups hide people: expand each OpenShift Group with oc get group <name> -o jsonpath='{.users}' so the report names humans, not just ops-admins. And custom ClusterRoles can be cluster-admin in all but name, so oc adm policy who-can create clusterrolebindings and who-can '*' '*' catch the escalation paths a name filter misses. Commit the CSV under reports/<date>/ from CI; the diff between two months is the "what changed in privileged access" evidence.

Automating platform deployments

A cluster's life has three phases and the JD wants all three automated. Day 0, creating the cluster, is Terraform in a pipeline (Post 29): the EKS module or the OpenShift installer driven from CI, plan reviewed, apply gated. Day 1, bootstrap, is a short idempotent script or playbook whose only job is to install the GitOps operator and point Argo CD at the platform configuration repo (Post 30). Everything else (operators, monitoring config, quotas, network policies, the CronJobs from this post) then arrives through Argo, and the bootstrap script never grows.

#!/usr/bin/env bash
# bootstrap.sh <cluster-name>: day 0 to day 1 in one idempotent run; afterwards Argo CD owns the cluster
set -euo pipefail
CLUSTER=${1:?cluster name, e.g. prod-east}
oc apply -f bootstrap/openshift-gitops-subscription.yaml
oc wait --for=condition=Available deployment/openshift-gitops-server -n openshift-gitops --timeout=600s
# Argo's controller needs enough rights to manage the platform layer (scope this down later, Post 30)
oc adm policy add-cluster-role-to-user cluster-admin \
   -z openshift-gitops-argocd-application-controller -n openshift-gitops
# the hand-over: one Application pointing at this cluster's folder in the platform config repo
sed "s/CLUSTER_NAME/$CLUSTER/" bootstrap/root-app.yaml | oc apply -f -
oc wait --for=jsonpath='{.status.health.status}'=Healthy application/platform-root \
   -n openshift-gitops --timeout=900s
echo "bootstrap complete: $CLUSTER is now reconciled from git"

Day 2 is where validation lives. After every upgrade, run a smoke test as a Job (triggered by the upgrade pipeline or as an Argo PostSync hook) that exercises the paths app teams depend on: scheduling, image pull from the internal registry, PVC provisioning, Service, Route, wildcard DNS and an HTTP round trip. Its ServiceAccount has edit in one namespace and nothing else.

#!/usr/bin/env bash
# smoke-test.sh: runs as a Job after every upgrade. The SA has 'edit' in platform-smoke only.
set -euo pipefail
NS=platform-smoke; IMG=registry.bank.example/platform/hello:1.0.3
trap 'oc -n $NS delete deploy,svc,route,pvc --all --wait=false' EXIT
oc -n $NS create deployment hello --image="$IMG" --port=8080
oc -n $NS set volume deployment/hello --add --name=data --type=persistentVolumeClaim \
   --claim-name=smoke --claim-size=1Gi --mount-path=/data
oc -n $NS rollout status deployment/hello --timeout=180s        # scheduling, pull, PVC bind
oc -n $NS wait pvc/smoke --for=jsonpath='{.status.phase}'=Bound --timeout=60s
oc -n $NS expose deployment hello && oc -n $NS expose service hello
HOST=$(oc -n $NS get route hello -o jsonpath='{.spec.host}')
getent hosts "$HOST" >/dev/null                                  # wildcard DNS still resolves
curl -sS --fail --max-time 10 --retry 6 --retry-delay 5 "http://$HOST/healthz" >/dev/null
echo "SMOKE OK: schedule, pull, pvc, service, route, dns, http"

Add one "chaos-lite" check in non-prod: a weekly CronJob that cordons and drains a single worker (oc adm drain --ignore-daemonsets --delete-emptydir-data) and then uncordons it. A drain that hangs on a PodDisruptionBudget or a single-replica StatefulSet in non-prod this week is an upgrade that would have stalled in prod next month. It is the cheapest rehearsal you will ever run.

Engineering quality for scripts

The difference between "I have scripts" and "we have automation" is a repository that another engineer can clone, run, and trust. A layout that holds up:

platform-automation/
├── README.md              # what runs where, how to run locally, who owns it, runbook links
├── scripts/               # shell: one job per file, shellcheck-clean, --help on every one
├── py/platform_ops/       # python package: shared client setup, logging, checks
├── tests/                 # pytest with fake API objects (no cluster needed)
├── ansible/               # playbooks, roles, requirements.yml; no secrets
├── deploy/                # CronJob, SA, RBAC, PrometheusRule per cluster (applied by Argo)
├── Containerfile          # pinned base image, pinned oc/jq versions
├── requirements.txt       # pinned with hashes
└── .github/workflows/     # lint + test on PR; build, scan and sign the image on tag

The habits that go with it: every change is a reviewed pull request; shellcheck and ruff (a fast Python linter) run in CI and block the merge; the image carries semver tags and CronJobs reference a tag, never latest; scripts log to stdout with timestamps and levels, and the important ones emit a metric (the Pushgateway gauge above, or simply Job success via kube-state-metrics); each script's --help links to its runbook; and every automation has a named owner in the README, because at a bank the question after "what does this do" is "who do I call at 03:00".

AI-assisted operations, soberly

In 2026 a platform engineer drafts the first version of a script, a PromQL query or a runbook with an LLM assistant, and interviewers may ask how you use these tools. The credible answer is specific and cautious. You use the bank-approved tool only (Red Hat ships OpenShift Lightspeed and Ansible Lightspeed so this can run inside the bank's boundary), you never paste secrets, tokens, customer data or unredacted cluster dumps into a prompt, and you treat the output as untrusted code from a fast junior: it goes through the same shellcheck, tests, review and dry-run as anything else. It genuinely helps with a half-remembered jq expression, the twentieth variant of a CronJob manifest, and the first pass of a runbook you then correct from experience. It does not decide whether a change is safe, and it does not replace your understanding of what the script does. Say that, and you sound like someone who will not paste a kubeconfig into a chat window.

How to talk about your automation in the interview

You will be asked "what have you automated?" and you will have 45 seconds before the interviewer moves on. Use a STAR shape (Post 35 covers the behavioural round in depth): the manual task and its cost, what you built, the measurable result, and what it prevented.

"We used to [manual task] by hand, which took [time] and missed [failure] at least [frequency]. I wrote [script/CronJob/playbook] that [what it checks or changes], running [schedule] as its own ServiceAccount with read-only RBAC, reporting to [sink] and alerting on [condition]. It cut [metric] from [before] to [after], and it caught [specific thing] before [incident it would have caused]."

Five to adapt, each one a script from this post:

• Health check: "Morning checks took 20 minutes across three clusters; a 15-minute CronJob now posts a table and pages on absence. It caught a Degraded image-registry operator at 02:00 before the 06:00 batch pulls."
• Pre-upgrade gate: "Upgrades were approved on a verbal 'looks fine'. The gate attaches PDB, backup, removed-API and update-graph evidence to the change ticket; it blocked an upgrade whose target minor removed an API a payments cron still called."
• Compliance report: "The monthly evidence pack took two engineers two days; a Job now builds the CSVs from the Compliance Operator, Kyverno and RBAC, commits them dated, and audit reads the diff. Findings dropped from 140 to 12 in a quarter because owners were named."
• Quota report: "Pending-pod tickets were the top ticket type; a Monday quota trend per namespace, pushed to the team's channel, cut them by two thirds and gave capacity planning a real forecast."
• Certificate expiry alert: "A custom ingress cert expired on a Saturday once. The checker reads every TLS Secret nightly, exports days-left as a metric, and the alert fires at 30 days. Zero certificate incidents since."

Use your own numbers, and if you have not run one of these in a job yet, run it on your lab cluster this week so that the sentence "I built and ran this" is true.

Likely interview questions

Next up: Post 33, the OpenShift, EKS, Terraform and GitOps question bank, where everything from Posts 18 to 32 becomes short, sharp answers you can deliver under pressure.