Monitoring and Logging: Prometheus, Alertmanager, Loki and Insights

At a bank, the sentence "we didn't get an alert" is not an excuse in a post-incident review. It is a finding, with an owner and a due date, and the owner is the platform team. Monitoring and logging on OpenShift are not something the app teams bolt on later; they are platform services you run, size, secure and are paged for. This post takes you through the whole stack as it ships in OpenShift 4.14–4.19: the built-in Prometheus and Alertmanager, user workload monitoring for app teams, the alerts you page on and the PromQL you read at 3 a.m., Loki-based logging with ClusterLogForwarder pipelines into Splunk, API audit logs as a compliance control, and the rest of the observability family (Insights, Network Observability, tracing). By the end you'll be able to explain each piece, configure it from YAML, troubleshoot the stack itself, and describe how it supports incident response in a regulated environment.

Why observability is a platform service, not a feature

In your earlier Kubernetes work, monitoring was probably something you installed: kube-prometheus-stack from a Helm chart, a Grafana with a few dashboards, maybe Fluent Bit shipping logs to Elasticsearch. On OpenShift, Red Hat ships a fully supported Cluster Monitoring stack with every cluster, and the platform team owns it the same way it owns the API server. App teams don't run their own Prometheus; they plug into yours through an RBAC-controlled interface called user workload monitoring. Logging is similar: a cluster-wide collector picks up every container's stdout and the platform decides where it goes.

That model matters at a bank for three reasons. Consistency: forty application teams cannot each invent their own alerting, so the platform sets the standard. Evidence: audit logs and metrics history are what the RCA (root cause analysis, the written post-incident report regulators and internal audit can ask to see) is built from. Compliance: forwarding audit logs to the SIEM (Security Information and Event Management system, the central security log store, typically Splunk) is a control auditors check, and it fails silently if nobody owns it. Interviewers test whether you have actually configured retention, wired a receiver, chased a missing ServiceMonitor or sized a LokiStack.

The built-in Cluster Monitoring stack

Everything lives in the openshift-monitoring namespace and is managed by the Cluster Monitoring Operator (CMO), itself a cluster operator (the monitoring entry in oc get co from Post 19). CMO reads one ConfigMap and reconciles the whole stack from it; if you hand-edit a Prometheus StatefulSet, CMO puts it back. The components:

• Cluster Monitoring Operator = the top-level operator; deploys everything below and reports health through the monitoring ClusterOperator.
• Prometheus Operator = turns Prometheus, Alertmanager, ServiceMonitor, PodMonitor and PrometheusRule custom resources into running Prometheus configuration. CMO drives it; you never edit it directly.
• prometheus-k8s = the platform Prometheus, a StatefulSet with 2 replicas scraping the same targets independently, so one can restart without a gap in data. It scrapes only platform components (namespaces starting with openshift-, kube-, and the node and control plane exporters).
• Alertmanager (alertmanager-main) = receives alerts from Prometheus, deduplicates, groups, silences and routes them to receivers (PagerDuty, Slack, email, webhooks). A StatefulSet with 2 replicas on current releases (early 4.x ran 3) that gossip with each other so a notification is sent once, not twice.
• Thanos Querier = the single query endpoint that fans out to both prometheus-k8s replicas (and the user workload Prometheus when enabled), deduplicates the results and enforces RBAC per namespace. The console and any Grafana you deploy query Thanos Querier, never a Prometheus pod directly.
• Thanos Ruler = evaluates alerting and recording rules for user workloads (lives in openshift-user-workload-monitoring when that is enabled).
• node-exporter = a DaemonSet exposing host metrics (CPU, memory, disk, filesystem, network) from every node.
• kube-state-metrics = turns Kubernetes object state into metrics: Deployment replicas, pods in CrashLoopBackOff, ResourceQuota usage. openshift-state-metrics does the same for OpenShift objects like Routes and BuildConfigs.
• metrics-server = serves the Metrics API that oc adm top and the HorizontalPodAutoscaler use. Upstream metrics-server on 4.16 and later; 4.14/4.15 clusters use prometheus-adapter for the same job.
• monitoring-plugin = the console plugin that renders the Observe menu (Alerting, Metrics, Dashboards, Targets).
• telemeter-client = sends a small set of anonymized health metrics to Red Hat so the Hybrid Cloud Console can show cluster status and Insights recommendations. Disconnected banks often disable it.

$ oc get pods -n openshift-monitoring
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-main-0                                      6/6     Running   0          3d2h
alertmanager-main-1                                      6/6     Running   0          3d2h
cluster-monitoring-operator-7b5c9d8f9c-x2k4m             1/1     Running   0          3d2h
kube-state-metrics-6d8b7c5f4d-h8z9l                      3/3     Running   0          3d2h
metrics-server-5f7d6c8b9-jt7wq                           1/1     Running   0          3d2h
monitoring-plugin-6c9d8f7b5-4rzxb                        1/1     Running   0          3d2h
node-exporter-2bx7f                                      2/2     Running   0          3d2h
node-exporter-8kq9m                                      2/2     Running   0          3d2h
node-exporter-c4vt6                                      2/2     Running   0          3d2h
openshift-state-metrics-7c8d9b6f5-vw5kd                  3/3     Running   0          3d2h
prometheus-k8s-0                                         6/6     Running   0          3d2h
prometheus-k8s-1                                         6/6     Running   0          3d2h
prometheus-operator-5d6f7c8b9-b8xqn                      2/2     Running   0          3d2h
telemeter-client-7f8e9d6c5-9c6vp                         3/3     Running   0          3d2h
thanos-querier-8a9b7c6d5-d2klm                           6/6     Running   0          3d2h
thanos-querier-8a9b7c6d5-w7pxs                           6/6     Running   0          3d2h

Notice the READY counts. prometheus-k8s-0 is 6/6 because Prometheus is wrapped by a config-reloader, a Thanos sidecar and kube-rbac-proxy containers that terminate TLS and check the caller's RBAC, which is why every access path, console or curl, goes through a bearer token. The console's Observe menu is the front door: Alerting (firing alerts, silences, rules), Metrics (a PromQL box), Dashboards and Targets (every scrape target and whether it is up). In an incident the console is faster than a query.

Configuring platform monitoring: the cluster-monitoring-config ConfigMap

By default the stack is functional but not production-shaped: Prometheus stores its time series on emptyDir, so a pod restart (or a node drain during an upgrade, see Post 20) throws away every metric that replica held. Default retention is 15 days, but only if the pod lives that long. At a bank the first thing you do on a new cluster is give monitoring persistent storage and pin it to infra nodes, all expressed in one ConfigMap, cluster-monitoring-config in openshift-monitoring, which does not exist until you create it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
    prometheusK8s:
      retention: 15d
      retentionSize: 80GB
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule
      volumeClaimTemplate:
        spec:
          storageClassName: gp3-csi
          resources:
            requests:
              storage: 100Gi
      externalLabels:
        cluster: ocp-prod-east-01
        environment: production
        region: ca-central-1

Still inside prometheusK8s, the remote write block that ships samples to the enterprise store:

      remoteWrite:
      - url: "https://mimir.observability.bank.internal/api/v1/push"
        authorization:
          type: Bearer
          credentials:
            name: mimir-remote-write
            key: token
        tlsConfig:
          ca:
            secret:
              name: mimir-remote-write
              key: ca.crt
        writeRelabelConfigs:
        - sourceLabels: [__name__]
          regex: "apiserver_request_duration_seconds_bucket|etcd_.*|kube_.*|node_.*|container_.*|cluster:.*"
          action: keep
        queueConfig:
          capacity: 10000
          maxShards: 20

    alertmanagerMain:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule
      volumeClaimTemplate:
        spec:
          storageClassName: gp3-csi
          resources:
            requests:
              storage: 10Gi
    thanosQuerier:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule
    kubeStateMetrics:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    telemeterClient:
      enabled: false

The keys an interviewer may ask about:

• retention = how long Prometheus keeps samples; retentionSize = a disk cap, and whichever is hit first wins. Keep retentionSize at roughly 80% of the PVC or Prometheus fills the volume and crashes.
• volumeClaimTemplate = a PVC per replica. Use a block StorageClass with decent IOPS (Post 21); Prometheus on slow NFS is a known bad time. Alertmanager needs only a small volume, but without it silences are lost on restart, and a lost silence means a re-page at 3 a.m.
• nodeSelector / tolerations = pin the stack to infra nodes, off the worker capacity app teams pay for and outside the Red Hat subscription count.
• externalLabels = labels stamped on every series this cluster sends out; with twenty clusters writing to one central store, cluster="ocp-prod-east-01" is the only way to tell them apart.
• remoteWrite = stream samples to a long-term store (Thanos Receive, Mimir, Cortex, a commercial backend, or Advanced Cluster Management's observability). Local retention stays at 15–30 days; the central store keeps 13 months. writeRelabelConfigs keeps only the series worth paying for.
• enableUserWorkload = switches on the second Prometheus for application metrics (next section).

Apply the ConfigMap and CMO rolls the StatefulSets; history on emptyDir is not migrated, so do this on day one. Verify with:

$ oc -n openshift-monitoring get pvc
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
alertmanager-main-db-alertmanager-main-0   Bound    pvc-0c3d7a8e-...   10Gi   RWO   gp3-csi   4m
alertmanager-main-db-alertmanager-main-1   Bound    pvc-5f19b2d1-...   10Gi   RWO   gp3-csi   4m
prometheus-k8s-db-prometheus-k8s-0         Bound    pvc-9a2e4c77-...   100Gi  RWO   gp3-csi   4m
prometheus-k8s-db-prometheus-k8s-1         Bound    pvc-b7d0e6f3-...   100Gi  RWO   gp3-csi   4m

$ oc -n openshift-monitoring get pods -l app.kubernetes.io/name=prometheus -o wide
NAME               READY   STATUS    RESTARTS   AGE   IP            NODE
prometheus-k8s-0   6/6     Running   0          3m    10.129.2.14   infra-1
prometheus-k8s-1   6/6     Running   0          2m    10.131.0.22   infra-2

User workload monitoring: how app teams get their own metrics and alerts

The platform Prometheus deliberately refuses to scrape application namespaces; one team exposing a metric labelled with a user ID could otherwise take down monitoring for the whole cluster. Instead, enableUserWorkload: true creates a second, isolated stack in openshift-user-workload-monitoring (its own Prometheus Operator, prometheus-user-workload-0/1 and thanos-ruler-user-workload-0/1), and Thanos Querier federates both. This is user workload monitoring (UWM). It has its own ConfigMap, user-workload-monitoring-config in that namespace, with the same shape of options (retention, PVCs, infra nodeSelector, remoteWrite) plus guardrails that only make sense for tenants:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      retention: 7d
      retentionSize: 40GB
      enforcedSampleLimit: 50000
      enforcedLabelLimit: 40
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule
      volumeClaimTemplate:
        spec:
          storageClassName: gp3-csi
          resources:
            requests:
              storage: 60Gi
    thanosRuler:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
    alertmanager:
      enabled: true
      enableAlertmanagerConfig: true

enforcedSampleLimit caps how many samples a single scrape may return, so one team's cardinality mistake fails their own target instead of the shared Prometheus. The alertmanager block deploys a separate alertmanager-user-workload and lets teams route their own alerts with the AlertmanagerConfig custom resource in their namespace, keeping the platform Alertmanager's routing tree under platform control.

What an app team writes

The app exposes /metrics (a Prometheus client library does this in every language). The team then creates three objects in their own namespace. First, a ServiceMonitor, which tells Prometheus "scrape the pods behind Services matching these labels, on the port named metrics":

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: payments-prod
  labels:
    app: payments-api
spec:
  selector:
    app: payments-api
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: metrics
    port: 9090
    targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: payments-prod
spec:
  selector:
    matchLabels:
      app: payments-api
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Two details break more ServiceMonitors than anything else: endpoints[].port is the name of the Service port, not the number, and in UWM the ServiceMonitor must live in the same namespace as the Service it selects. A PodMonitor is the same idea without a Service in the middle, handy for batch workers that never receive traffic:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: settlement-worker
  namespace: payments-prod
spec:
  selector:
    matchLabels:
      app: settlement-worker
  podMetricsEndpoints:
  - port: metrics
    interval: 30s

Then a PrometheusRule for the team's own alerts, evaluated by Thanos Ruler; the resulting alert carries the namespace label so it can be routed to that team:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-api-alerts
  namespace: payments-prod
spec:
  groups:
  - name: payments-api.rules
    rules:
    - alert: PaymentsApiHighErrorRate
      expr: |
        sum(rate(http_requests_total{namespace="payments-prod",job="payments-api",code=~"5.."}[5m]))
          /
        sum(rate(http_requests_total{namespace="payments-prod",job="payments-api"}[5m])) > 0.05
      for: 10m
      labels:
        severity: warning
        team: payments
      annotations:
        summary: "payments-api 5xx rate above 5% for 10 minutes"
        description: "{{ $value | humanizePercentage }} of requests are failing."
        runbook_url: https://runbooks.bank.internal/payments-api/high-error-rate
    - alert: PaymentsApiDown
      expr: sum(up{namespace="payments-prod",job="payments-api"}) == 0
      for: 5m
      labels:
        severity: critical
        team: payments

Once applied, the team opens Observe in their project and sees their metrics, their alerts and the built-in namespace dashboard, RBAC-scoped to what they own. No Grafana required for the basics.

RBAC for monitoring

OpenShift ships roles so you never hand out admin just so a team can add a ServiceMonitor (RBAC fundamentals are in Post 22):

$ oc policy add-role-to-user monitoring-edit oncall-dev -n payments-prod
clusterrole.rbac.authorization.k8s.io/monitoring-edit added: "oncall-dev"
$ oc policy add-role-to-group monitoring-rules-view payments-support -n payments-prod
clusterrole.rbac.authorization.k8s.io/monitoring-rules-view added: "payments-support"

Project members with view can already read their metrics; these roles control who can change what is scraped and alerted. Most banks bind monitoring-edit to the team's deployer group in the project template at onboarding (Post 26).

Alerting: Alertmanager, receivers, silences and the alerts that matter

Prometheus decides that something is wrong (a rule's expression is true for the for: duration); Alertmanager decides who hears about it and how. A fresh cluster sends everything to a receiver named default with no destination, which is why it fires AlertmanagerReceiversNotConfigured until you fix it.

Configuring receivers and routes

The configuration lives in the secret alertmanager-main in openshift-monitoring, key alertmanager.yaml. The console can edit it (Administration → Cluster Settings → Configuration → Alertmanager); for anything you want in Git, pull, edit and replace:

$ oc -n openshift-monitoring get secret alertmanager-main \
    --template='{{ index .data "alertmanager.yaml" }}' | base64 -d > alertmanager.yaml
$ vi alertmanager.yaml
$ oc -n openshift-monitoring create secret generic alertmanager-main \
    --from-file=alertmanager.yaml --dry-run=client -o yaml \
  | oc -n openshift-monitoring replace secret --filename=-
secret/alertmanager-main replaced

A realistic bank configuration: critical platform alerts page through PagerDuty, warnings open a ServiceNow incident via webhook, everything mirrors to Slack, and Watchdog goes to a dead man's switch. Matchers select on labels, so severity and namespace do the routing:

global:
  resolve_timeout: 5m
  slack_api_url: https://hooks.slack.com/services/T000/B000/XXXX
route:
  receiver: platform-slack
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
  - matchers:
    - alertname = Watchdog
    receiver: deadman-switch
    group_wait: 0s
    repeat_interval: 2m
  - matchers:
    - severity = critical
    - namespace =~ "openshift-.*|kube-.*|default"
    receiver: platform-pagerduty
    continue: true
  - matchers:
    - severity = critical
    - team = payments
    receiver: payments-pagerduty
    continue: true
  - matchers:
    - severity = warning
    receiver: servicenow
    continue: true

And the receivers those routes point at, plus one inhibition rule:

receivers:
- name: platform-slack
  slack_configs:
  - channel: '#ocp-prod-alerts'
    send_resolved: true
    title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .CommonLabels.severity }})'
    text: '{{ range .Alerts }}{{ .Annotations.summary }} {{ .Annotations.runbook_url }}\n{{ end }}'
- name: platform-pagerduty
  pagerduty_configs:
  - routing_key: 0123456789abcdef0123456789abcdef
    severity: critical
- name: payments-pagerduty
  pagerduty_configs:
  - routing_key: fedcba9876543210fedcba9876543210
- name: servicenow
  webhook_configs:
  - url: https://snow.bank.internal/api/x_ocp/alertmanager
    send_resolved: true
    http_config:
      authorization:
        type: Bearer
        credentials: REPLACE_WITH_SNOW_TOKEN
- name: deadman-switch
  webhook_configs:
  - url: https://nosnch.in/0123456789
inhibit_rules:
- source_matchers: [severity = critical]
  target_matchers: [severity = warning]
  equal: [alertname, namespace]

Vocabulary: group_by = alerts sharing these labels become one notification; group_wait = how long to collect alerts before the first notification; repeat_interval = how often to re-send while firing; continue: true = keep evaluating sibling routes, which is how one alert reaches both PagerDuty and Slack; inhibit_rules = suppress the warning while the critical version fires. Email, OpsGenie and Teams receivers exist too; at banks PagerDuty plus a ServiceNow webhook is the usual pair because an incident ticket must exist for audit.

Silences

A silence = a time-boxed mute of alerts matching a label set, with an author and comment, created for planned maintenance (a node pool patch window, Post 20) or a known issue being fixed. Use the console (Observe → Alerting → Silences) or amtool inside the Alertmanager container:

$ oc -n openshift-monitoring exec -c alertmanager alertmanager-main-0 -- \
    amtool silence add alertname=KubeNodeNotReady node=~"worker-1[0-2].*" \
    --alertmanager.url=http://localhost:9093 \
    --author="oncall" --comment="CHG0045812 worker pool patching" --duration=3h
9c1e2f0a-5b6d-4e7f-8a9b-0c1d2e3f4a5b

$ oc -n openshift-monitoring exec -c alertmanager alertmanager-main-0 -- \
    amtool silence query --alertmanager.url=http://localhost:9093
ID                                    Matchers                                        Ends At                  Created By  Comment
9c1e2f0a-5b6d-4e7f-8a9b-0c1d2e3f4a5b  alertname="KubeNodeNotReady" node=~"worker-1[0-2].*"  2026-09-08 21:00:00 UTC  oncall       CHG0045812 worker pool patching

Two habits mark a mature team: every silence carries a change number, and every silence expires. "Until further notice" is how a real outage gets muted for three weeks.

The built-in alerts you must recognise

OpenShift ships several hundred alerting rules; you know the ones that page platform on-call and what to do first. Severity is a label: critical = page a human now; warning = ticket, actioned in business hours; info = dashboards only. A bank's paging policy adds that a critical in an openshift-* namespace pages the platform team, a critical in an app namespace pages that team, and an alert without a runbook is a bug.

Read a firing alert from the CLI when the console is slow, straight from the Thanos Querier API:

$ HOST=$(oc -n openshift-monitoring get route thanos-querier -o jsonpath='{.spec.host}')
$ curl -s -H "Authorization: Bearer $(oc whoami -t)" \
    "https://$HOST/api/v1/query" --data-urlencode 'query=ALERTS{alertstate="firing",severity="critical"}' | jq -r '.data.result[].metric | "\(.alertname)\t\(.namespace)\t\(.pod // "-")"'
etcdHighFsyncDurations   openshift-etcd       etcd-master-2
KubePodCrashLooping      openshift-ingress    router-default-7d9f8c6b5-x2p4l

PromQL you must be able to read and write

PromQL = the query language of Prometheus, and an interviewer will happily put a query on the whiteboard and ask what it returns. Core ideas: a metric is a named time series with labels; a counter only goes up (restarts, requests, CPU seconds) so you wrap it in rate() or increase(); a gauge is a current value (memory in use, replicas available) you read directly; aggregations like sum by (namespace) collapse label dimensions. The metrics you will use most:

• container_memory_working_set_bytes = the memory the kernel counts against a container's limit and what the OOM killer looks at, so compare it with limits.memory.
• container_cpu_usage_seconds_total = a counter of CPU seconds consumed; rate() over 5 minutes gives cores in use.
• kube_pod_container_status_restarts_total = restart counter per container, from kube-state-metrics.
• kube_deployment_status_replicas_unavailable = gauge; above 0 for long means a Deployment is not fully healthy.
• node_filesystem_avail_bytes / node_filesystem_size_bytes = disk headroom per mount from node-exporter.
• up = 1 if the last scrape of a target succeeded, 0 if not. The first query when "metrics are missing".
• ALERTS = a synthetic metric, one series per active alert with alertstate (pending/firing) and all its labels; how you graph alerting history.

# Top 10 pods by memory (working set), whole cluster
topk(10, sum by (namespace, pod) (container_memory_working_set_bytes{container!="", container!="POD"}))

# Containers that restarted in the last hour, with counts
sum by (namespace, pod, container) (increase(kube_pod_container_status_restarts_total[1h])) > 0

# Node CPU saturation as a fraction (0..1) per node
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# PVC usage in percent, from the kubelet's volume stats
100 * kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes

# API server request latency p99 in seconds, by verb, excluding long-lived WATCH
histogram_quantile(0.99,
  sum by (le, verb) (rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT"}[5m])))

# CPU actually used per namespace (cores) vs CPU requested per namespace (cores)
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})

# Deployments not fully available right now
kube_deployment_status_replicas_unavailable > 0

# Node root filesystem free percent
100 * node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"} / node_filesystem_size_bytes

How to read the p99 query, since it is the one most people fumble: apiserver_request_duration_seconds_bucket is a histogram, a set of counters per latency bucket labelled le ("less than or equal"). rate() turns each bucket into a per-second rate, sum by (le, verb) keeps the bucket boundaries while collapsing everything else, and histogram_quantile(0.99, …) interpolates the value below which 99% of requests fell. Drop le from the by clause and the function returns nonsense; that is the classic mistake. The container!="" filter matters too: cAdvisor exports a series for the pod's cgroup as a whole (empty container label) and for the POD pause container, so a naive sum double counts.

Dashboards, capacity planning and showback

OpenShift removed its bundled Grafana in 4.11; the console's Observe → Dashboards carries the same panels natively (etcd, API Performance, Kubernetes / Compute Resources at cluster, namespace, node, pod and workload level, Networking, Node Exporter USE Method, Persistent Volumes), which is enough for triage. For anything shared with management or app teams you deploy Grafana yourself, either the community Grafana Operator from OperatorHub (Post 23) or the enterprise Grafana the bank already licences. Either way the datasource is Thanos Querier, authenticated with a service account token that carries cluster-monitoring-view:

$ oc new-project grafana
$ oc create serviceaccount grafana-reader -n grafana
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-reader -n grafana
$ oc create token grafana-reader -n grafana --duration=8760h
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik...

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: thanos-querier
  namespace: grafana
spec:
  instanceSelector:
    matchLabels:
      dashboards: platform
  datasource:
    name: OpenShift Prometheus
    type: prometheus
    access: proxy
    url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
    jsonData:
      httpHeaderName1: Authorization
      tlsSkipVerify: false
      tlsAuthWithCACert: true
    secureJsonData:
      httpHeaderValue1: "Bearer ${GRAFANA_TOKEN}"
      tlsCACert: "${SERVICE_CA}"

Port 9091 is the cluster-wide endpoint (needs cluster-monitoring-view); port 9092 is the tenant endpoint that enforces a namespace= query parameter, which is how you give an app team a Grafana that can only see their project. The service CA comes from the openshift-service-ca.crt ConfigMap that exists in every namespace.

A platform dashboard shows, top to bottom: cluster capacity (allocatable vs requested vs used CPU and memory, pods per node against the 250 default), node health (Ready count, pressure conditions, recent reboots), etcd (leader changes, fsync p99, DB size against the 8 GB quota), ingress (HAProxy request rate, 5xx ratio, backends up) and per-namespace quota usage. Capacity planning is arithmetic on the same metrics: headroom = allocatable minus requests, and predict_linear(sum(kube_pod_container_resource_requests{resource="memory"})[30d:1h], 90*24*3600) projects requests 90 days out, which a bank wants on a slide before the budget meeting, not after the cluster fills. Showback (cost per team without billing) sums requests per namespace joined to a cost-centre label, either through the kube-state-metrics labelsAllowList under kubeStateMetrics in recent CMO versions or the Cost Management Metrics Operator that feeds Red Hat's cost management service; multi-cluster estates centralise dashboards in Advanced Cluster Management's observability instead of per-cluster Grafana.

Logging: Vector, LokiStack and the ClusterLogForwarder

Logging on OpenShift is a separate operator you install, Red Hat OpenShift Logging (the Cluster Logging Operator, CLO), plus the Loki Operator for storage. On current releases (Logging 6.x, which is what a 4.16+ cluster gets) the architecture has three parts:

• Vector = the collector, a DaemonSet named collector in openshift-logging. One pod per node reads container logs from /var/log/pods on the host, journald for node services and the API audit log files, adds Kubernetes metadata (namespace, pod, container, labels), and forwards them to the outputs you define.
• LokiStack = the log store, deployed by the Loki Operator from a LokiStack CR. Loki indexes only labels (namespace, pod, container, log type) and keeps the raw log lines compressed in object storage: S3 on AWS, ODF's NooBaa/Multicloud Object Gateway on-prem, GCS or Azure Blob elsewhere. Far cheaper than Elasticsearch per gigabyte, at the cost of slower free-text search.
• Console log view = Observe → Logs, a console UI plugin (in Logging 6 installed through the Cluster Observability Operator's UIPlugin of type Logging). App teams query their own namespace's logs with LogQL; cluster-admins see infrastructure and audit logs too.

The retired stack was EFK (Fluentd, the Elasticsearch Operator, Kibana), deprecated in Logging 5.x and removed in 6.0; you will still meet clusters running it, and "Loki or Elasticsearch?" is a standard question. Every log line is tagged with one of three log types, which you route independently: application (containers in non-openshift-* namespaces), infrastructure (containers in openshift-*, kube-* and default, plus node journald), and audit (Kubernetes API server, OpenShift API server, OAuth server, and node auditd events). Application teams see application logs; audit logs go to security.

LokiStack sizing and retention

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  storage:
    schemas:
    - version: v13
      effectiveDate: "2025-01-01"
    secret:
      name: logging-loki-s3
      type: s3
  storageClassName: gp3-csi
  tenants:
    mode: openshift-logging
  limits:
    global:
      retention:
        days: 30
    tenants:
      audit:
        retention:
          days: 400

size is a t-shirt size fixing replica counts and resources: 1x.demo (single replica, not for production), 1x.pico and 1x.extra-small (tens of GB/day), 1x.small (roughly 500 GB/day), 1x.medium (about 2 TB/day). Pick by ingest volume, measured by running the small size for a week and reading sum(rate(vector_component_received_bytes_total[1h])). The secret holds access_key_id, access_key_secret, bucketnames, endpoint and region (STS works on AWS). Retention is per tenant, so audit keeps 13 months while application logs keep 30 days.

ClusterLogForwarder: pipelines, outputs, filters

The ClusterLogForwarder (CLF) describes the whole collection graph: inputs (the three log types or narrower custom inputs), filters (transformations), outputs (destinations) and pipelines (which inputs go through which filters to which outputs). In Logging 6 the API is observability.openshift.io/v1 and it needs a service account with the collect-* ClusterRoles; Logging 5 used logging.openshift.io/v1 alongside a ClusterLogging CR that no longer exists. A bank-shaped example: application and infrastructure logs to Loki, audit to Splunk and Loki:

$ oc -n openshift-logging create serviceaccount collector
$ for r in collect-application-logs collect-infrastructure-logs collect-audit-logs; do
    oc adm policy add-cluster-role-to-user $r -z collector -n openshift-logging; done
$ oc -n openshift-logging create secret generic splunk-hec --from-literal=hecToken=REPLACE_ME

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: openshift-logging
spec:
  managementState: Managed
  serviceAccount:
    name: collector
  outputs:
  - name: loki-internal
    type: lokiStack
    lokiStack:
      target:
        name: logging-loki
        namespace: openshift-logging
      authentication:
        token:
          from: serviceAccount
    tls:
      ca:
        key: service-ca.crt
        configMapName: openshift-service-ca.crt
  - name: splunk-siem
    type: splunk
    splunk:
      url: https://splunk-hec.bank.internal:8088
      index: ocp_prod_audit
      authentication:
        token:
          secretName: splunk-hec
          key: hecToken
    tls:
      ca:
        key: ca-bundle.crt
        configMapName: bank-root-ca

Continuing the same outputs list with a Kafka destination, then the filters, a custom input and the pipelines that wire it all together:

  - name: kafka-apps
    type: kafka
    kafka:
      url: tls://kafka-bootstrap.streams.bank.internal:9093/ocp-app-logs
    tls:
      ca:
        key: ca-bundle.crt
        configMapName: bank-root-ca

  filters:
  - name: multiline
    type: detectMultilineException
  - name: parse-json
    type: parse
  - name: drop-healthchecks
    type: drop
    drop:
    - test:
      - field: .message
        matches: '(GET|HEAD) /(healthz|readyz|livez)'
  - name: prune-noise
    type: prune
    prune:
      in: [.kubernetes.labels."pod-template-hash", .kubernetes.annotations]
  inputs:
  - name: payments-only
    type: application
    application:
      includes:
      - namespace: "payments-*"
      selector:
        matchLabels:
          tier: backend
  pipelines:
  - name: apps-and-infra-to-loki
    inputRefs: [application, infrastructure]
    filterRefs: [multiline, parse-json, drop-healthchecks, prune-noise]
    outputRefs: [loki-internal]
  - name: audit-to-siem
    inputRefs: [audit]
    outputRefs: [splunk-siem, loki-internal]
  - name: payments-to-kafka
    inputRefs: [payments-only]
    filterRefs: [parse-json]
    outputRefs: [kafka-apps]

Read it as a wiring diagram. Output types to know: lokiStack and loki, splunk (via HEC, the HTTP Event Collector; common at banks because Splunk is already the SIEM), elasticsearch, kafka, cloudwatch (IAM role or access key), syslog (RFC 5424 for legacy collectors), azureMonitor, googleCloudLogging, http and otlp. The filters replace what used to be custom Fluentd config: detectMultilineException stitches a Java stack trace back into one event; parse turns JSON lines into fields so level="error" is queryable; drop discards health-check spam before it costs money; prune strips fields nobody searches. Custom inputs select by namespace glob and pod label, so one team's logs go to their Kafka topic without touching anyone else's.

The audit-to-SIEM pipeline is a compliance control: it satisfies the "privileged activity is logged to a tamper-resistant central store" requirement in PCI DSS, SOC 2 and OSFI-style guidelines. A control nobody alerts on is not a control, so pair it with a Prometheus alert on vector_component_errors_total{component_id="splunk-siem"} and a Splunk-side "no events from cluster X in 15 minutes" search. Post 31 covers the paperwork.

$ oc -n openshift-logging get clusterlogforwarder collector -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
observability.openshift.io/Authorized=True permitted to collect log types: [application audit infrastructure]
observability.openshift.io/Valid=True
Ready=True

$ oc -n openshift-logging get pods -l app.kubernetes.io/component=collector -o wide | head -4
NAME              READY   STATUS    RESTARTS   AGE   IP            NODE
collector-4gk2x   1/1     Running   0          2h    10.128.4.9    worker-1
collector-7zq9p   1/1     Running   0          2h    10.129.2.31   infra-1
collector-b8mvd   1/1     Running   0          2h    10.130.0.17   master-0

Loki queries use LogQL, PromQL for logs: a label selector, then line filters and parsers, as in {kubernetes_namespace_name="payments-prod", kubernetes_container_name="api"} |= "ERROR" | json | level="error". Log-based alerts come from the AlertingRule CR in loki.grafana.com/v1, useful for "more than 50 PaymentDeclined lines in 5 minutes" without a metrics library.

Audit logging: who did what to the API

The API audit log = a record of every request to the Kubernetes API server: who (user or service account, groups, source IP, user agent), what (verb, resource, namespace, name), when, and the response code. It is the only trustworthy answer to "who deleted the production Deployment at 02:14?" because oc history on someone's laptop proves nothing. OpenShift writes it on every control plane node, one file per API server:

• /var/log/kube-apiserver/audit.log for core Kubernetes resources,
• /var/log/openshift-apiserver/audit.log for OpenShift resources (Routes, Projects, ImageStreams, BuildConfigs),
• /var/log/oauth-apiserver/audit.log and /var/log/oauth-server/audit.log for tokens and logins.

Detail is set by the audit profile on the APIServer cluster resource: Default logs metadata for every request; WriteRequestBodies adds the body of every create, update, patch and delete (what the object looked like when changed, typical for banks); AllRequestBodies adds read bodies too (large, rarely justified); None disables auditing and should never appear in production. Per-group custom rules can log a noisy automation group at a lower level.

$ oc get apiserver cluster -o jsonpath='{.spec.audit}{"\n"}'
{"profile":"Default"}
$ oc patch apiserver cluster --type=merge -p '{"spec":{"audit":{"profile":"WriteRequestBodies"}}}'
apiserver.config.openshift.io/cluster patched

Changing the profile rolls the API server pods (a few minutes, no downtime with three masters). You read the logs without SSH through oc adm node-logs, which streams files from the node's /var/log via the API. Finding who deleted a Deployment:

$ oc adm node-logs --role=master --path=kube-apiserver/
master-0 audit-2026-09-07T14-22-11.301.log
master-0 audit.log
master-1 audit-2026-09-07T13-58-40.877.log
master-1 audit.log
master-2 audit.log

$ oc adm node-logs --role=master --path=kube-apiserver/audit.log \
  | jq -c 'select(.verb=="delete" and .objectRef.resource=="deployments" and .objectRef.namespace=="payments-prod")
           | {t:.requestReceivedTimestamp, user:.user.username, groups:.user.groups, ip:.sourceIPs[0], ua:.userAgent, name:.objectRef.name, code:.responseStatus.code}'
{"t":"2026-09-08T02:14:37.512884Z","user":"jsmith@bank.com","groups":["payments-deployers","system:authenticated"],"ip":"10.40.12.77","ua":"oc/4.18.0 (darwin/arm64) kubernetes/e1f2a3b","name":"payments-api","code":200}

Each event also carries auditID, stage (ResponseComplete is usually the one you want) and an authorization.k8s.io/decision annotation of allow or forbid, so the same file answers "who tried and was denied". Three facts interviewers probe: the files rotate on the node and a busy cluster keeps only a day or two, so the SIEM copy is the record; --role=master queries all three masters because the request may have landed on any of them; and oc adm node-logs needs cluster-admin and is itself audited. Retention, access reviews and evidence requests are in Post 31.

The rest of the observability family

Metrics, alerts and logs are the core, but the JD's word "observability" covers a few more operators you should be able to name and place:

• Insights Operator (openshift-insights) = gathers anonymised cluster configuration and uploads it to Red Hat, where the Hybrid Cloud Console advisor turns it into recommendations ("etcd disks slower than recommended", "this Operator version has a known bug") and known-CVE exposure, also shown on the console overview page. Disconnected banks proxy the upload through controlled egress or disable it and accept the InsightsDisabled info alert. It is why Red Hat support already knows your cluster's state when you open a case.
• Network Observability Operator = an eBPF agent DaemonSet that captures network flows (source, destination, port, bytes, drops) and shows them in Observe → Network Traffic as topology and flow tables; flows can be stored in Loki or, since 1.6, kept as Prometheus metrics only. It answers "which pods talk to the mainframe gateway, and did that change after the NetworkPolicy rollout?" (Post 21).
• Distributed tracing = the Red Hat build of OpenTelemetry (an OpenTelemetryCollector CR that receives OTLP spans from apps) plus the Tempo Operator (TempoStack on object storage) for storage and query; Jaeger is deprecated. App teams instrument once with OpenTelemetry and get traces in the console's tracing plugin; the platform runs the collector and Tempo.
• Power monitoring (Kepler-based per-pod energy estimates for sustainability reporting, with Tech Preview status that has moved over the releases) and the Cluster Observability Operator (COO, the umbrella for the console UI plugins, a troubleshooting panel correlating alerts with logs and traces, and extra MonitoringStack instances when a team needs its own Prometheus) round out the family.
• must-gather = oc adm must-gather collects cluster state, operator logs and, with the logging operator's image, the full logging configuration and collector logs for a Red Hat support case; monitoring data from the last few hours is included. Post 25 walks through it.

For app teams, the per-project Observe view (Dashboards, Metrics, Alerts, Logs, Events) is the product you deliver. Show them that tab, the three UWM objects and the log query for their namespace at onboarding, and most "can you check the logs for us?" tickets disappear.

Troubleshooting the monitoring and logging stack itself

The observability platform is a workload too, and when it breaks you are the on-call. The failure modes you will actually see, in order of frequency:

Prometheus OOMKilled or slow: cardinality

Prometheus memory tracks the number of active series. One app labelling a metric with a request or customer ID creates a series per request; a million series later, Prometheus is OOMKilled every twenty minutes. Confirm with prometheus_tsdb_head_series, then find the offender:

# Which metric names have the most series (expensive query; run once, on Thanos Querier)
topk(10, count by (__name__) ({__name__=~".+"}))

# Which scrape jobs produce the most series
topk(10, count by (job) ({__name__=~".+"}))

# Series per namespace in user workload monitoring
topk(10, count by (namespace) ({__name__=~".+", namespace!~"openshift-.*|kube-.*"}))

Cheaper than that PromQL is the TSDB status endpoint, curl -H "Authorization: Bearer $TOKEN" https://$HOST/api/v1/status/tsdb, which returns top series counts by metric and label pair without a full scan. The fix is never "add memory and move on": drop the label with metricRelabelings in the team's ServiceMonitor, cap them with enforcedSampleLimit, and put "no unbounded label values" in the platform standard. More memory through the ConfigMap is temporary relief while the team fixes their exporter.

Targets down or metrics missing

Start at Observe → Targets, or up == 0. Then the same four causes every time: the Service selector matches no pods (empty Endpoints), the ServiceMonitor's port is not a named Service port, the namespace is labelled openshift.io/user-monitoring=false or starts with openshift-, or a NetworkPolicy blocks ingress from the monitoring namespaces. The last one appears the week after security rolls out default-deny, and the fix belongs in the project template:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-openshift-monitoring
  namespace: payments-prod
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: monitoring
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-user-workload-monitoring

$ oc -n openshift-user-workload-monitoring logs prometheus-operator-6f8c9d7b5-qz4rn -c prometheus-operator | grep -i "payments-prod" | tail -2
level=warn ts=2026-09-08T09:12:44Z caller=operator.go msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=payments-prod/payments-api
$ oc -n payments-prod get endpoints payments-api
NAME           ENDPOINTS                           AGE
payments-api   10.128.6.12:9090,10.131.2.40:9090   14d

The Prometheus Operator logs tell you when it rejected a ServiceMonitor (unsupported fields, cross-namespace selectors in UWM); the Endpoints object tells you whether there is anything to scrape at all.

Alertmanager not sending

Alerts fire in the console but nobody is notified. In order: oc -n openshift-monitoring logs alertmanager-main-0 -c alertmanager for notify retry cancelled or receiver HTTP errors; amtool check-config alertmanager.yaml before replacing the secret; amtool config routes test --config.file=alertmanager.yaml severity=critical namespace=openshift-etcd to see which route an alert takes; the egress path (a proxy or firewall between the cluster and PagerDuty is the usual bank culprit; set http_config.proxy_url); and a silence someone forgot.

LokiStack unhealthy

oc get lokistack logging-loki -n openshift-logging -o jsonpath='{.status.conditions}' gives the operator's view. Ingesters in CrashLoopBackOff almost always mean object storage: wrong keys, a missing bucket, an endpoint blocked by the proxy, or no CA for an on-prem S3; the ingester log says it plainly (AccessDenied, NoSuchBucket, x509: certificate signed by unknown authority). Ingesters and the compactor also need PVCs for their write-ahead log, so a Pending PVC means the wrong StorageClass. Queries that time out mean an under-sized querier, which is a size: change.

Collector pods CrashLooping on one node

One unhealthy collector-xxxxx pod while the rest are fine is a node problem, not a config problem. Vector opens an inotify watch per container log file, so a node running hundreds of pods hits fs.inotify.max_user_instances or max_user_watches and Vector fails with "too many open files". Check with oc debug node/<node> -- chroot /host sysctl fs.inotify and raise the limit for the pool with a MachineConfig sysctl (Post 20). The other cause is a full /var: Vector buffers to /var/lib/vector when an output is slow, and a Splunk outage plus a small disk fills the node.

On-call, runbooks and SLO thinking

A page must answer three questions: what is wrong, how bad, and where is the runbook. OpenShift's important alerts carry a runbook_url annotation pointing at Red Hat's public runbooks repository; the bank standard is that every PrometheusRule has one pointing at the internal wiki, which is why the Alertmanager template earlier prints it in Slack.

SLO thinking = defining "the platform is healthy" as a number and alerting on the trend toward breaking it rather than on every symptom. An SLI (service level indicator) is the measurement; an SLO (objective) is the target. Sensible platform SLIs: API server availability and latency (apiserver_request_total by code plus the p99 query; KubeAPIErrorBudgetBurn is already a burn-rate alert on this), ingress success rate (haproxy_server_http_responses_total by code) and scheduling latency. The golden signals (latency, traffic, errors, saturation) are the checklist for every service dashboard.

Alert fatigue is the practical enemy. Tuning tools, in order: raise for: so flapping conditions do not page; demote to warning anything that does not need a human within 30 minutes; group and inhibit (KubeNodeNotReady should suppress the fifteen KubePodNotReady underneath it); route app alerts to app teams; review the page log monthly and fix or delete every alert that fired without action. A page that is always acknowledged and closed will be ignored on the night it matters.

What a bank expects during an incident (Post 25 is the full playbook): acknowledge within the SLA (often 15 minutes for Sev-1), open or confirm the ServiceNow incident, join the bridge, communicate on a fixed cadence, preserve evidence (must-gather, audit extracts, the ALERTS timeline), and drive an RCA whose timeline starts at the first metric deviation, not the first human noticing. If the alert fired late or not at all, that is a monitoring action item with your name on it, which is where this post started.

Likely interview questions

Next up: the OpenShift troubleshooting playbook and incident response, where the alerts and logs from this post become an ordered method for node, network, storage and application failures, must-gather for Red Hat support, and the RCA a bank expects afterwards.