Operators and OLM: How OpenShift Extends Itself

Ask an OpenShift administrator what they did today and the answer almost always contains the word "operator". The ingress controller they tuned, the logging stack they upgraded, the cert-manager install that sat in Pending for an hour, the storage cluster that refused to follow the cluster to the next minor: every one is an operator, and the machinery that installs, upgrades and heals them is the Operator Lifecycle Manager. The job description compresses this into "manage OpenShift Operators". In practice it means knowing six API objects by heart, having an ordered method for the ways an install gets stuck, and explaining how a bank decides which operators are allowed on the cluster at all. After this post you can do all three, with the exact commands.

Why operators are the whole story on OpenShift

You met the controller idea in Post 1: a loop that watches desired state, compares it with actual state, and acts on the difference. OpenShift applies that idea to itself. DNS, the console, the image registry, the OAuth server, the ingress routers, the machine configuration on every node, the monitoring stack, even the tool that upgrades the cluster: each is an operator with its own custom resources and its own idea of "healthy". Nothing on an OpenShift cluster is installed by a person running a script; it is declared, and an operator makes it true.

So when you tune the router you edit an IngressController, when you add a kernel argument you create a MachineConfig (Post 20), when a team needs certificates you install cert-manager and hand them an Issuer, and when logging breaks the first question is "is the logging operator healthy". Operators are the interface through which you touch everything else, which is why the JD lists them beside RBAC and MachineConfig as a daily duty.

The operator pattern, recapped properly

An operator is three things bundled together: one or more Custom Resource Definitions (CRDs, which teach the API server a new kind of object such as Certificate or LokiStack), a controller (a long-running pod that watches those objects), and the reconciliation loop inside it, which turns each object's spec into real Deployments, Secrets and Services. You write kind: LokiStack with size: 1x.small; the Loki operator works out that this means ingesters, queriers, a gateway, storage credentials and PVCs, creates them, and keeps checking that they still match.

What separates an operator from a package is what happens after install. A Helm chart (Post 12) renders templates and applies them once; after helm install nothing is running on the chart's behalf, so if the database primary dies at 3 a.m., Helm does not know. An operator runs the whole time, and a good one encodes what a human expert would do: upgrade in the right order, take a consistent backup, promote a replica on failure, resize storage without losing data. That "everything after install" is day-2 operations, the reason the pattern exists. Helm answers "how do I deploy this"; an operator answers "how do I run this for three years".

The Operator Framework grades this in five capability levels, shown on every OperatorHub tile: I Basic Install (deploys from a CR), II Seamless Upgrades (upgrades software and CRDs preserving data), III Full Lifecycle (backup, restore, failover, scaling), IV Deep Insights (metrics, alerts, meaningful status) and V Auto Pilot (tunes and self-corrects). For stateful software in a bank's production, Level III is the floor, because "we restored using the operator's own Restore CR" is a sentence auditors accept.

Two families: cluster operators and OLM-managed operators

Every operator on an OpenShift cluster belongs to one of two families, and confusing them is the most common interview mistake on this topic.

Cluster operators make OpenShift OpenShift. They ship inside the release payload, the single signed image that defines an OpenShift version, and they are installed, upgraded and watched by the Cluster Version Operator (CVO). You do not choose them and cannot uninstall them; oc adm upgrade rolls all of them to the new version in a fixed order. You read them with oc get clusteroperators, short form oc get co (Post 19):

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.21   True        False         False      6d
dns                                        4.16.21   True        False         False      12d
ingress                                    4.16.21   True        False         False      12d
machine-config                             4.16.21   True        False         False      12d
marketplace                                4.16.21   True        False         False      12d
monitoring                                 4.16.21   True        False         False      12d
network                                    4.16.21   True        False         False      12d
operator-lifecycle-manager                 4.16.21   True        False         False      12d
operator-lifecycle-manager-catalog         4.16.21   True        False         False      12d
operator-lifecycle-manager-packageserver   4.16.21   True        False         False      6d
storage                                    4.16.21   True        False         False      12d

Notice the three operator-lifecycle-manager rows and marketplace: OLM is itself delivered as cluster operators. That is the bridge between the families. The CVO keeps OLM healthy, and OLM keeps everything you install on top healthy.

OLM-managed operators are everything you add: logging, cert-manager, GitOps, storage, compliance scanning. They come from a catalog shown in the console as OperatorHub, are installed with a Subscription, and each upgrades on its own schedule. They never appear in oc get co; you list them with oc get csv -A.

OLM classic: the six objects and how they connect

OpenShift 4.14 through 4.19 ship the original OLM, called OLM v0 or classic OLM, as the default, fully supported way to install operators. It runs in openshift-operator-lifecycle-manager as two controllers: catalog-operator, which reads catalogs, resolves what to install and writes InstallPlans, and olm-operator, which takes an approved InstallPlan and creates the operator's Deployment, RBAC and CRDs. Around them sit six API objects in the operators.coreos.com group. Learn the chain in order, because troubleshooting is walking the chain to the first broken link.

CatalogSource: where operators come from

A CatalogSource points at an index image, a container image holding a database of operator bundles and the upgrade paths between them; OLM runs it as a pod serving the index over gRPC. The Marketplace cluster operator creates four default sources in openshift-marketplace on every connected cluster: redhat-operators (built and supported by Red Hat), certified-operators (partner-built, Red Hat certified, partner supported), community-operators (upstream, unsupported) and redhat-marketplace (paid third-party software).

$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     12d
community-operators   Community Operators   grpc   Red Hat     12d
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     12d
redhat-operators      Red Hat Operators     grpc   Red Hat     12d

$ oc get catalogsource redhat-operators -n openshift-marketplace \
    -o jsonpath='{.spec.image}{"\n"}{.status.connectionState.lastObservedState}{"\n"}'
registry.redhat.io/redhat/redhat-operator-index:v4.16
READY

Two things to notice. The image tag is v4.16, the cluster's minor: when the cluster upgrades to 4.17 the marketplace operator repoints the default sources to the v4.17 index, and operator versions not built for 4.17 quietly disappear from OperatorHub. And lastObservedState: READY is the health check for a catalog; anything else means no install or upgrade from it will resolve.

PackageManifest: the listing page

Once a catalog is READY, OLM's packageserver exposes each operator in it as a read-only PackageManifest. This is what OperatorHub renders, and it is how you find the exact package name, channels and install modes for a Subscription without guessing:

$ oc get packagemanifests -n openshift-marketplace | grep -i logging
cluster-logging                                    Red Hat Operators     12d
loki-operator                                      Red Hat Operators     12d

$ oc describe packagemanifest cluster-logging -n openshift-marketplace
Name:         cluster-logging
Status:
  Catalog Source:    redhat-operators
  Default Channel:   stable-6.2
  Package Name:      cluster-logging
  Channels:
    Name:         stable-6.1
    Current CSV:  cluster-logging.v6.1.4
    Current CSV Desc:
      Install Modes:
        Type:       OwnNamespace
        Supported:  true
        Type:       AllNamespaces
        Supported:  false
    Name:         stable-6.2
    Current CSV:  cluster-logging.v6.2.3
  ...

Channel names and versions differ by catalog date, so always describe the PackageManifest on the cluster you are working on rather than copying from a blog.

OperatorGroup: which namespaces the operator may serve

An OperatorGroup is a namespaced object that tells OLM which namespaces an operator installed here may watch, and therefore where OLM must create its RBAC. Its one important field is spec.targetNamespaces. Own namespace: it lists only the namespace the OperatorGroup lives in; this is how cert-manager and logging are installed. Single namespace: it lists one other namespace; rare. All namespaces: omitted or empty; the operator is cluster-wide and its CSV is copied into every namespace. The openshift-operators namespace ships with an OperatorGroup named global-operators for this, which is why console installs of "all namespaces" operators land there.

The rule that causes the most tickets: a namespace may have exactly one OperatorGroup. Zero and nothing installs; two and every CSV in the namespace fails with reason TooManyOperatorGroups. A subtler failure is UnsupportedOperatorGroup: an all-namespaces OperatorGroup for an operator whose PackageManifest says AllNamespaces: Supported: false. Match the install mode to what the operator supports.

Subscription: "install this and keep it updated"

A Subscription is the object you author. It names the package, the catalog, the channel (the upgrade track, which OLM only ever walks forward), and installPlanApproval: Automatic installs the first version and every later one as soon as the catalog publishes it; Manual makes every InstallPlan, including the first, wait for a human. startingCSV starts from a specific version instead of the channel head, which with Manual approval is the closest thing classic OLM has to pinning. The optional config block sets node selectors, tolerations, resources and environment variables on the operator's own Deployment; a bank uses it to keep operators on infra nodes. Once created, status.installedCSV is what runs now and status.currentCSV is what the channel offers; when they differ, an upgrade is waiting.

InstallPlan: the permissions prompt

When catalog-operator resolves a Subscription it writes an InstallPlan listing everything it intends to create: the CSV, CRDs, ServiceAccounts, Roles and ClusterRoles. Reading the plan is how you learn what an operator will be allowed to do before you let it in. With Manual approval it sits in phase RequiresApproval until someone flips one boolean:

$ oc get installplan -n cert-manager-operator
NAME            CSV                             APPROVAL   APPROVED
install-7k2xr   cert-manager-operator.v1.14.1   Manual     false

$ oc get ip install-7k2xr -n cert-manager-operator \
    -o jsonpath='{.spec.clusterServiceVersionNames}{"\n"}'
["cert-manager-operator.v1.14.1"]

$ oc patch installplan install-7k2xr -n cert-manager-operator \
    --type merge -p '{"spec":{"approved":true}}'
installplan.operators.coreos.com/install-7k2xr patched

ip is the short name. Always print spec.clusterServiceVersionNames before approving, for a reason the upgrade section explains.

ClusterServiceVersion: the installed operator

The ClusterServiceVersion (CSV) is the operator's full description at one version: its Deployment spec, the CRDs it owns, the RBAC it requires, its install modes, and replaces, the previous version in the chain. olm-operator drives it through phases: Pending (checking requirements such as CRDs and ServiceAccounts), InstallReady, Installing, Succeeded (operator pod ready) or Failed; during an upgrade the old CSV passes through Replacing and Deleting. The CSV is what you describe when something is wrong and what you delete to uninstall. For an all-namespaces operator it is copied into every namespace with the label olm.copiedFrom; the copies are read-only and deleting one just makes OLM recreate it.

A complete install, as YAML and in the console

Here is the chain for the cert-manager Operator for Red Hat OpenShift, which uses own-namespace mode. Three objects in one file; the operator will in turn create the actual cert-manager controllers in a separate cert-manager namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cert-manager-operator
  namespace: cert-manager-operator
spec:
  targetNamespaces:
  - cert-manager-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager-operator
spec:
  name: openshift-cert-manager-operator   # package name from the PackageManifest
  source: redhat-operators               # CatalogSource
  sourceNamespace: openshift-marketplace
  channel: stable-v1
  installPlanApproval: Manual
  # startingCSV: cert-manager-operator.v1.14.1   # optional pin
  config:                                # optional: keep the operator on infra nodes
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    tolerations:
    - key: node-role.kubernetes.io/infra
      effect: NoSchedule

$ oc apply -f cert-manager-install.yaml
namespace/cert-manager-operator created
operatorgroup.operators.coreos.com/cert-manager-operator created
subscription.operators.coreos.com/openshift-cert-manager-operator created

$ oc get sub,ip,csv -n cert-manager-operator
NAME                                                                PACKAGE                           SOURCE             CHANNEL
subscription.operators.coreos.com/openshift-cert-manager-operator   openshift-cert-manager-operator   redhat-operators   stable-v1

NAME                                             CSV                             APPROVAL   APPROVED
installplan.operators.coreos.com/install-7k2xr   cert-manager-operator.v1.14.1   Manual     false

No CSV yet, because the plan is waiting. Approve it with the oc patch above and watch the chain complete:

$ oc get csv -n cert-manager-operator -w
NAME                            DISPLAY                                       VERSION   REPLACES   PHASE
cert-manager-operator.v1.14.1   cert-manager Operator for Red Hat OpenShift   1.14.1               Pending
cert-manager-operator.v1.14.1   cert-manager Operator for Red Hat OpenShift   1.14.1               InstallReady
cert-manager-operator.v1.14.1   cert-manager Operator for Red Hat OpenShift   1.14.1               Installing
cert-manager-operator.v1.14.1   cert-manager Operator for Red Hat OpenShift   1.14.1               Succeeded

$ oc get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5f7b8c9d6-x2k4p               1/1     Running   0          40s
cert-manager-cainjector-6d8f4b7c5-m9q2w    1/1     Running   0          40s
cert-manager-webhook-7c9d5f8b4-h3n7z       1/1     Running   0          40s

The console path is Operators → OperatorHub, search "cert-manager", pick the tile published by Red Hat (not the community one), Install, channel stable-v1, installation mode "A specific namespace on the cluster" with the recommended namespace, update approval Manual, Install. The "Approve" button that appears is the same patch on the same InstallPlan, and the status column under Installed Operators is the CSV phase. Every console click maps to one of these six objects, which is what lets you script and troubleshoot it. The logging stack is the same pattern twice: the Loki Operator in openshift-operators-redhat and the Cluster Logging operator in openshift-logging, each with an own-namespace OperatorGroup and a Subscription on a matching stable-6.x channel, with the namespaces labelled openshift.io/cluster-monitoring: "true" so platform Prometheus scrapes them. Post 24 builds on that.

Upgrading operators without surprises

An upgrade is triggered two ways: the catalog publishes a newer version on the channel your Subscription follows, or you edit the Subscription to a newer channel (stable-6.1 to stable-6.2). Either way catalog-operator resolves a new CSV and writes a new InstallPlan; once approved, the operator's Deployment rolls to the new image, new CRD versions are applied, and a well-written operator then upgrades its operands (the software it manages) by its own logic.

Automatic vs Manual, and why prod is Manual

Automatic is fine in a sandbox. In a bank's production cluster it is a change with no ticket, no approver and no window, happening whenever Red Hat publishes a bundle, possibly during quarter-end batch. Regulated shops run production Subscriptions with installPlanApproval: Manual, and approval becomes a step in the change process: the pending InstallPlan is the evidence an upgrade exists, the release notes and compatibility check go into the change record, the change board approves, and the engineer runs the oc patch inside the window with a tested rollback plan. Non-prod runs Automatic so the team meets new versions there first. The daily check for pending upgrades, which belongs in the health-check script from Post 32:

$ oc get subscriptions.operators.coreos.com -A \
    -o custom-columns='NS:.metadata.namespace,PACKAGE:.spec.name,CHANNEL:.spec.channel,APPROVAL:.spec.installPlanApproval,INSTALLED:.status.installedCSV,AVAILABLE:.status.currentCSV'
NS                          PACKAGE                           CHANNEL       APPROVAL   INSTALLED                           AVAILABLE
cert-manager-operator       openshift-cert-manager-operator   stable-v1     Manual     cert-manager-operator.v1.14.1       cert-manager-operator.v1.15.0
openshift-gitops-operator   openshift-gitops-operator         gitops-1.14   Manual     openshift-gitops-operator.v1.14.2   openshift-gitops-operator.v1.14.2
openshift-logging           cluster-logging                   stable-6.2    Manual     cluster-logging.v6.2.3              cluster-logging.v6.2.3

$ oc get installplan -A | grep -v true
NAMESPACE               NAME            CSV                             APPROVAL   APPROVED
cert-manager-operator   install-9p4wd   cert-manager-operator.v1.15.0   Manual     false

Ordering against cluster upgrades

Operators and the cluster upgrade on separate tracks, but they are not independent. A bundle can declare the highest OpenShift minor it supports through the olm.maxOpenShiftVersion property in its CSV. The OLM cluster operator reads every installed CSV, and if one says "not beyond 4.16" it sets Upgradeable=False and the CVO refuses the minor upgrade until you act:

$ oc adm upgrade
Cluster version is 4.16.21

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking minor version upgrades to 4.17.0 or higher:
  - maximum supported OCP version for openshift-gitops-operator.v1.11.7 is 4.16

Channel: stable-4.16
...

So the working order before any minor cluster upgrade, part of the checklist in Post 20, is: list installed operators (oc get csv -A -l '!olm.copiedFrom') and check each one's documented compatibility (ODF, ACM and Service Mesh have their own matrices and move in lockstep with specific minors); upgrade anything that does not support the target minor before the cluster; in a disconnected cluster, mirror the target minor's catalog index first; upgrade the cluster; then upgrade operators whose newer version requires the new minor after it (ODF 4.17 wants OpenShift 4.17).

Pinning, and why there is no rollback

Classic OLM has no first-class "install exactly version X and stay there". The pin is startingCSV plus installPlanApproval: Manual, and it is how a bank keeps every cluster in an environment on the same operator version. OLM v1, below, adds real version ranges.

When an upgrade changes CRD versions

The most disruptive upgrades change the shape of CRs your teams already have: a new CRD version, a changed storage version, or a removed old version. OLM runs a safety check during the InstallPlan, validating every existing CR against the new schema and failing the plan rather than orphaning data. The example every admin remembers is Logging 5 to Logging 6, where ClusterLogging was retired and forwarding moved to ClusterLogForwarder in a new group, observability.openshift.io/v1. That was a migration with a written plan, not a Tuesday approval. Before approving, read the release notes for API changes, run oc get crd <name> -o jsonpath='{.spec.versions[*].name}' to see which versions exist today, and test in non-prod with copies of real CRs.

Troubleshooting OLM: an ordered method

Because OLM is a chain, troubleshooting is walking from the object you created toward the pod you expected and stopping at the first link that is missing or unhappy. Say this order out loud in the interview:

1. Subscription conditions. oc describe sub <name> -n <ns>: CatalogSourcesUnhealthy, ResolutionFailed, InstallPlanPending, InstallPlanFailed. This list points at the broken link most of the time.
2. InstallPlan. oc get ip -n <ns>. None means resolution never succeeded; APPROVED false means it is waiting for you.
3. CSV phase and reason. oc get csv -n <ns>, then oc describe csv for Status.Reason and Status.Message.
4. OperatorGroup. oc get og -n <ns>: exactly one, with a supported targetNamespaces.
5. Catalog health. oc get catalogsource -n openshift-marketplace and the pods behind them.
6. The operator's own pod once the CSV is Succeeded: oc logs deploy/<operator> -n <ns> and the status of its CRs.
7. OLM's own logs when the chain is silent: oc logs deploy/catalog-operator -n openshift-operator-lifecycle-manager for resolution and InstallPlans, deploy/olm-operator for CSV installs.

CSV stuck in Pending or Failed

Pending with reason RequirementsNotMet is OLM waiting for something the operator needs: a CRD, a ServiceAccount, an API it depends on, or an RBAC rule OLM has not managed to grant. The requirement list in describe names it:

$ oc describe csv cert-manager-operator.v1.14.1 -n cert-manager-operator
...
Status:
  Phase:    Pending
  Reason:   RequirementsNotMet
  Message:  one or more requirements couldn't be found
  Requirement Status:
    Group:    apiextensions.k8s.io
    Kind:     CustomResourceDefinition
    Name:     certificates.cert-manager.io
    Status:   Present
    Kind:     ServiceAccount
    Name:     cert-manager-operator-controller-manager
    Status:   PresentNotSatisfied
    Dependents:
      Group:    rbac.authorization.k8s.io
      Kind:     PolicyRule
      Status:   NotSatisfied
      Message:  cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["nodes"]}

The ServiceAccount exists but lacks a cluster rule the CSV asks for: usually the InstallPlan's RBAC step failed part way (oc describe ip shows a Failed phase and why) or someone "cleaned up" a ClusterRoleBinding OLM had created. Delete the CSV and the Subscription produces a fresh plan. Failed with InstallCheckFailed or ComponentUnhealthy means the Deployment exists but never became ready, so go to the pod with describe and logs as in Post 14: an unpullable image (very common in disconnected clusters), a spec.config node selector no node matches, or a webhook that cannot get its certificate.

InstallPlan never appears

Subscription present, catalog READY, no InstallPlan after a couple of minutes: look at the OperatorGroup. None, and OLM has nowhere to scope the install. Two, and the CSV reports TooManyOperatorGroups. A targetNamespaces asking for an unsupported install mode gives UnsupportedOperatorGroup.

$ oc get og -n openshift-logging
NAME                AGE
openshift-logging   3m
platform-og         1m

$ oc get csv -n openshift-logging \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.reason}{"\n"}{end}'
cluster-logging.v6.2.3	Failed	TooManyOperatorGroups

Delete the extra OperatorGroup and the CSV recovers on its own. The realistic cause is two engineers, or a GitOps application and a console click, both creating one.

"constraints not satisfiable"

This is the resolver's error, surfaced as a ResolutionFailed condition on the Subscription. It reads like a logic proof, but there are only a few root causes:

$ oc get sub cluster-logging -n openshift-logging \
    -o jsonpath='{.status.conditions[?(@.type=="ResolutionFailed")].message}{"\n"}'
constraints not satisfiable: no operators found from catalog redhat-operators in namespace openshift-marketplace referenced by subscription cluster-logging, subscription cluster-logging exists

• "no operators found from catalog ... referenced by subscription": wrong package, channel or source name, or the catalog for this cluster minor no longer carries that channel. Check oc describe packagemanifest.
• "clusterserviceversion X exists and is not referenced by a subscription": an orphan CSV left by an earlier install whose Subscription was deleted. Delete the orphan.
• "subscription A requires at least one of ... subscription B requires ...": two Subscriptions in one namespace want incompatible versions of a shared dependency, or an operator needs an API nothing in any enabled catalog provides (typical after disabling the community catalog). Separate namespaces, or mirror the missing dependency.

Related: a Failed InstallPlan blocks every later resolution in its namespace until you delete it and the CSV it created.

CatalogSource unhealthy, and disconnected clusters

If lastObservedState is anything but READY, the catalog pod itself is the problem:

$ oc get pods -n openshift-marketplace
NAME                                    READY   STATUS             RESTARTS   AGE
certified-operators-h8k2p               1/1     Running            0          2d
community-operators-fq7wn               0/1     ImagePullBackOff   0          14m
marketplace-operator-6b8d9c7f5-tl4vz    1/1     Running            0          12d
redhat-operators-x9m3c                  1/1     Running            0          2d

$ oc get catalogsource community-operators -n openshift-marketplace \
    -o jsonpath='{.status.connectionState.lastObservedState}{"\n"}'
TRANSIENT_FAILURE

On a connected cluster the usual causes are an egress proxy that blocks registry.redhat.io, an expired pull secret, or a broken index that heals on the next poll. On a bank's disconnected cluster, with no route to the internet at all, the default sources will always fail and that is expected: you disable them and replace them with mirrored catalogs. The tool is oc-mirror. You describe what you want in an ImageSetConfiguration; it pulls the index and every referenced bundle and operand image, pushes them to your internal registry, and generates the cluster resources:

apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    packages:
    - name: cluster-logging
      channels:
      - name: stable-6.2
    - name: loki-operator
      channels:
      - name: stable-6.2
    - name: openshift-cert-manager-operator
    - name: compliance-operator
    - name: openshift-gitops-operator

The generated resources are a CatalogSource whose spec.image points at the mirrored index (for example mirror.bank.internal:8443/redhat/redhat-operator-index:v4.16, with updateStrategy.registryPoll.interval so it re-reads on a schedule) and an ImageDigestMirrorSet (IDMS, the 4.13+ replacement for ImageContentSourcePolicy) telling every node "when a pod asks for registry.redhat.io, pull from the mirror instead". The IDMS changes CRI-O's registry config through the machine-config operator, so expect a rolling node update. Operationally, every operator upgrade in a disconnected bank starts with a mirror run, and a CSV that fails with an image pull error almost always means "that operand image was not in the ImageSetConfiguration"; oc describe pod will show the internal registry returning 404.

Uninstalling cleanly

Uninstalling is two deletions, and it deliberately leaves things behind:

$ oc delete sub openshift-cert-manager-operator -n cert-manager-operator
subscription.operators.coreos.com "openshift-cert-manager-operator" deleted

$ oc delete csv cert-manager-operator.v1.14.1 -n cert-manager-operator
clusterserviceversion.operators.coreos.com "cert-manager-operator.v1.14.1" deleted

$ oc get crd | grep cert-manager.io
certificaterequests.cert-manager.io     2026-08-02T14:11:09Z
certificates.cert-manager.io            2026-08-02T14:11:09Z
clusterissuers.cert-manager.io          2026-08-02T14:11:09Z
issuers.cert-manager.io                 2026-08-02T14:11:09Z

Deleting the Subscription stops upgrades. Deleting the CSV removes the operator Deployment and its RBAC. What remains, on purpose: the CRDs, every CR teams created, and the operands (for cert-manager, the controllers in the cert-manager namespace and every issued certificate Secret). OLM never deletes CRDs, because deleting a CRD deletes every object of that kind cluster-wide. So the software keeps running with nobody reconciling it: sometimes what you want (swap operators, keep the workload), sometimes a silent time bomb (certificates that will never renew). The console's Uninstall Operator dialog can delete operand instances first; it does not delete CRDs either.

Leftover CRDs are also behind the "reinstall is stuck" ticket: a different version ships a CRD whose schema conflicts with the one still on the cluster, or a CRD you deleted hangs with a deletionTimestamp because a CR still holds a finalizer the now-absent operator was supposed to process. For the second, clear the finalizer (oc patch <kind> <name> --type merge -p '{"metadata":{"finalizers":[]}}'), let the CRD deletion complete, then reinstall. The cluster-scoped Operator object is your inventory of what OLM still thinks it owns, even after the CSV is gone:

$ oc get operators
NAME                                                     AGE
cluster-logging.openshift-logging                        41d
compliance-operator.openshift-compliance                 60d
openshift-cert-manager-operator.cert-manager-operator    12m
openshift-gitops-operator.openshift-gitops-operator      90d

$ oc get operator openshift-cert-manager-operator.cert-manager-operator \
    -o jsonpath='{range .status.components.refs[*]}{.kind}{"\t"}{.name}{"\n"}{end}'
CustomResourceDefinition	certificates.cert-manager.io
CustomResourceDefinition	issuers.cert-manager.io
ClusterRole	cert-manager-operator-controller-manager
...

OLM v1: what is changing

Red Hat has been building a redesigned lifecycle manager, OLM v1, alongside the classic one. It reached general availability in the 4.18 timeframe and in current releases it coexists with classic OLM: the OperatorHub console flow, Subscriptions and CSVs still work and remain the primary supported path, while OLM v1 is available for operators packaged to work with it. Treat the details as "know the shape, check the docs for your version", not as a drop-in replacement yet.

It has two components, catalogd (serves catalog content) and operator-controller (installs and upgrades extensions), and two main APIs in the olm.operatorframework.io group. A ClusterCatalog is the cluster-scoped equivalent of a CatalogSource; the default Red Hat catalogs appear as ClusterCatalogs automatically. A ClusterExtension replaces the OperatorGroup + Subscription + InstallPlan + CSV chain with one object:

apiVersion: olm.operatorframework.io/v1
kind: ClusterExtension
metadata:
  name: cert-manager
spec:
  namespace: cert-manager-operator
  serviceAccount:
    name: cert-manager-installer
  source:
    sourceType: Catalog
    catalog:
      packageName: openshift-cert-manager-operator
      channels:
      - stable-v1
      version: "1.15.x"
      upgradeConstraintPolicy: CatalogProvided

Three changes matter for a platform team. Real version ranges: version: "1.15.x" pins to a minor line, the pinning classic OLM never had. Least-privilege installs: the extension is installed with a ServiceAccount you provide, carrying exactly the RBAC the bundle needs, instead of OLM acting as cluster-admin on your behalf; a real win for a bank's security review and more work up front. No OperatorGroups and no copied CSVs: an extension is installed once, cluster-scoped, with status on the object itself (oc get clusterextension). OLM v1 supports a subset of what classic OLM does, with the boundaries around install modes, dependencies and webhooks moving release by release, which is why classic OLM is still what most bank clusters run day to day. The safe interview framing: "I run classic OLM in production, I know OLM v1 replaces the Subscription chain with ClusterExtension and adds version ranges and scoped service accounts, and I would check the release notes for the exact support boundaries on our version."

The operators a bank platform team actually runs

You will be asked "which operators have you managed", so know the landscape. Each row is one you should be able to say two sentences about: what it does and the CR you touch.

Writing your own operator, and when not to

The Operator SDK is the toolkit for building operators, with three starting points. A Go operator is a real controller with full control of the reconcile logic; every serious operator in the table is written this way. An Ansible operator maps a CR to an Ansible role, so each reconcile runs a playbook; a fast way for an ops team to wrap existing automation in a Kubernetes API. A Helm operator wraps an existing chart, the CR's spec becoming the chart's values; by construction it is capability Level I or II, because a chart has no failover logic to encode. Red Hat's downstream build of the SDK CLI is deprecated from OpenShift 4.16; the upstream project continues, and the bundles it produces are still what OLM installs.

Should a bank's platform team write one? Rarely, and the interviewer wants to hear you say so with reasons. It is worth it for software you run many times across clusters that needs genuine day-2 logic, or for a "golden path" abstraction where a team creates one CR and gets a namespace, quotas, NetworkPolicies and an Argo CD Application. It is not worth it for "run this check nightly", "rotate this credential monthly" or "report on X": a CronJob running a Python script against the API, as in Post 32, does that with a fraction of the maintenance and no CRD to version forever. An operator is a product you own for years; a script is a task.

The operating model at a bank

Three questions define how a regulated shop manages operators: who may install them, which catalogs are allowed, and how you prove what is installed.

Who installs: the platform team, through Git

By default OLM ships aggregated ClusterRoles that let a project admin create Subscriptions in their own project. A bank does not run that way: operators carry elevated RBAC and add CRDs everyone sees, so installation is a platform-team responsibility executed through the GitOps repository (Post 30). The Namespace, OperatorGroup and Subscription YAML lives in Git, Argo CD applies it, the pull request is the change record, and the InstallPlan approval in the change window is the one deliberately imperative step. Application teams request an operator through onboarding (Post 26); they never see OperatorHub. Check rather than assume:

$ oc auth can-i create subscriptions.operators.coreos.com -n team-payments --as=dev-lead@bank.example
no

$ oc auth can-i create subscriptions.operators.coreos.com -n team-payments \
    --as=system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller
yes

Which catalogs: curated

Community operators are unsupported, not vetted by Red Hat and sometimes abandoned, so a bank disables that catalog outright and keeps redhat-operators plus, after review, certified-operators. The switch is the cluster-scoped OperatorHub resource, which the marketplace operator honours:

$ oc patch operatorhub cluster --type merge \
    -p '{"spec":{"sources":[{"name":"community-operators","disabled":true},{"name":"redhat-marketplace","disabled":true}]}}'
operatorhub.config.openshift.io/cluster patched

$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     12d
redhat-operators      Red Hat Operators     grpc   Red Hat     12d

On a disconnected cluster you set spec.disableAllDefaultSources: true instead and add only the mirrored CatalogSources from oc-mirror. The OperatorHub resource belongs in Git too, so a rebuilt cluster comes up with the same curated catalogs.

Proving what is installed: the inventory report

Auditors will ask "what operators are on production, which versions, and are any behind". The truth lives in CSVs and Subscriptions, so a short script answers it, and the same script becomes the drift check between clusters in one environment. In shell it is two lines; in Python it becomes a dated table per cluster stored as evidence (Post 32 builds that version):

$ oc get csv -A -l '!olm.copiedFrom' \
    -o custom-columns='NS:.metadata.namespace,OPERATOR:.spec.displayName,VERSION:.spec.version,PHASE:.status.phase'
NS                            OPERATOR                                       VERSION   PHASE
cert-manager-operator         cert-manager Operator for Red Hat OpenShift    1.14.1    Succeeded
openshift-compliance          Compliance Operator                            1.6.1     Succeeded
openshift-gitops-operator     Red Hat OpenShift GitOps                       1.14.2    Succeeded
openshift-logging             Red Hat OpenShift Logging                      6.2.3     Succeeded
openshift-operators-redhat    Loki Operator                                  6.2.3     Succeeded
openshift-storage             OpenShift Data Foundation                      4.16.5    Succeeded

The -l '!olm.copiedFrom' selector drops the read-only copies of all-namespaces operators, which would otherwise repeat once per namespace. Pair it with the Subscription view from the upgrade section (installed vs available) and you have the two reports every platform team ends up writing.

Likely interview questions

Next up: the operators you just learned to install become the observability stack itself, as Post 24 walks through Prometheus, Alertmanager, Loki and Insights on OpenShift, from platform alerts to forwarding application logs to a bank's SIEM.