CI/CD for Containers and GitOps with Argo CD

You have built Jenkins and GitHub Actions pipelines before, so this post does not teach you what a stage is. It teaches the two things the JD is really asking about: how a pipeline builds and proves a container image (scan, SBOM, signature, immutable tag), and how GitOps with Argo CD makes a cluster match Git instead of letting a pipeline push whatever it wants. A bank loves GitOps for one reason above all: every production change becomes a reviewed commit with an audit trail, and the cluster corrects itself when someone drifts from it. After this post you can draw the Argo CD architecture on a whiteboard, write an Application and an ApplicationSet from memory, explain sync waves versus hooks, handle secrets without ever committing one, and debug an app that is stuck OutOfSync.

CI proves the image; GitOps makes the cluster match Git

Separate the two halves cleanly, because interviewers probe whether you blur them. Continuous Integration (CI) = the pipeline that turns a commit into a tested, scanned, signed image in a registry; its output is an artifact plus evidence about that artifact. Continuous Delivery (CD) = getting that artifact running in an environment. The traditional way is push: the pipeline holds a kubeconfig and runs kubectl apply or helm upgrade. The GitOps way is pull: the pipeline only commits a new image tag to a config repository, and an agent inside the cluster (Argo CD) notices, pulls the change and reconciles the cluster to match.

A bank prefers pull because no CI system holds production credentials, every production change is a Git commit with an author, a reviewer and a timestamp, and the desired state of every cluster can be reconstructed from a repository at any point in time. When an auditor asks "who changed what in production in March, and who approved it," a push shop answers with Jenkins job logs and a spreadsheet; a GitOps shop answers with git log.

Anatomy of a container CI pipeline

Every container pipeline has the same skeleton. What separates a strong candidate from a script-copier is saying why each stage exists and what evidence it produces.

1. Checkout. Clone the app repo at the exact commit and record the SHA; it becomes part of the image tag and the provenance record.
2. Unit tests and lint. Fail fast on the cheapest checks. Include a Dockerfile linter (hadolint) and a manifest linter (kubeconform, helm lint) so bad YAML never reaches the config repo.
3. Build the image. Use a multi-stage Dockerfile: a build stage with compilers and test tools, then a tiny runtime stage (UBI minimal or distroless) that copies only the binary, so there are fewer CVEs to explain. The builder depends on where CI runs: Docker with BuildKit on a VM runner; buildah or podman for rootless, daemonless builds in a container; kaniko when the build must run as an unprivileged pod with no Docker socket. OpenShift defaults to buildah, which needs no privileged daemon and is what BuildConfigs and Tekton tasks use.
4. Generate an SBOM. SBOM (Software Bill of Materials) = a machine-readable inventory of every package in the image; syft produces one in SPDX or CycloneDX format. When the next Log4Shell lands, the security team greps SBOMs instead of rebuilding every image to see what is inside.
5. Vulnerability scan. trivy, grype or Clair scan the image (or the SBOM) against CVE databases. Policy: fail on CRITICAL, and on HIGH with a fix available. Scanning that never fails is theatre.
6. Sign the image. cosign from the sigstore project signs the image digest. Keyless signing = no long-lived private key: the pipeline proves its identity with an OIDC token (GitHub Actions or a Jenkins OIDC plugin), gets a short-lived certificate from Fulcio, and the signature is logged in the Rekor transparency log. The cluster can later verify "this image was built by this workflow in this repo," which a stolen key could never prove.
7. Push with immutable tags. Tag with the Git SHA and, on release, a semver tag: payments-api:1.14.2 and payments-api:sha-9f3c2d1. Never deploy latest: it is a moving pointer, not a version. Better still, reference images by digest (@sha256:...) in the config repo, because a tag can be re-pushed and a digest cannot.
8. Update the config repo. The pipeline opens a pull request against the deployment repository that bumps the image reference; a human (or a bot in lower environments) merges it.
9. GitOps deploys. Argo CD sees the merge and reconciles. CI never touches the cluster.

Two repositories, two jobs

GitOps almost always means two repos. The app repo holds source code and the Dockerfile. The config repo (also called the deployment or environment repo) holds the manifests, Helm values or Kustomize overlays for every environment. Keeping them apart means a code change and a config change are different PRs with different reviewers, and a busy app repo does not trigger a hundred Argo syncs.

The same pipeline in GitHub Actions, Jenkins and Tekton

GitHub Actions

Notice permissions.id-token: write: it lets the job request an OIDC token, which configure-aws-credentials exchanges for a short-lived role session, so no static AWS keys live in GitHub. The same token is what cosign uses for keyless signing.

name: build-payments-api
on: { push: { branches: [main] } }
permissions: { id-token: write, contents: read, packages: read }
env: { ECR: 123456789012.dkr.ecr.ca-central-1.amazonaws.com/payments-api }
jobs:
  build:
    runs-on: ubuntu-latest
    outputs: { digest: ${{ steps.push.outputs.digest }} }
    steps:
      - uses: actions/checkout@v4
      - run: make test lint
      - uses: aws-actions/configure-aws-credentials@v4
        with: { role-to-assume: arn:aws:iam::123456789012:role/gha-payments-ci, aws-region: ca-central-1 }
      - uses: aws-actions/amazon-ecr-login@v2
      - uses: docker/setup-buildx-action@v3
      - id: push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ env.ECR }}:sha-${{ github.sha }}
          cache-from: type=registry,ref=${{ env.ECR }}:buildcache
          cache-to: type=registry,ref=${{ env.ECR }}:buildcache,mode=max
      - uses: aquasecurity/trivy-action@0.28.0
        with: { image-ref: "${{ env.ECR }}@${{ steps.push.outputs.digest }}", severity: CRITICAL, exit-code: 1 }
      - uses: anchore/sbom-action@v0
        with: { image: "${{ env.ECR }}@${{ steps.push.outputs.digest }}", format: spdx-json }
      - uses: sigstore/cosign-installer@v3
      - run: cosign sign --yes "$ECR@${{ steps.push.outputs.digest }}"
  bump-config:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { repository: bank/payments-config, token: ${{ secrets.CONFIG_REPO_PAT }} }
      - run: |
          cd envs/dev && kustomize edit set image payments-api=$ECR@${{ needs.build.outputs.digest }}
      - uses: peter-evans/create-pull-request@v6
        with: { title: "dev: payments-api ${{ github.sha }}", branch: bump/payments-${{ github.sha }} }

The scan runs against the digest that was actually pushed, not the tag, and the config repo PR references that same digest, so nothing in the chain can be swapped underneath you.

Jenkins declarative pipeline on OpenShift

On OpenShift, Jenkins agents are pods launched from a pod template, so each build gets a clean, throwaway environment. The hard part is building without a Docker daemon: use a buildah container (rootless buildah needs the right SCC or the Red Hat-documented unprivileged build image) or hand the build to kaniko. Cross-cutting logic (scan, sign, open PR) lives in a shared library so fifty teams do not maintain fifty copies of the same Groovy.

@Library('platform-pipeline-lib') _
pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          serviceAccountName: jenkins-builder
          containers:
          - name: buildah
            image: registry.redhat.io/rhel9/buildah:latest
            command: ["sleep", "infinity"]
            securityContext: { runAsUser: 1000, capabilities: { add: [SETUID, SETGID] } }
          - name: tools
            image: quay.io/bank/ci-tools:2026.03    # trivy, syft, cosign, kustomize, gh
            command: ["sleep", "infinity"]
      '''
    }
  }
  environment { IMAGE = "quay.bank.internal/payments/payments-api" }
  stages {
    stage('Test')  { steps { sh 'make test lint' } }
    stage('Build') { steps { container('buildah') {
      sh 'buildah bud --isolation=chroot -t $IMAGE:sha-$GIT_COMMIT .'
      sh 'buildah push --digestfile digest.txt $IMAGE:sha-$GIT_COMMIT'
    } } }
    stage('Scan & SBOM') { steps { container('tools') {
      sh 'syft $IMAGE@$(cat digest.txt) -o spdx-json > sbom.json'
      sh 'trivy image --severity CRITICAL --exit-code 1 $IMAGE@$(cat digest.txt)'
    } } }
    stage('Sign') { steps { container('tools') { withCredentials([string(credentialsId: 'cosign-oidc', variable: 'SIGSTORE_ID_TOKEN')]) {
      sh 'cosign sign --yes $IMAGE@$(cat digest.txt)'
    } } } }
    stage('Bump config') { steps { container('tools') {
      platformLib.openConfigPR(repo: 'bank/payments-config', env: 'dev', image: "$IMAGE@" + readFile('digest.txt').trim())
    } } }
  }
  post { always { archiveArtifacts artifacts: 'sbom.json' } }
}

OpenShift Pipelines (Tekton)

Tekton = a Kubernetes-native CI system where every piece is a CRD. A Task is a sequence of steps that run as containers in one pod; a Pipeline wires Tasks together with parameters, results and workspaces (shared volumes); a PipelineRun is one execution of a Pipeline, the way a Job is one execution of a template; Triggers (EventListener, TriggerBinding, TriggerTemplate) turn a GitHub or GitLab webhook into a PipelineRun. OpenShift Pipelines is Red Hat's supported operator packaging of Tekton, with the tkn CLI and a console view.

A bank might standardize on it because pipelines are YAML in Git governed by the same RBAC, quotas and SCCs as any other workload (Post 22), there is no Jenkins controller to patch and no plugin sprawl, and catalog Tasks (git-clone, buildah, trivy) become a curated internal library the platform team pins and signs. The cost is that Tekton is lower-level than Jenkins, so the platform team must build the golden-path pipeline for app teams.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata: { name: build-and-scan }
spec:
  params:
    - { name: git-url, type: string }
    - { name: image, type: string }
  workspaces: [{ name: src }]
  tasks:
    - name: clone
      taskRef: { resolver: cluster, params: [{ name: kind, value: task }, { name: name, value: git-clone }, { name: namespace, value: openshift-pipelines }] }
      params: [{ name: URL, value: $(params.git-url) }]
      workspaces: [{ name: output, workspace: src }]
    - name: build
      runAfter: [clone]
      taskRef: { resolver: cluster, params: [{ name: kind, value: task }, { name: name, value: buildah }, { name: namespace, value: openshift-pipelines }] }
      params: [{ name: IMAGE, value: "$(params.image):sha-$(tasks.clone.results.commit)" }]
      workspaces: [{ name: source, workspace: src }]
    - name: scan
      runAfter: [build]
      taskRef: { name: trivy-scan }
      params: [{ name: image, value: "$(params.image)@$(tasks.build.results.IMAGE_DIGEST)" }]

$ tkn pipeline start build-and-scan -p git-url=https://git.bank.internal/payments/api \
    -p image=quay.bank.internal/payments/payments-api -w name=src,claimName=ci-workspace
PipelineRun started: build-and-scan-run-7k2xq
$ tkn pipelinerun logs build-and-scan-run-7k2xq -f
[clone : clone] + git fetch --depth=1 origin 9f3c2d1
[build : build] STEP 12/12: ENTRYPOINT ["/app/payments-api"]
[build : push] Writing manifest to image destination
[scan : scan] Total: 0 (CRITICAL: 0)

Registry and image hygiene

A bank does not pull from Docker Hub in production. It runs a corporate registry: Red Hat Quay (often bundled with OpenShift Platform Plus), JFrog Artifactory, or Amazon ECR for EKS. Most of the compliance controls actually live there, so know the vocabulary.

• Mirroring and pull-through caches. Quay mirrors upstream repositories on a schedule; Artifactory and ECR offer pull-through cache rules that fetch on first request and serve locally afterwards. You get one chokepoint to scan and something that keeps working when Docker Hub rate-limits you. On OpenShift, an ImageDigestMirrorSet (the 4.13+ replacement for ImageContentSourcePolicy) makes CRI-O redirect pulls to your mirror; it is also how disconnected clusters get their release payload.
• Retention. Quay tag expiration, Artifactory cleanup policies and ECR lifecycle rules ("keep the last 30 sha- tags, expire untagged manifests after 7 days") stop the registry filling with ten thousand CI builds. Exempt semver release tags, and never expire a tag a running environment references.
• Promotion by digest. Promote from the dev registry (or dev repository) to prod by copying the exact digest, not by rebuilding: skopeo copy docker://quay.bank.internal/dev/payments-api@sha256:ab12... docker://quay.bank.internal/prod/payments-api:1.14.2. The bytes that were scanned and signed are the bytes that run; a rebuild would be a different image with no evidence attached.
• Allowed registries at cluster level. Post 22 covered registrySources.allowedRegistries in image.config.openshift.io/cluster, which makes CRI-O refuse to pull from anywhere else. On EKS the same control is a Kyverno or Gatekeeper policy on image fields. Either way, an image from a personal Docker Hub account becomes impossible rather than merely discouraged.
• Rebuild cadence. A CVE in the base image (UBI, distroless) does not fix itself. Rebuild application images weekly at minimum, plus on-demand when Red Hat publishes an errata for the UBI you use, and let the pipeline scan, sign and open the tag-bump PRs. The registry's own scanner (Quay ships Clair; ECR has enhanced scanning via Inspector) rescans stored images continuously so you learn about new CVEs in old images.
• Provenance and SLSA. SLSA (Supply-chain Levels for Software Artifacts) = a framework grading how trustworthy a build is: L1 is "provenance exists", L2 is provenance signed by a hosted build platform, L3 adds a hardened, isolated builder so the provenance cannot be forged. GitHub's actions/attest-build-provenance and Tekton Chains attach these attestations to the image alongside the cosign signature.
• Verify at admission. Signing is pointless if nobody checks. A Kyverno verifyImages rule, the sigstore policy-controller, or Red Hat Advanced Cluster Security (ACS) signature integration rejects any pod whose image lacks a valid signature from your pipeline identity. Newer OpenShift releases also add a native sigstore ClusterImagePolicy; run oc explain clusterimagepolicy to see whether your version has it and whether it is GA.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-bank-images }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: require-cosign-keyless
      match: { any: [{ resources: { kinds: [Pod], namespaces: ["payments-*"] } }] }
      verifyImages:
        - imageReferences: ["quay.bank.internal/prod/*"]
          mutateDigest: true          # rewrite tag to digest so what was verified is what runs
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/bank/payments-api/.github/workflows/build.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor: { url: https://rekor.sigstore.dev }

GitOps principles, and why pull beats push

The OpenGitOps project (a CNCF working group) defines GitOps with four principles; quoting them lands well because they are the reference definition rather than a vendor's:

1. Declarative. Desired state is expressed declaratively (manifests, not scripts).
2. Versioned and immutable. Desired state is stored with immutability, versioning and a complete history. Git is the obvious store but not the only one; an OCI registry can hold manifests too.
3. Pulled automatically. Software agents pull the desired state from the source; nothing pushes it into the cluster.
4. Continuously reconciled. Agents keep observing actual state and converge it to desired state, forever, not once at deploy time.

Principle 3 is the security argument. In a push model CI needs write credentials to every cluster it deploys to, which makes it the most valuable target in the company: compromise Jenkins and you own production. In a pull model the cluster holds a read-only Git credential and the pipeline holds a Git write credential to a branch-protected repo, so the blast radius of a compromised pipeline drops from "run anything in prod" to "open a PR someone has to approve." Principle 4 is the operations argument: if someone hot-fixes a Deployment with oc edit, the agent reverts it, so the repo can never silently disagree with reality.

The two major implementations are Argo CD and Flux, both CNCF graduated. When a JD says "Argo CD or similar," Flux is the "similar," along with Rancher Fleet, Jenkins X and the GitOps modes of Spinnaker or Harness.

For this role the answer is Argo CD, because Red Hat ships and supports it as OpenShift GitOps, but saying "Flux would be my choice for a lightweight per-cluster agent with true Helm semantics" shows you chose rather than defaulted.

Argo CD architecture

Argo CD is a set of cooperating components in one namespace (argocd upstream, openshift-gitops on OpenShift). Name each and say what breaks when it is unhealthy.

• argocd-server: the API server and web UI. The CLI and UI talk to it and it enforces Argo's RBAC. If it is down, deployments still happen; you just cannot see them.
• argocd-repo-server: clones repositories and renders manifests (helm template, kustomize build, or config management plugins), caching the result. Big Helm charts make this pod run out of memory, not the controller.
• argocd-application-controller: the reconciliation loop. For each Application it fetches rendered manifests from the repo-server, compares them with live cluster state, computes sync and health status, and performs syncs. It runs as a StatefulSet so it can be sharded across clusters.
• argocd-redis: cache for rendered manifests and cluster state. Losing it means slow refreshes, not lost data; the source of truth is Git plus the Application CRs.
• argocd-dex-server: the OIDC broker for SSO. On OpenShift it federates to the cluster's OAuth server so users log in with their normal identity and Argo sees their OpenShift groups.
• argocd-notifications-controller: watches Application events and posts to Slack, Teams, email or webhooks.
• argocd-applicationset-controller: expands ApplicationSet CRs into many Applications from generators.
• Argo CD Image Updater (optional, separate project): watches registries for new tags and writes bumps back to Git.

Installing on OpenShift: the OpenShift GitOps operator

On OpenShift you do not helm install Argo CD. You install the Red Hat OpenShift GitOps operator from OperatorHub (Post 23 covers OLM). It watches an ArgoCD custom resource and creates the whole component set from it, so you tune Argo by editing the CR rather than patching Deployments, and it creates a default instance in the openshift-gitops namespace.

$ oc get csv -n openshift-gitops-operator
NAME                                DISPLAY                    VERSION   PHASE
openshift-gitops-operator.v1.15.0   Red Hat OpenShift GitOps   1.15.0    Succeeded
$ oc get argocd -A
NAMESPACE          NAME               AGE
openshift-gitops   openshift-gitops   14d
payments-gitops    payments           3d
$ oc get pods -n openshift-gitops
NAME                                                          READY   STATUS    RESTARTS   AGE
cluster-6f8c9d7b4c-x2k9p                                      1/1     Running   0          14d
openshift-gitops-application-controller-0                     1/1     Running   0          14d
openshift-gitops-applicationset-controller-5d9f6b8c7d-tq4mm   1/1     Running   0          14d
openshift-gitops-dex-server-7c4b5f9d8-8zlxw                   1/1     Running   0          14d
openshift-gitops-redis-6b7d8c9f5-plv2n                        1/1     Running   0          14d
openshift-gitops-repo-server-84f7c6d9b-hn3rk                  1/1     Running   0          14d
openshift-gitops-server-5b8d7f6c9-wq7zd                       1/1     Running   0          14d

Two design decisions come up in interviews. Cluster-scoped versus namespace-scoped instances: the default openshift-gitops instance is cluster-scoped (the operator lists it in its ARGOCD_CLUSTER_CONFIG_NAMESPACES environment variable), so it can manage Namespaces, ClusterRoles, Operators and MachineConfigs. Any other ArgoCD CR is namespace-scoped: it can only deploy into namespaces labelled argocd.argoproj.io/managed-by: <its-namespace>, for which the operator creates the Role and RoleBinding. The bank pattern: the platform team owns the cluster-scoped instance for cluster configuration, and large tenants get a namespace-scoped instance whose UI shows only their apps and which cannot touch anyone else's namespace.

The cluster-admin question: the default instance's controller ServiceAccount gets a deliberately limited ClusterRole, so the first time you point it at cluster configuration you see namespaces is forbidden. Every blog post then says run oc adm policy add-cluster-role-to-user cluster-admin -z openshift-gitops-argocd-application-controller -n openshift-gitops. It works, and in a bank it is the wrong answer, because it makes a Git merge equivalent to cluster-admin. The better answer is a purpose-built ClusterRole covering exactly the kinds the cluster-config repo manages, with that repo protected by CODEOWNERS and required reviews.

The ArgoCD CR configures SSO, RBAC, resource limits and health checks in one place:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata: { name: openshift-gitops, namespace: openshift-gitops }
spec:
  server:
    route: { enabled: true, tls: { termination: reencrypt } }
  sso:
    provider: dex
    dex: { openShiftOAuth: true }          # log in with OpenShift users and groups
  rbac:
    defaultPolicy: role:readonly
    scopes: "[groups]"
    policy: |
      g, platform-admins, role:admin
      p, role:payments-dev, applications, sync, payments/*, allow
      g, payments-developers, role:payments-dev
  repo:
    resources: { limits: { memory: 2Gi }, requests: { cpu: 500m, memory: 512Mi } }
  controller:
    resources: { limits: { memory: 4Gi }, requests: { cpu: "1", memory: 1Gi } }
  resourceHealthChecks:
    - group: operators.coreos.com
      kind: Subscription
      check: |
        hs = {}
        if obj.status ~= nil and obj.status.state == "AtLatestKnown" then
          hs.status = "Healthy"; hs.message = "Operator installed"; return hs
        end
        hs.status = "Progressing"; hs.message = "Waiting for operator install"; return hs

Installing on EKS: Helm

On EKS there is no operator: install the community chart, then let Argo manage its own upgrades by pointing an Application at the same chart.

$ helm repo add argo https://argoproj.github.io/argo-helm
$ helm install argocd argo/argo-cd -n argocd --create-namespace \
    --set server.ingress.enabled=true --set redis-ha.enabled=true --set controller.replicas=2
$ argocd admin initial-password -n argocd
Vx7pQ2mN9kLr4sTb

Logging in and listing apps looks the same on both platforms. With SSO configured, --sso opens a browser; the initial admin password is a bootstrap-only path to disable once SSO works (on OpenShift the bootstrap secret is openshift-gitops-cluster).

$ argocd login $(oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}') --sso --grpc-web
Opening browser for authentication
Authentication successful
'oncall@bank.internal' logged in successfully
$ argocd app list
NAME                              CLUSTER                         NAMESPACE        PROJECT    STATUS     HEALTH       SYNCPOLICY  REPO                                          PATH              TARGET
openshift-gitops/cluster-config   https://kubernetes.default.svc  openshift-config platform   Synced     Healthy      Auto-Prune  https://git.bank.internal/platform/cluster    clusters/prod-ca  main
openshift-gitops/payments-dev     https://kubernetes.default.svc  payments-dev     payments   Synced     Healthy      Auto-Prune  https://git.bank.internal/payments/config     envs/dev          main
openshift-gitops/payments-prod    https://kubernetes.default.svc  payments-prod    payments   OutOfSync  Progressing  Manual      https://git.bank.internal/payments/config     envs/prod         main

Read the last line as an interviewer would: prod is Manual (a human presses sync, or a sync window allows it), OutOfSync because a tag bump was merged and not yet applied, and Progressing because a previous rollout is still bringing pods up.

The Application CRD, field by field

An Application = one Argo CD unit of deployment: "take the manifests at this source, render them, and keep them applied to this destination, under the rules of this project." You will write dozens, so learn the spec rather than copying it.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-prod
  namespace: openshift-gitops              # must be the namespace Argo watches
  finalizers: [resources-finalizer.argocd.argoproj.io]   # deleting the App deletes its resources
spec:
  project: payments                        # AppProject that constrains sources/destinations
  source:
    repoURL: https://git.bank.internal/payments/config.git
    targetRevision: main                   # branch, tag, or commit SHA; a chart version for Helm repos
    path: envs/prod                        # directory in the repo (use `chart:` instead for a Helm repo)
    kustomize: { }                         # or helm: { valueFiles: [values-prod.yaml] }
  destination:
    server: https://kubernetes.default.svc # or name: prod-ca-central-1 for a remote cluster
    namespace: payments-prod
  syncPolicy:
    automated:
      prune: true                          # delete live resources that vanished from Git
      selfHeal: true                       # revert manual changes to the cluster
      allowEmpty: false                    # refuse to sync a path that renders to nothing
    syncOptions:
      - CreateNamespace=true               # create the destination namespace if missing
      - ServerSideApply=true               # use SSA; needed for huge CRDs and for field-manager-aware diffs
      - ApplyOutOfSyncOnly=true            # only apply resources that differ (faster on big apps)
      - PruneLast=true                     # prune after the sync succeeds, not before
      - RespectIgnoreDifferences=true      # do not overwrite ignored fields on sync
    retry:
      limit: 5
      backoff: { duration: 10s, factor: 2, maxDuration: 3m }
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas]       # HPA owns replicas; do not fight it
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions: [.webhooks[].clientConfig.caBundle]   # cert-manager injects this
  revisionHistoryLimit: 10
  info:
    - { name: owner, value: payments-platform@bank.internal }
    - { name: runbook, value: https://wiki.bank.internal/payments/runbook }

The fields interviewers ask about:

• prune makes deleting a file in Git delete the object in the cluster. Without it, removed manifests are reported as extraneous but left running, which quietly breaks "Git is the truth." Replace=true is the rarely-used sibling that does kubectl replace instead of apply, for resources that reject in-place updates (immutable Job specs).
• selfHeal makes oc edit pointless: Argo notices the live object differs and re-applies Git within seconds.
• ignoreDifferences exists because some fields are legitimately owned by something else: the HPA sets replicas, cert-manager or the OpenShift service CA injects caBundle, admission webhooks add defaults. Without it these apps show OutOfSync forever, and with selfHeal they fight the other controller in a loop.
• The finalizer decides what deleting the Application does: with it, Argo cascades and deletes every managed resource; without it, the Application disappears and the workloads stay. Decide deliberately, especially for anything holding a PVC.

Sync status and health status are two different questions

Sync status answers "does live state match Git?": Synced, OutOfSync, or Unknown (Argo could not compare, usually a repo problem). Health status answers "is the thing that is running actually working?": Healthy (Deployment has all replicas ready, Service has endpoints), Progressing (rollout in flight), Degraded (rollout failed, pods crashlooping, Job failed), Suspended (paused rollout or suspended CronJob), Missing (in Git but not live), Unknown. An app can be Synced and Degraded (Git is applied, the pods crash) or OutOfSync and Healthy (a new tag was merged, sync is manual); naming that matrix is a five-second answer that shows real usage.

Argo ships health logic for core kinds and many popular CRDs, but a CRD it does not know is reported Healthy immediately because there is nothing to check. That is dangerous for OpenShift Routes, Operator Subscriptions, Argo Rollouts, or anything where "exists" is not "working." Add a custom health check in Lua, either in argocd-cm under resource.customizations.health.<group>_<Kind> or, on OpenShift, under spec.resourceHealthChecks in the ArgoCD CR as shown earlier. Check the argo-cd repo's resource_customizations directory first: OpenShift Routes and several OLM kinds already have built-in checks in recent versions.

$ argocd app get payments-prod
Name:               openshift-gitops/payments-prod
Project:            payments
Server:             https://kubernetes.default.svc
Namespace:          payments-prod
Repo:               https://git.bank.internal/payments/config.git
Target:             main
Path:               envs/prod
SyncWindow:         Sync Denied
Sync Policy:        Manual
Sync Status:        OutOfSync from main (4d1e7a0)
Health Status:      Healthy

GROUP  KIND        NAMESPACE      NAME           STATUS     HEALTH   HOOK  MESSAGE
       ConfigMap   payments-prod  payments-cfg   Synced
apps   Deployment  payments-prod  payments-api   OutOfSync  Healthy        deployment.apps/payments-api configured
       Service     payments-prod  payments-api   Synced     Healthy

SyncWindow: Sync Denied is the line a bank cares about: outside the maintenance window, even a manual sync is refused until the window opens.

Ordering and lifecycle: sync waves and hooks

By default Argo applies everything in one pass, ordered only by kind (Namespaces and CRDs before things that need them, then by name). Real apps need more control, and Argo gives you two tools that interviewers love to ask you to distinguish.

Sync waves order resources within a sync. The annotation argocd.argoproj.io/sync-wave: "N" (default 0, negatives allowed) groups resources; Argo applies wave -1, waits until everything in it is Healthy, then applies wave 0, and so on. Waves are for dependencies: the Namespace and CRDs in wave -1, the operator Subscription in wave 0, the operator's custom resource in wave 1, the app that uses it in wave 2.

Resource hooks run extra resources at phases of the sync. The annotation argocd.argoproj.io/hook takes PreSync (before anything is applied: database schema migrations), Sync (alongside the normal resources), PostSync (after everything is Healthy: smoke tests, cache warmers, notifications), SyncFail (cleanup or paging when the sync fails), and PostDelete (after the app is deleted). Hooks are typically Jobs; if a PreSync hook Job fails, the sync stops and nothing else is applied, which is exactly what you want for a failed migration. argocd.argoproj.io/hook-delete-policy controls cleanup: HookSucceeded deletes the Job after success (keeping failed ones for debugging), HookFailed deletes on failure, and BeforeHookCreation (the default) deletes the previous run's hook right before creating a new one, which lets a Job with a fixed name run again on the next sync.

apiVersion: batch/v1
kind: Job
metadata:
  name: payments-db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-5"
spec:
  backoffLimit: 0                # a failed migration must fail the sync, not retry silently
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: payments-migrator
      containers:
        - name: migrate
          image: quay.bank.internal/prod/payments-api@sha256:ab12c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2
          args: ["migrate", "up"]
          envFrom: [{ secretRef: { name: payments-db } }]

Two ordering problems show up in every real OpenShift GitOps repo. First, CRDs and their instances in the same sync: Argo dry-runs every manifest before applying, and the dry-run of a custom resource fails with "the server could not find the requested resource" if its CRD is not installed yet. Put the CRD in an earlier wave and add the sync option SkipDryRunOnMissingResource=true on the CR (as an annotation argocd.argoproj.io/sync-options). Second, the operator-then-CR problem: you put the Subscription in wave 0 and the operator's CR in wave 1, but wave 0 completes instantly because Argo considers a Subscription Healthy the moment it exists, long before OLM has installed the CSV and registered the CRD. The fix is the custom Subscription health check from the ArgoCD CR earlier (Healthy only when status.state == AtLatestKnown), so wave 0 genuinely waits. Explaining this pain point tells an interviewer you have deployed operators through Argo, not just read about waves.

Scaling GitOps: App of Apps, ApplicationSets, projects, RBAC and windows

App of Apps

Hand-creating Applications in the UI does not scale and is not GitOps. The App of Apps pattern = one root Application whose source path contains nothing but other Application manifests; bootstrap a new cluster with oc apply -f root-app.yaml and Argo creates every child from Git. It is simple and still common for the platform's own cluster-config layer.

ApplicationSet

The ApplicationSet CRD is the more powerful version: a template for an Application plus one or more generators that produce parameter sets, and the controller stamps out one Application per parameter set.

• List: a literal list of parameters (three environments, five clusters).
• Cluster: one Application per cluster registered in Argo, optionally filtered by cluster secret labels (env: prod).
• Git directory: one Application per subdirectory of a repo path. Git file: one per JSON/YAML file matched by a glob, with the file's contents as parameters; this is how a single file onboards a tenant.
• Matrix: the cross product of two generators (every app in every cluster). Merge combines generators on a key.
• Pull request: one Application per open PR, which gives you ephemeral preview environments that disappear when the PR closes.
• SCM provider: discovers repositories in a GitHub or GitLab org. Cluster decision resource: reads cluster lists from another controller, which is how RHACM Placements feed Argo.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: tenants, namespace: openshift-gitops }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://git.bank.internal/platform/tenants.git
              revision: main
              files: [{ path: "tenants/*/tenant.yaml" }]     # one file per team
          - clusters:
              selector: { matchLabels: { tier: workload } } # every registered workload cluster
  template:
    metadata:
      name: "{{ .team }}-{{ .name }}"                         # e.g. payments-prod-ca-central-1
      labels: { team: "{{ .team }}" }
    spec:
      project: "{{ .team }}"
      source:
        repoURL: "{{ .configRepo }}"
        targetRevision: main
        path: "envs/{{ index .metadata.labels \"env\" }}"
      destination: { server: "{{ .server }}", namespace: "{{ .team }}-{{ index .metadata.labels \"env\" }}" }
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

With tenants/payments/tenant.yaml containing team: payments and configRepo: https://git.bank.internal/payments/config.git, one merged file yields an Application in every workload cluster. Combined with the namespace, quota, RBAC and NetworkPolicy templates from Post 26, "onboard a team" is a pull request reviewed by the platform team, with zero clicks.

Multi-cluster: hub-and-spoke or Argo per cluster

Argo registers a remote cluster as a Secret labelled argocd.argoproj.io/secret-type: cluster holding the API URL and credentials; argocd cluster add <context> creates it. Hub-and-spoke (one Argo managing many clusters) gives one UI, one RBAC model and fleet-wide ApplicationSets, but the hub holds every cluster's credentials and is a single point of failure. One Argo per cluster has no cross-cluster credentials and survives losing the hub, at the cost of N UIs and N upgrades. Banks often run per-cluster Argo for cluster configuration plus a hub for application fleets. With Red Hat Advanced Cluster Management (RHACM), a GitOpsCluster resource registers every ManagedCluster matched by a Placement into Argo automatically.

AppProject: the tenant boundary

An AppProject = the guardrail around a group of Applications: which repos they may pull from, which clusters and namespaces they may deploy to, which cluster-scoped kinds they may create, plus project-level roles and sync windows. Every Application belongs to one; the built-in default project allows everything and should never be used by a tenant.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: payments, namespace: openshift-gitops }
spec:
  description: Payments platform team
  sourceRepos: ["https://git.bank.internal/payments/*"]
  destinations:
    - { server: https://kubernetes.default.svc, namespace: "payments-*" }
  clusterResourceWhitelist: []            # tenants may not create Namespaces, ClusterRoles, CRDs
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }  # platform owns quotas and limits
    - { group: "", kind: LimitRange }
    - { group: networking.k8s.io, kind: NetworkPolicy }
  roles:
    - name: release-manager
      policies: ["p, proj:payments:release-manager, applications, sync, payments/payments-prod, allow"]
      groups: [payments-release-managers]
  syncWindows:
    - kind: allow
      schedule: "0 22 * * 2,4"            # Tue/Thu 22:00, the bank's change window
      duration: 3h
      applications: ["payments-prod"]
      manualSync: true
    - kind: deny
      schedule: "0 0 * * *"
      duration: 24h
      applications: ["payments-prod"]
      timeZone: America/Toronto

Sync windows are the feature that makes change managers smile: a deny window blocks all syncs of prod apps by default, an allow window opens them only during the approved change slot, and manualSync: true permits a human-initiated sync inside an allow window while automated syncs stay blocked. This is the technical enforcement of a change calendar; connect it to the CAB (Change Advisory Board) process rather than treating it as a curiosity.

RBAC and SSO

Argo's RBAC is a Casbin policy.csv: lines of the form p, <subject>, <resource>, <action>, <object>, allow plus g, <group>, <role> mappings. Resources include applications, projects, repositories, clusters, logs and exec; actions include get, sync, override, delete, action/*. Groups arrive from SSO through dex; on OpenShift they are your OpenShift groups, which typically mirror Active Directory groups, so "who can sync prod" is answered by the same directory the rest of the bank uses. Set defaultPolicy: role:readonly so a logged-in stranger sees but cannot touch, and never hand out exec in prod.

Notifications

The notifications controller reads a ConfigMap of services (Slack token, Teams webhook), triggers (conditions such as app.status.operationState.phase in ['Error', 'Failed']) and templates; apps subscribe with an annotation like notifications.argoproj.io/subscribe.on-sync-failed.slack: payments-alerts. In a bank the value is the evidence trail: every prod sync posts who triggered it, the commit and the result into a retained channel.

Disaster recovery

The GitOps DR pitch in one sentence: rebuild the cluster from Terraform (Post 29), install Argo, apply the root Application, and the cluster reassembles itself from Git. What is not in Git needs its own backup: Argo's cluster and repo credential Secrets (argocd admin export), application data on PVCs, and secrets held in Vault or a cloud secret manager, which live outside the cluster anyway. Interviewers like hearing "our DR test is deleting a non-prod cluster and timing the rebuild."

Helm and Kustomize with Argo CD

Argo does not care how manifests are produced, only that the repo-server can render them. The three common shapes, each with one operational trap:

• Helm. source.helm.valueFiles: [values.yaml, values-prod.yaml] layers per-environment values and helm.parameters overrides single keys. Argo runs helm template, so there is no Helm release in the cluster (helm list shows nothing) and Helm hooks become Argo hooks. To combine a vendor chart from a Helm repository with values in your Git repo, use multiple sources: a chart: source plus a Git source with ref: values, then valueFiles: [$values/envs/prod/values.yaml], so you never fork the chart.
• Kustomize. A base/ plus overlays/dev, overlays/test, overlays/prod with patches for replicas, resources and image digests. Argo detects a kustomization.yaml automatically; source.kustomize.images and namePrefix can override from the Application. Most bank config repos take this shape because there is no templating language to review, just YAML diffs.
• Helm inside Kustomize. A helmCharts: block in kustomization.yaml inflates a chart and lets you patch its output. Argo needs kustomize.buildOptions: --enable-helm in argocd-cm (or spec.kustomizeBuildOptions in the ArgoCD CR) or the render fails with a confusing "must specify --enable-helm" error.

The rendered manifests pattern is worth knowing by name: CI renders the final plain YAML per environment and commits it, and Argo points at the rendered output instead of templating at sync time. A PR then shows the exact diff that will hit the cluster, the repo-server does no heavy work, and an auditor can read prod state without tooling, at the cost of a CI step and repo churn. Teams under strict change review increasingly choose it.

Day-to-day drift handling from the CLI:

$ argocd app diff payments-prod
===== apps/Deployment payments-prod/payments-api ======
41c41
<         image: quay.bank.internal/prod/payments-api@sha256:ab12c3d4...
---
>         image: quay.bank.internal/prod/payments-api@sha256:77e0f9a1...
$ argocd app sync payments-prod --prune
TIMESTAMP                  GROUP  KIND        NAMESPACE      NAME          STATUS   HEALTH       HOOK  MESSAGE
2026-03-03T22:04:11-05:00  apps   Deployment  payments-prod  payments-api  Synced   Progressing        deployment.apps/payments-api configured
Operation:          Sync
Phase:              Succeeded
Message:            successfully synced (all tasks run)
$ argocd app history payments-prod
ID  DATE                           REVISION
17  2026-02-25 22:03:40 -0500 EST  main (3b9d2c1)
18  2026-03-03 22:04:09 -0500 EST  main (77e0f9a)
$ argocd app rollback payments-prod 17
FATA[0000] rpc error: code = FailedPrecondition desc = rollback cannot be initiated when auto-sync is enabled

That last error is deliberate. argocd app rollback re-applies a previously synced revision, but with automated sync on, Argo would immediately re-sync to Git's HEAD, so it refuses. The GitOps rollback is git revert of the tag-bump commit and a merge: fast, auditable, and Git and the cluster keep agreeing. Use argocd app rollback only on manually-synced apps as a stopgap while the revert PR merges.

Secrets in GitOps

The one rule: a Kubernetes Secret manifest in Git is plaintext with base64 makeup. Never commit one, even in a private repo, because Git history is forever and repos get cloned to laptops. Four patterns solve this; a bank usually runs one primary and tolerates one secondary.

The realistic bank answer is ESO backed by Vault (or AWS Secrets Manager on EKS), with Kubernetes auth so no static Vault tokens exist, and etcd encryption enabled on the cluster (Post 31). The manifest that goes in Git:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: payments-db, namespace: payments-prod }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: bank-vault, kind: ClusterSecretStore }
  target:
    name: payments-db                      # the Kubernetes Secret ESO will create
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@pg-prod.bank.internal:5432/payments"
  data:
    - secretKey: username
      remoteRef: { key: secret/payments/prod/db, property: username }
    - secretKey: password
      remoteRef: { key: secret/payments/prod/db, property: password }

$ oc get externalsecret -n payments-prod
NAME          STORE        REFRESH INTERVAL   STATUS         READY
payments-db   bank-vault   1h                 SecretSynced   True
$ oc get secret payments-db -n payments-prod -o jsonpath='{.metadata.ownerReferences[0].kind}'
ExternalSecret

If STATUS shows SecretSyncedError, oc describe externalsecret gives the vault's error verbatim: a wrong path, a policy that does not grant read, or a Kubernetes auth role that does not trust this namespace's ServiceAccount.

Progressive delivery

A plain Deployment rollout is all-or-nothing per pod, gated only by readiness. Progressive delivery shifts traffic gradually and gates each step on real metrics. Argo Rollouts is the sister project: a Rollout CRD that replaces Deployment and adds blue-green (a full second stack behind a preview Service, promoted by switching the active Service) and canary (a small percentage of traffic to the new version, increasing in steps). Steps can include an AnalysisTemplate that queries Prometheus (Post 24) and automatically aborts and rolls back when the success condition fails. Traffic shaping needs a router Rollouts can control: Istio or OpenShift Service Mesh, NGINX, the AWS ALB controller, Gateway API, or the OpenShift Route traffic-router plugin.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: payments-api, namespace: payments-prod }
spec:
  replicas: 10
  selector: { matchLabels: { app: payments-api } }
  template: { }        # identical to a Deployment pod template
  strategy:
    canary:
      canaryService: payments-api-canary
      stableService: payments-api
      steps:
        - setWeight: 10
        - pause: { duration: 10m }
        - analysis:
            templates: [{ templateName: error-rate }]
            args: [{ name: service, value: payments-api-canary }]
        - setWeight: 50
        - pause: { }    # indefinite: a human runs `kubectl argo rollouts promote`
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: error-rate, namespace: payments-prod }
spec:
  args: [{ name: service }]
  metrics:
    - name: http-5xx-ratio
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: https://thanos-querier.openshift-monitoring.svc:9091
          query: |
            sum(rate(http_requests_total{service="{{args.service}}",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{service="{{args.service}}"}[2m]))

Argo CD understands Rollout health (Progressing while stepping, Suspended at an indefinite pause, Degraded when analysis aborts), so the Application status tells the story without a second UI.

Argo CD Image Updater closes the loop the other way: it polls the registry, sees a new tag matching a rule (argocd-image-updater.argoproj.io/image-list with a semver or regex constraint) and writes the bump back to Git or to the Application spec. It is convenient for dev environments. For prod a bank prefers the CI-opens-a-PR flow, because the PR is where the approval, the change ticket link and the reviewer's name live; a bot's automatic commit has none of that. Say "Image Updater in dev, PR-based bumps in prod" and you have the balanced answer.

Feature flags are the other half of progressive delivery: deploy dark and enable per-segment at runtime (OpenFeature, LaunchDarkly, Unleash), so the deploy and the release are separate decisions.

Promotion and environments

The layout decision that shapes everything else: branches per environment (a dev, test and prod branch, promote by merging) or directories per environment (one main branch with envs/dev, envs/test, envs/prod). Directories win in practice, and you should be able to say why: merging between environment branches carries every environment-specific difference along with it, so you end up cherry-picking and the branches drift into permanently divergent snowflakes, while a directory layout with a Kustomize base makes "what differs between test and prod" a readable diff of two overlay folders, and one commit can touch several environments atomically when it should.

payments-config/
├── base/                     # Deployment, Service, Route, HPA, ExternalSecret
│   └── kustomization.yaml
├── envs/
│   ├── dev/kustomization.yaml    # images: [{name: payments-api, digest: sha256:77e0...}], replicas 1
│   ├── test/kustomization.yaml   # same digest after promotion PR, replicas 2, test Vault path
│   └── prod/kustomization.yaml   # promoted digest, replicas 10, PDB, sync-wave annotations
├── argocd/                   # one Application per env (or one ApplicationSet)
└── CODEOWNERS                # envs/prod/ @bank/payments-release-managers @bank/platform-change

Promotion is then a PR that copies the digest from envs/test to envs/prod. Branch protection requires reviews from the CODEOWNERS of envs/prod, the PR description links the change ticket, and the merge commit is the evidence the CAB wants: what changed, who requested it, who approved it, when it landed. Many teams automate the copy with a bot (Kargo is the emerging purpose-built promotion tool from the Argo maintainers) but keep the human approval on the prod PR. A release train batches promotions into the scheduled sync window instead of one PR per change.

The emergency path still goes through Git: an expedited PR with one named approver and an emergency label, merged and synced in a break-glass window, is faster than any change board and leaves the same audit trail. If someone truly must oc edit production mid-incident, the runbook says disable selfHeal on that Application first (or Argo undoes the fix), record the change in the incident ticket, and commit the equivalent to Git before closing the incident. Break-glass with an audit trail is acceptable in a bank; silent drift is not.

Finally, separate platform config from app config. Cluster add-ons, operators, MachineConfigs, SCCs, policies, quotas and the AppProjects themselves live in a cluster-config repo owned by the platform team and deployed by the cluster-scoped Argo instance; application repos are owned by tenants and constrained by their AppProject. Different owners, different reviewers, different blast radius.

Troubleshooting Argo CD

State the method first, then the specific causes, exactly as in Post 25: read the Application's status and conditions → argocd app diff to see what Argo thinks differs → the operation state for the last sync error → repo-server logs for rendering problems, controller logs for comparison and apply problems → the target cluster's own events.

$ oc get application -n openshift-gitops
NAME             SYNC STATUS   HEALTH STATUS
cluster-config   Synced        Healthy
payments-dev     Unknown       Healthy
payments-prod    OutOfSync     Degraded
$ oc get application payments-dev -n openshift-gitops -o jsonpath='{.status.conditions}' | jq
[{"type":"ComparisonError","message":"rpc error: code = Unknown desc = error fetching chart: failed to fetch https://charts.bank.internal/... : 401 Unauthorized"}]
$ oc logs statefulset/openshift-gitops-application-controller -n openshift-gitops --since=10m | grep payments-prod
$ oc logs deploy/openshift-gitops-repo-server -n openshift-gitops --since=10m | grep -i "error\|oom"

• OutOfSync forever, and syncing does not fix it. Something else mutates the object after Argo applies it: an HPA changing replicas, a mutating webhook (Istio sidecar injection, the OpenShift service CA adding caBundle, a policy engine adding labels), or API-server defaulting of fields you did not set. argocd app diff shows the exact field; then add an ignoreDifferences entry, enable ServerSideApply=true so Argo only owns the fields it sets, or set defaulted fields explicitly in Git.
• ComparisonError. Argo could not render or fetch. The condition message says which: repository authentication (an expired token in the repo credential Secret, an unknown SSH host key), a Helm dependency that needs helm dependency build against a chart repo Argo has no credential for, a Kustomize build that needs --enable-helm, or a path that does not exist at that revision. argocd repo list shows connection status per repo.
• Degraded on a CRD that is actually fine. A custom health check with a bug, or a Lua check that expects a status field the CRD version no longer sets. argocd app get shows the per-resource HEALTH and MESSAGE columns; the message comes straight from your Lua.
• Permission denied. "application repo ... is not permitted in project" or "resource Namespace is not permitted in project" means the AppProject's sourceRepos, destinations or clusterResourceWhitelist is blocking it, by design. "namespaces is forbidden: User system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller cannot create resource" means the Argo ServiceAccount itself lacks RBAC on the target cluster; for a namespace-scoped instance, check the argocd.argoproj.io/managed-by label on the namespace.
• Sync hangs. A hook Job that never completes (no activeDeadlineSeconds, waiting on something that will never come) holds the whole sync: argocd app terminate-op, fix the Job, re-sync. A wave that never becomes Healthy has the same effect, usually a resource with no health check that you assumed had one, or the opposite.
• Pruning deleted something it should not have. A manifest was removed from Git or a rename changed a resource's identity, and prune: true did its job. Protect precious objects (PVCs, namespaces, the cluster-config app itself) with argocd.argoproj.io/sync-options: Prune=false, use PruneLast=true so new resources exist before old ones go, and for cluster-scoped instances consider prune: false plus alerts on extraneous resources.
• repo-server OOMKilled on big Helm charts. Rendering a large chart, or many apps at once, spikes memory; oc describe pod shows OOMKilled in the last state. Raise spec.repo.resources in the ArgoCD CR, add repo-server replicas, lower reposerver.parallelism.limit, or pre-render the worst offenders.

Measuring and improving pipelines

"Enhance CI/CD pipelines" in a JD means you can measure them. The DORA metrics are the shared vocabulary: deployment frequency (how often prod changes), lead time for changes (commit to running in prod), change failure rate (share of deploys causing an incident or rollback) and mean time to restore. GitOps gives you the raw data for free: Argo's sync history and notifications timestamp every prod deploy and Git timestamps every commit, so lead time and frequency are a query, not a survey, and change failure rate is the count of revert PRs over the count of promotion PRs.

Improvements that move those numbers, in the order you would try them:

• Reusable pipeline templates. GitHub reusable workflows (on: workflow_call, invoked with uses: bank/platform-workflows/.github/workflows/build-image.yml@v3), Jenkins shared libraries (vars/buildImage.groovy), and a curated Tekton Pipeline consumed through the cluster or git resolver. One tested, versioned implementation of build-scan-sign-bump owned by the platform team is the "reusable deployment pattern" the JD names; when Trivy changes a flag, you fix one file, not sixty.
• Caching. Layer cache to the registry (cache-to: type=registry,mode=max in BuildKit; buildah's --layers with a cache repo), dependency caches (actions/cache, a PVC workspace in Tekton), and Dockerfile instructions ordered so dependency layers come before source layers.
• Parallelism. Matrix builds per architecture or service; Tekton tasks without runAfter run concurrently; test sharding.
• Flaky test policy. A flaky test retried until green trains everyone to ignore red. Quarantine flaky tests into a non-blocking suite, track them as tickets with an owner and an expiry, and cap automatic retries at one.
• Ephemeral preview environments. The ApplicationSet pull-request generator deploys every open PR to its own namespace with a Route or Ingress named after the PR number and tears it down on close. Reviewers test the real thing, and the shared "dev" environment stops being a queue.

# .github/workflows/build.yml in an app repo: the whole pipeline is one call
jobs:
  image:
    uses: bank/platform-workflows/.github/workflows/build-scan-sign.yml@v3
    with: { image: payments-api, config-repo: bank/payments-config, env: dev }
    secrets: inherit
    permissions: { id-token: write, contents: read }

Likely interview questions

Next up: Security, Compliance and Hardening for a Regulated Platform, where the signing, scanning and secrets controls from this post become the audit-ready evidence a bank's security team actually asks for.