Onboarding Application Teams: Projects, Quotas and Golden Paths

A cluster that only the platform team knows how to deploy to is a science project, not a platform. A bank runs hundreds of application teams, and every one of them has to land on OpenShift the same way: through a request that leaves an audit trail, into a project that already carries the right guardrails, using a deployment pattern that passes every policy on the first try. This post walks the whole path from intake form to first deploy. After it you will be able to describe the onboarding workflow end to end, name the exact objects the platform stamps into every project, defend the defaults with numbers, and explain how a "golden path" lets a team ship without ever filing a ticket that says "please give me root".

Onboarding is the platform's product

The JD lines "support onboarding of application teams" and "define platform operating models and governance controls" describe one job. Tenant = an application team that consumes the platform without administering it. Operating model = the written agreement about who does what: the platform team owns the cluster, the defaults and the paved road; the tenant owns its application, its manifests and its on-call. Golden path (also "paved road") = the supported, pre-approved way to build, package and deploy an application, with templates and automation, so that following it is easier than not following it.

Three properties matter in a regulated shop. Repeatable: the hundredth team gets exactly what the first team got, because a machine stamps it out. Auditable: every project, role binding and quota traces back to a ticket, a pull request and an approver, because an auditor will ask. Self-service: once a team is onboarded it deploys, scales and reads its own logs without paging you. If onboarding is a spreadsheet plus a senior engineer's afternoon, none of the three holds, and the platform team becomes the bottleneck for the whole bank.

The onboarding workflow, end to end

Here is the sequence a mature bank platform follows. Each step produces an artifact you can point to later.

1. Intake request. A ServiceNow or Jira catalog item, never an email. Fields: application name, CMDB (configuration management database) application id, target environments (dev, uat, prod), technical owner and manager, Active Directory group names for admins, developers and read-only users, data classification (public, internal, confidential, restricted), resource tier (t-shirt size), storage needs (RWO or RWX, how many GB), external connectivity (which vendor APIs, which internal systems, inbound from where), and whether the workload is online, batch or both. The CMDB id is mandatory because alerts, showback and audit all key off it.
2. Approvals. The application owner's manager approves the spend, security approves the classification and the connectivity, the platform team sanity-checks the resource ask. The approvals become the change record.
3. Project creation via GitOps. A platform engineer (or the form itself, through automation) opens a pull request to the namespace-as-code repository, adding one file per project. A second platform engineer reviews. On merge, Argo CD renders the Namespace, RoleBindings, ResourceQuota, LimitRange and NetworkPolicies. The PR is the audit trail.
4. RBAC from AD groups. The team's AD groups (synced into OpenShift as Group objects) are bound to admin, edit or view inside the project only.
5. Quotas and limits. The tier from the form maps to a ResourceQuota and a LimitRange.
6. Network baseline. Default-deny ingress plus the allow rules every app needs: router, monitoring, same namespace.
7. Egress and firewall. If the app calls out, an EgressIP gives the corporate firewall a stable source address, an EgressFirewall restricts destinations, and a firewall change request is raised against the perimeter.
8. Registry access. A robot account in Quay or Artifactory with push rights for CI, and a pull secret linked to the project's service accounts.
9. Secrets integration. A Vault path (or AWS Secrets Manager prefix) and a Kubernetes auth role bound to the project's service account, so the External Secrets Operator can sync secrets without anyone pasting them into Git.
10. CI/CD hooks. A pipeline from the platform's template library and an Argo CD AppProject that restricts the team to its own namespaces and repositories.
11. Monitoring and logging visibility. User workload monitoring for the project, a Grafana folder, log access in the console, alert routing to the team's channel.
12. Handover. A one-page document listing project names, groups, quota tier, links to Argo CD, Grafana and the runbook template, and how to request changes. The team deploys the sample application from the starter chart. The clock from ticket creation to that first green deploy is your time-to-first-deploy metric.

ServiceNow intake ──> approvals (owner, security, platform)
        │
        ▼
tenants repo PR (namespace-as-code) ──review──> merge
        │
        ▼
Argo CD ApplicationSet renders, per project:
  Namespace + labels   RoleBindings (AD groups)   ResourceQuota
  LimitRange           NetworkPolicies            EgressFirewall
        │
        ▼
Vault role + registry robot + pipeline template + AppProject
        │
        ▼
UWM / Grafana folder / log access / alert route ──> handover ──> first deploy

Project creation the OpenShift way

In Kubernetes, creating a Namespace requires create on namespaces, which in practice means cluster-admin. OpenShift adds the Project API (project.openshift.io): a Project is a Namespace plus display-name, description and requester annotations, and a ProjectRequest is a special object that ordinary users may create even though they cannot create namespaces. When someone runs oc new-project, the API server handles the ProjectRequest by instantiating the project request template, so whatever objects that template contains appear in every new project. That template is your first governance control.

Customizing the project request template

$ oc adm create-bootstrap-project-template -o yaml > project-request.yaml
$ vi project-request.yaml        # add quota, limits, network policies
$ oc create -f project-request.yaml -n openshift-config
template.template.openshift.io/project-request created
$ oc patch project.config.openshift.io/cluster --type merge \
    -p '{"spec":{"projectRequestTemplate":{"name":"project-request"}}}'
project.config.openshift.io/cluster patched
$ oc get pods -n openshift-apiserver -w    # apiserver pods roll to pick it up

The bootstrap output contains only a Project and a RoleBinding that makes the requester a project admin. Here is the full template with the platform defaults added. The quota numbers are the "small" tier; the template is a floor, and the GitOps path overrides it per tenant.

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: project-request
objects:
- apiVersion: project.openshift.io/v1
  kind: Project
  metadata:
    name: ${PROJECT_NAME}
    annotations:
      openshift.io/description: ${PROJECT_DESCRIPTION}
      openshift.io/display-name: ${PROJECT_DISPLAYNAME}
      openshift.io/requester: ${PROJECT_REQUESTING_USER}
    labels:
      platform.bank.example/managed-by: project-template
- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    name: admin
    namespace: ${PROJECT_NAME}
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: admin
  subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ${PROJECT_ADMIN_USER}
- apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: default-quota
    namespace: ${PROJECT_NAME}
  spec:
    hard:
      requests.cpu: "2"
      requests.memory: 4Gi
      limits.cpu: "4"
      limits.memory: 8Gi
      pods: "20"
      persistentvolumeclaims: "5"
      requests.storage: 50Gi

- apiVersion: v1
  kind: LimitRange
  metadata:
    name: default-limits
    namespace: ${PROJECT_NAME}
  spec:
    limits:
    - type: Container
      defaultRequest: {cpu: 100m, memory: 128Mi}
      default: {cpu: 500m, memory: 512Mi}
      max: {cpu: "2", memory: 4Gi}
- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: default-deny-ingress
    namespace: ${PROJECT_NAME}
  spec:
    podSelector: {}
    policyTypes: [Ingress]
- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-from-openshift-ingress
    namespace: ${PROJECT_NAME}
  spec:
    podSelector: {}
    ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            network.openshift.io/policy-group: ingress

- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-from-openshift-monitoring
    namespace: ${PROJECT_NAME}
  spec:
    podSelector: {}
    ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            network.openshift.io/policy-group: monitoring
- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-same-namespace
    namespace: ${PROJECT_NAME}
  spec:
    podSelector: {}
    ingress:
    - from:
      - podSelector: {}
parameters:
- name: PROJECT_NAME
- name: PROJECT_DISPLAYNAME
- name: PROJECT_DESCRIPTION
- name: PROJECT_ADMIN_USER
- name: PROJECT_REQUESTING_USER

Disabling self-provisioning

By default every authenticated user can run oc new-project, because the self-provisioners ClusterRoleBinding grants the self-provisioner role to the group system:authenticated:oauth. A bank removes that, so that the only way to get a project is the intake process. Two patches: empty the subjects, and tell the cluster not to reconcile the binding back to its default.

$ oc describe clusterrolebinding.rbac self-provisioners | grep -A3 Subjects
Subjects:
  Kind   Name                        Namespace
  ----   ----                        ---------
  Group  system:authenticated:oauth
$ oc patch clusterrolebinding.rbac self-provisioners -p '{"subjects": null}'
$ oc patch clusterrolebinding.rbac self-provisioners \
    -p '{"metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"false"}}}'
$ oc patch project.config.openshift.io/cluster --type merge -p \
  '{"spec":{"projectRequestMessage":"Projects are provisioned through the Platform Onboarding catalog item in ServiceNow."}}'
$ oc new-project scratch          # as a normal user, afterwards
Error from server (Forbidden): Projects are provisioned through the Platform Onboarding catalog item in ServiceNow.

Naming and metadata conventions

Name projects <app>-<env>: payments-notifications-dev, payments-notifications-prod. One app per project keeps RBAC, quota and network policy boundaries aligned with team ownership; one environment per project stops a dev deploy from consuming prod quota. Labels carry the facts every downstream system needs: platform.bank.example/team, cost-center, data-classification, tier and env as labels (selectable by policies, ClusterResourceQuota and showback queries), with the CMDB id, owner mailbox and requester as annotations (informational, not selectable). A Kyverno policy that says "restricted-data namespaces may only pull from the internal registry" is only possible because the classification is a label. You will see the full Namespace manifest in the worked example.

RBAC delegation for tenants

Post 22 covered how the OAuth server maps LDAP or OIDC groups into OpenShift Group objects. Onboarding consumes that: one AD group per role per project, bound at project scope to the built-in ClusterRoles. admin can manage RoleBindings and most objects in the namespace but cannot edit ResourceQuota, LimitRange or the namespace itself; edit can create and change workloads but not bindings; view is read-only and cannot read Secrets.

$ oc adm policy add-role-to-group admin "AD-PAYMENTS-NOTIF-ADMINS" -n payments-notifications-dev
clusterrole.rbac.authorization.k8s.io/admin added: "AD-PAYMENTS-NOTIF-ADMINS"
$ oc adm policy add-role-to-group edit "AD-PAYMENTS-NOTIF-DEV" -n payments-notifications-dev
clusterrole.rbac.authorization.k8s.io/edit added: "AD-PAYMENTS-NOTIF-DEV"
$ oc adm policy add-role-to-group view "AD-PAYMENTS-NOTIF-READONLY" -n payments-notifications-dev
clusterrole.rbac.authorization.k8s.io/view added: "AD-PAYMENTS-NOTIF-READONLY"
$ oc get rolebindings -n payments-notifications-dev
NAME                    ROLE                               AGE
admin                   ClusterRole/admin                  2m
edit                    ClusterRole/edit                   2m
view                    ClusterRole/view                   2m
system:deployers        ClusterRole/system:deployer        5m
system:image-builders   ClusterRole/system:image-builder   5m
system:image-pullers    ClusterRole/system:image-puller    5m
$ oc adm policy who-can delete resourcequota -n payments-notifications-dev

In the GitOps path those commands become RoleBinding manifests in the tenants repo, but the shape is the same. Three rules hold regardless of mechanism. First, no cluster-scoped grants for tenants, ever: no ClusterRoleBindings, no cluster-reader, no custom ClusterRole that lists nodes. If a team "needs" to see nodes, they need a dashboard, not RBAC. Second, service accounts for automation: if a CI system must talk to the cluster, it gets a dedicated ServiceAccount with edit in that project and short-lived tokens from oc create token ci-deployer -n payments-notifications-dev --duration=1h. Better still, with Argo CD pulling from Git, CI never needs cluster credentials at all. Third, audit the bindings: a scheduled job lists every RoleBinding and ClusterRoleBinding whose subject is an AD group and diffs it against the tenants repo; anything not in Git is drift and gets a ticket. Note that edit can create NetworkPolicies. That is namespace-local and only ever additive within the tenant's own namespace, so many banks allow it; others gate it with a policy so that only the platform-owned baseline exists.

Quotas and limits, with numbers

You met requests, limits and quotas in Post 9. Onboarding is where you turn them into a contract. A ResourceQuota caps what a namespace can consume in total; a LimitRange sets per-container defaults and bounds; a ClusterResourceQuota caps a team across several namespaces. The medium tier looks like this:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tier-medium
  namespace: payments-notifications-prod
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "60"
    persistentvolumeclaims: "10"
    requests.storage: 200Gi
    gp3-csi.storageclass.storage.k8s.io/requests.storage: 200Gi
    efs-sc.storageclass.storage.k8s.io/persistentvolumeclaims: "0"
    count/cronjobs.batch: "10"
    services.loadbalancers: "0"

Read the last three lines carefully, because they are the ones interviewers notice. Storage is capped per StorageClass, so the expensive RWX class is off by default ("0") until the team asks for it. LoadBalancer Services are zero because in a bank the Route through the shared router is the only supported ingress; a tenant creating a LoadBalancer Service would provision a cloud load balancer with its own bill and public IP.

apiVersion: v1
kind: LimitRange
metadata:
  name: tier-medium
  namespace: payments-notifications-prod
spec:
  limits:
  - type: Container
    defaultRequest: {cpu: 100m, memory: 128Mi}
    default: {cpu: 500m, memory: 512Mi}
    min: {cpu: 10m, memory: 32Mi}
    max: {cpu: "4", memory: 8Gi}
    maxLimitRequestRatio: {cpu: "10", memory: "2"}
  - type: PersistentVolumeClaim
    max: {storage: 100Gi}

Why is the LimitRange not optional? Because once a quota constrains requests.cpu, the admission controller rejects any pod that does not state a CPU request, with failed quota: tier-medium: must specify requests.cpu. And a HorizontalPodAutoscaler targeting 70% CPU utilization cannot compute a percentage of a request that does not exist. Two admission-time mechanisms fill in what tenants leave out: the restricted-v2 SCC assigns the UID and drops capabilities (Post 22), and the LimitRange assigns requests and limits. The maxLimitRequestRatio for memory of 2 stops a team from requesting 128Mi and limiting at 8Gi, which is how overcommitted nodes end up evicting neighbours.

ClusterResourceQuota for a team across projects

A team with dev, uat and prod projects would otherwise get three full quotas. A ClusterResourceQuota selects namespaces by label and caps the sum. Project admins see their share through the AppliedClusterResourceQuota view.

apiVersion: quota.openshift.io/v1
kind: ClusterResourceQuota
metadata:
  name: team-payments
spec:
  selector:
    labels:
      matchLabels:
        platform.bank.example/team: payments
  quota:
    hard:
      requests.cpu: "40"
      requests.memory: 80Gi
      pods: "300"

$ oc describe clusterresourcequota team-payments | grep -A5 Resource
Resource          Used   Hard
--------          ----   ----
pods              47     300
requests.cpu      9500m  40
requests.memory   19Gi   80Gi
$ oc get appliedclusterresourcequota -n payments-notifications-dev

Sizing: t-shirt tiers

Tiers keep the intake form simple and stop every request turning into a negotiation. A team picks a size; moving up is a PR that changes one label.

Size the tiers from data, not guesses: after a quarter, query Prometheus for the 95th percentile of namespace:container_cpu_usage:sum against requests per namespace and see how many teams sit below half of what they asked for. Requests that are never used are capacity you paid for and cannot schedule onto.

What happens at the quota

A Deployment does not fail loudly when the quota is hit. The Deployment looks fine, the ReplicaSet cannot create pods, and the evidence is an event on the ReplicaSet.

$ oc scale deployment notifications --replicas=12 -n payments-notifications-dev
$ oc get deploy notifications -n payments-notifications-dev
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
notifications   8/12    8            8           3d
$ oc get events -n payments-notifications-dev --field-selector reason=FailedCreate | tail -n 1
Warning  FailedCreate  replicaset/notifications-6c9d7f9b4  Error creating: pods "notifications-6c9d7f9b4-x2k9p" is forbidden: exceeded quota: default-quota, requested: requests.cpu=250m, used: requests.cpu=2, limited: requests.cpu=2
$ oc describe quota default-quota -n payments-notifications-dev
Name:            default-quota
Resource         Used   Hard
--------         ----   ----
pods             8      20
requests.cpu     2      2
requests.memory  2Gi    4Gi

The teaching moment for tenants is right there in the message: the quota is consumed by requests, not by usage. A team whose pods idle at 50m CPU but request 250m each is paying for five times what it uses. Show them oc adm top pods next to the quota, and the requests-versus-limits conversation from Post 9 lands.

The network baseline per project

The four NetworkPolicies in the template are the baseline: deny all ingress, then allow from the router, from monitoring, and from the same namespace. The namespace selectors rely on labels that OpenShift places on its own namespaces (network.openshift.io/policy-group=ingress on openshift-ingress, monitoring on openshift-monitoring). Verify them with oc get ns -l network.openshift.io/policy-group before you trust the template on a new cluster. On clusters where the ingress controller uses the HostNetwork publishing strategy (common on bare metal and vSphere), router traffic arrives from the node itself and you must also allow from the default namespace, labelled accordingly. Post 21 has the OVN-Kubernetes details.

Egress is where the bank's firewall team meets Kubernetes. Pod IPs are ephemeral and NAT to whichever node the pod is on, which a perimeter firewall cannot write a rule for. An EgressIP pins a namespace's outbound traffic to a fixed address on a node labelled k8s.ovn.org/egress-assignable=""; the firewall request then says "source 10.40.8.21, destination vendor 203.0.113.20 port 443". An EgressFirewall (one per namespace, always named default) enforces the same allow-list inside the cluster, so that even if the perimeter is loose, the workload cannot reach anything unapproved.

apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: payments-notifications-prod
spec:
  egressIPs:
  - 10.40.8.21
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: payments-notifications-prod
---
apiVersion: k8s.ovn.org/v1
kind: EgressFirewall
metadata:
  name: default
  namespace: payments-notifications-prod
spec:
  egress:
  - type: Allow
    to:
      dnsName: api.sms-vendor.example
    ports:
    - protocol: TCP
      port: 443
  - type: Allow
    to:
      cidrSelector: 10.20.0.0/16      # internal services zone
  - type: Deny
    to:
      cidrSelector: 0.0.0.0/0

Exceptions follow the same path as everything else: a PR to the tenants repo adding an allow rule (cross-namespace ingress from a partner app, a new vendor endpoint), reviewed by platform and security, with a ticket reference in the commit. Tenants do not edit the baseline; Argo CD's self-heal reverts any manual change within minutes and the AppProject blocks them from managing those kinds at all.

Deployment patterns the platform standardizes

The golden path tells teams exactly what a compliant workload looks like, so that the policy engine is confirming, not surprising. The standard, in one list: a Deployment, not a DeploymentConfig (deprecated since 4.14, and Deployments work everywhere including EKS); startup, readiness and liveness probes on distinct endpoints; requests and limits on every container; a PodDisruptionBudget; an HPA where the workload scales; topology spread across zones; a non-root image that passes restricted-v2 (Post 22); config from ConfigMaps and Secrets, never baked into the image; graceful shutdown on SIGTERM; images referenced by digest; and the app.kubernetes.io/* labels so dashboards, cost reports and policies can find the workload. Here is the reference Deployment that passes every platform policy.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notifications
  labels:
    app.kubernetes.io/name: notifications
    app.kubernetes.io/instance: notifications-dev
    app.kubernetes.io/version: "1.4.2"
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: payments-notifications
    app.kubernetes.io/managed-by: argocd
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: notifications
      app.kubernetes.io/instance: notifications-dev
  strategy:
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: notifications
        app.kubernetes.io/instance: notifications-dev
        app.kubernetes.io/version: "1.4.2"
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: notifications

      containers:
      - name: app
        image: quay.bank.example/payments/notifications@sha256:9f1c0d2e...e2a7
        ports:
        - name: http
          containerPort: 8080
        envFrom:
        - configMapRef: {name: notifications-config}
        - secretRef: {name: notifications-secrets}
        resources:
          requests: {cpu: 250m, memory: 256Mi}
          limits: {cpu: "1", memory: 512Mi}
        securityContext:
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          capabilities: {drop: [ALL]}
          seccompProfile: {type: RuntimeDefault}
        startupProbe:
          httpGet: {path: /healthz/startup, port: http}
          failureThreshold: 30
          periodSeconds: 2
        readinessProbe:
          httpGet: {path: /healthz/ready, port: http}
          periodSeconds: 5
        livenessProbe:
          httpGet: {path: /healthz/live, port: http}
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:
            sleep: {seconds: 5}    # lets the router drain; needs OCP 4.17+

Notice what is not there: no runAsUser (the SCC assigns one from the project's range), no hostPort, no privileged, no :latest. The preStop sleep gives the router and endpoint controllers a few seconds to stop sending traffic before SIGTERM arrives, which is the difference between a clean rollout and a handful of 502s per deploy. The Service and Route complete the trio.

apiVersion: v1
kind: Service
metadata:
  name: notifications
  labels: {app.kubernetes.io/name: notifications, app.kubernetes.io/instance: notifications-dev}
spec:
  selector:
    app.kubernetes.io/name: notifications
    app.kubernetes.io/instance: notifications-dev
  ports:
  - name: http
    port: 8080
    targetPort: http
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: notifications
spec:
  host: notifications-dev.apps.ocp-dev.bank.example
  to: {kind: Service, name: notifications}
  port: {targetPort: http}
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: notifications}
spec:
  minAvailable: 1
  selector:
    matchLabels: {app.kubernetes.io/name: notifications, app.kubernetes.io/instance: notifications-dev}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: notifications}
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: notifications}
  minReplicas: 2
  maxReplicas: 6
  metrics:
  - type: Resource
    resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

Packaging: manifests, Kustomize, Helm, Templates, Operators

Teams will ask which packaging format to use. The honest answer is that the platform supports two well and tolerates the rest.

The "paved road" is what turns those choices into a five-minute start. The platform team owns a library chart (type: library in Chart.yaml), which contains the named templates for the golden Deployment, Service, Route, PDB and HPA above, plus a values.schema.json that refuses values without an image digest, probes or resources. A team's application chart declares the library as a dependency and is mostly a values.yaml. When the platform tightens a default, it bumps the library version and every team picks it up on their next release, which is how you change a standard across two hundred apps without two hundred tickets.

platform-charts/
  bank-app/                 # library chart, type: library, version 2.3.0
    Chart.yaml
    values.schema.json      # rejects values missing image.digest, probes, size
    templates/_deployment.tpl  _service.tpl  _route.tpl  _pdb.tpl  _hpa.tpl
starter-app/                # cloned by every new team
  Chart.yaml                # dependencies: bank-app ^2.0 from oci://quay.bank.example/platform/charts
  values.yaml               # image, port, size, env, secrets, route
  templates/all.yaml        # {{ include "bank-app.all" . }}
platform-ci/
  tekton/build-and-push.yaml   # or Jenkins shared library / GitHub Actions reusable workflow
tenants/                    # namespace-as-code, one file per project
  payments-notifications/dev.yaml  uat.yaml  prod.yaml

Onboarding itself is an Argo CD ApplicationSet with a Git files generator: every YAML under tenants/ becomes an Application that renders the tenant chart (Namespace, bindings, quota, LimitRange, policies) with that file as its values. Adding a team is one file and one PR. Post 30 goes deep on Argo CD; here is the shape.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenants
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: https://git.bank.example/platform/tenants.git
      revision: main
      files:
      - path: "tenants/*/*.yaml"
  template:
    metadata:
      name: "tenant-{{ .app }}-{{ .env }}"
    spec:
      project: platform-tenants
      source:
        repoURL: https://git.bank.example/platform/tenants.git
        targetRevision: main
        path: charts/tenant
        helm:
          valueFiles:
          - "../../tenants/{{ .app }}/{{ .env }}.yaml"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{ .app }}-{{ .env }}"
      syncPolicy:
        automated: {prune: true, selfHeal: true}

Builds and images

Two ways to get an image. Build in CI: GitHub Actions, Jenkins or Tekton (OpenShift Pipelines) builds the image, generates an SBOM, scans it, signs it and pushes to the corporate registry (Quay or Artifactory); the cluster only ever pulls. Build in-cluster: a BuildConfig, typically Source-to-Image (S2I), where oc new-app or oc start-build turns source into an image inside an ImageStream in the internal registry. S2I is a fine developer experience for sandboxes, but a bank standardizes on CI builds: one place for scanning and signing gates, one provenance record, and the same pipeline whether the target is OpenShift or EKS.

An ImageStream is OpenShift's pointer to image tags, with change tracking and triggers. Even with CI builds it is useful as a stable in-cluster reference that records which digest a tag pointed at when the deploy happened.

$ oc import-image notifications:1.4.2 \
    --from=quay.bank.example/payments/notifications:1.4.2 \
    --confirm --scheduled -n payments-notifications-dev
imagestream.image.openshift.io/notifications imported
$ oc get istag notifications:1.4.2 -n payments-notifications-dev -o jsonpath='{.image.metadata.name}'
sha256:9f1c0d2e...e2a7

Image policy is cluster-wide and belongs to the platform team. The image.config.openshift.io/cluster object lists allowed registries; anything else fails to pull with a clear denial. Scanning gates live in the registry (Quay's Clair, Artifactory Xray) and in Red Hat Advanced Cluster Security's admission controller, which can refuse a deployment whose image has a critical CVE with a fix available. Post 31 covers signing and verification.

apiVersion: config.openshift.io/v1
kind: Image
metadata:
  name: cluster
spec:
  registrySources:
    allowedRegistries:
    - quay.bank.example
    - registry.redhat.io
    - quay.io
    - image-registry.openshift-image-registry.svc:5000

Secrets and config for tenants

The rule teams hear first: nothing sensitive in Git, ever, and "encoded" is not "encrypted". Git holds ConfigMaps with non-sensitive settings, Helm values files, and references to secrets. The secret material lives in the bank's vault (HashiCorp Vault on-prem, AWS Secrets Manager for EKS), and the External Secrets Operator (ESO) syncs it into Kubernetes Secrets at runtime. Onboarding creates the Vault path and a Kubernetes auth role that trusts one ServiceAccount in one namespace, which is how Vault knows that a request really comes from payments-notifications-dev.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault
  namespace: payments-notifications-dev
spec:
  provider:
    vault:
      server: https://vault.bank.example
      path: kv/payments-notifications
      version: v2
      auth:
        kubernetes:
          mountPath: ocp-dev
          role: payments-notifications-dev
          serviceAccountRef: {name: external-secrets-reader}
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: notifications-secrets
  namespace: payments-notifications-dev
spec:
  refreshInterval: 1h
  secretStoreRef: {name: vault, kind: SecretStore}
  target: {name: notifications-secrets}
  data:
  - secretKey: SMS_VENDOR_API_KEY
    remoteRef: {key: dev/sms-vendor, property: api_key}

The Deployment then references notifications-secrets exactly as in the golden manifest; rotation in Vault flows through on the next refresh. Where ESO cannot be used (a disconnected cluster, a vendor that ships a static credential), Sealed Secrets is the fallback: kubeseal encrypts a Secret with the cluster's public key into a SealedSecret object that is safe to commit, and only the controller in the cluster can decrypt it. It is safe but it is not a secret manager: no rotation, no audit of reads, and a key per cluster to back up. Post 6 has the Kubernetes basics.

Observability for tenants

Everything in Post 24 was platform-side. Tenant-side, onboarding gives a team four things. Metrics: with user workload monitoring enabled (enableUserWorkload: true in the cluster-monitoring-config ConfigMap), the team creates a ServiceMonitor in its project and its metrics appear in the console's Observe pages and in Grafana, scoped to projects the team can see. The monitoring-edit role, bound at project scope, is what allows that. Alerts: PrometheusRule objects in the project, with monitoring-rules-edit. Routing: an AlertmanagerConfig in the project (requires enableAlertmanagerConfig: true in the user workload config and the alert-routing-edit role) sends the team's alerts to the team's channel or PagerDuty service, not to the platform on-call.

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: team-routing
  namespace: payments-notifications-prod
spec:
  route:
    receiver: payments-notif-oncall
    groupBy: [alertname]
  receivers:
  - name: payments-notif-oncall
    pagerdutyConfigs:
    - routingKey: {name: pagerduty-key, key: routingKey}    # synced by ESO

Logs: with Loki and the console logging plugin, a developer opens Observe → Logs and sees only namespaces they hold get pods on; oc logs keeps working for the live tail. The handover doc tells them all four exist, because a team that does not know it can route its own alerts will ask you to do it forever.

Governance controls

Governance is not a review meeting; it is a set of admission and drift controls that make the standard the default. Policy engine: Kyverno (YAML policies, easy to author), Gatekeeper (Rego, shipped by Red Hat with Advanced Cluster Management) or Red Hat Advanced Cluster Security (policies plus vulnerability context, the usual pick for a bank because it is supported). Whichever engine, the tenant baseline is the same: required ownership labels, probes present, no :latest, no privileged or host-namespace pods, images only from allowed registries, no cluster-scoped RBAC created by tenants. Pod Security Admission labels are synced from SCCs on OpenShift, so the "no privileged" rule is largely enforced already; the policy engine adds the operational rules.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-baseline
spec:
  validationFailureAction: Enforce    # newer Kyverno: validate.failureAction
  background: true
  rules:
  - name: require-ownership-labels
    match:
      any:
      - resources:
          kinds: [Deployment, StatefulSet, CronJob]
    exclude:
      any:
      - resources:
          namespaces: ["openshift-*", "kube-*"]
    validate:
      message: "app.kubernetes.io/name and app.kubernetes.io/part-of are required."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"
            app.kubernetes.io/part-of: "?*"
  - name: disallow-latest-tag
    match:
      any:
      - resources:
          kinds: [Pod]
    exclude:
      any:
      - resources:
          namespaces: ["openshift-*", "kube-*"]
    validate:
      message: "Images must be pinned by digest or version, not :latest."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Around the engine sit the process controls. ClusterResourceQuota caps a team's total spend. Audit logging on the API server (profile WriteRequestBodies for a bank) records who changed what, and a break-glass action is only acceptable because it is in that log. Exceptions are objects in Git with an owner, a ticket, a justification and an expiry date; a scheduled job fails the build when an exception passes its expiry, so nothing is "temporary" for three years. Change management: CI/CD through Argo CD is the approved deployment path, the merged PR with its approvals is the change record, and production Applications are gated on a change ticket in the pipeline. Nobody runs oc apply against prod. Emergency access is break-glass: a privileged access management tool (CyberArk or similar) issues a time-boxed credential, the platform's automation creates a RoleBinding with a recorded expiry, the audit log captures every call, and a post-incident review explains why it was needed.

Documentation, standards and the onboarding FAQ

A platform handbook is the product manual. It contains: a getting-started page that takes a new team from intake to first deploy in under an hour; the golden paths (Helm, Kustomize, batch, event-driven) with copyable examples; the policies, each with the "why" and the exact error message a team will see; runbooks for the situations tenants hit (quota exceeded, image pull denied, route 503, alert not routing); an FAQ; the support model with SLAs and what "golden path" versus "best effort" means; and office hours. It is docs-as-code: Markdown in Git, rendered by MkDocs or Backstage TechDocs, reviewed through PRs, versioned with the platform release, so that the docs for the cluster you are on are the docs you are reading. Ship a tenant runbook template too (service overview, dependencies, dashboards, alerts and what to do about each, rollback procedure, on-call rota), because a team that fills it in during onboarding has a runbook on the night it matters.

Measure onboarding or you cannot improve it: time-to-first-deploy (target under two days from approved ticket), tickets raised per onboarding, the share of production deploys that go through the golden path, and policy violations per month trending down. The FAQ writes itself from the tickets you see most.

Worked example: onboarding payments-notifications

The intake ticket reads: application Payments Notifications, CMDB APP0012345, team payments, environments dev and prod, classification confidential (customer phone numbers), tier S for dev and M for prod, RWO storage only, outbound to api.sms-vendor.example on 443 and to the internal services zone, inbound from the internet through the standard router, AD groups AD-PAYMENTS-NOTIF-ADMINS, -DEV and -READONLY. Security approves on condition that the vendor key lives in Vault. The platform engineer opens one PR adding two files. Here is the dev one.

# tenants/payments-notifications/dev.yaml
app: payments-notifications
env: dev
team: payments
cmdbId: APP0012345
costCenter: CC-4471
dataClassification: confidential
tier: S
owner: payments-notifications-oncall@bank.example
requester: jsmith@bank.example
rbac:
  admins: [AD-PAYMENTS-NOTIF-ADMINS]
  developers: [AD-PAYMENTS-NOTIF-DEV]
  viewers: [AD-PAYMENTS-NOTIF-READONLY]
egress:
  allowDns: [api.sms-vendor.example]
  allowCidrs: [10.20.0.0/16]
storageClasses: [gp3-csi]
gitops:
  repo: https://git.bank.example/payments/notifications-deploy.git

On merge, the ApplicationSet renders tenant-payments-notifications-dev, which creates the Namespace with its labels and annotations, three RoleBindings, the tier-S ResourceQuota and LimitRange, the four baseline NetworkPolicies, the EgressFirewall, the ESO SecretStore and a ServiceAccount for it, and the AppProject that fences the team's own Argo CD Application.

apiVersion: v1
kind: Namespace
metadata:
  name: payments-notifications-dev
  labels:
    platform.bank.example/app: payments-notifications
    platform.bank.example/env: dev
    platform.bank.example/team: payments
    platform.bank.example/cost-center: CC-4471
    platform.bank.example/data-classification: confidential
    platform.bank.example/tier: S
    argocd.argoproj.io/managed-by: openshift-gitops
  annotations:
    openshift.io/display-name: "Payments Notifications (dev)"
    openshift.io/description: "Push and SMS notifications for payments. CMDB APP0012345."
    openshift.io/requester: jsmith@bank.example
    platform.bank.example/cmdb-id: APP0012345
    platform.bank.example/owner: payments-notifications-oncall@bank.example
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developers-edit
  namespace: payments-notifications-dev
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: edit}
subjects:
- {apiGroup: rbac.authorization.k8s.io, kind: Group, name: AD-PAYMENTS-NOTIF-DEV}

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments-notifications
  namespace: openshift-gitops
spec:
  sourceRepos:
  - https://git.bank.example/payments/notifications-deploy.git
  destinations:
  - server: https://kubernetes.default.svc
    namespace: payments-notifications-*
  clusterResourceWhitelist: []            # nothing cluster-scoped, ever
  namespaceResourceBlacklist:
  - {group: "", kind: ResourceQuota}
  - {group: "", kind: LimitRange}
  - {group: networking.k8s.io, kind: NetworkPolicy}
  - {group: k8s.ovn.org, kind: EgressFirewall}
  - {group: rbac.authorization.k8s.io, kind: RoleBinding}
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-notifications-dev
  namespace: openshift-gitops
spec:
  project: payments-notifications
  source:
    repoURL: https://git.bank.example/payments/notifications-deploy.git
    targetRevision: main
    path: chart
    helm:
      valueFiles: [values-dev.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: payments-notifications-dev
  syncPolicy:
    automated: {prune: true, selfHeal: true}

The team clones the starter chart into its deploy repo, and its entire deployment is a values file. Everything else comes from the library chart.

# notifications-deploy/chart/values-dev.yaml  (rendered by the bank-app library chart)
image:
  repository: quay.bank.example/payments/notifications
  digest: sha256:9f1c0d2e...e2a7
port: 8080
replicas: 2
size: small                 # requests 250m/256Mi, limits 1/512Mi
probes:
  startup: /healthz/startup
  ready: /healthz/ready
  live: /healthz/live
env:
  LOG_LEVEL: info
  VENDOR_URL: https://api.sms-vendor.example
externalSecrets:
- name: notifications-secrets
  keys:
    SMS_VENDOR_API_KEY: {key: dev/sms-vendor, property: api_key}
route:
  host: notifications-dev.apps.ocp-dev.bank.example
pdb: {minAvailable: 1}
hpa: {min: 2, max: 4, cpuUtilization: 70}

CI builds the image, pushes it to Quay with the robot account, and updates image.digest in the values file through a PR. Argo CD syncs. The engineer confirms with the team on a call, then writes the handover.

$ oc get application -n openshift-gitops | grep payments-notifications
payments-notifications-dev          Synced   Healthy
tenant-payments-notifications-dev   Synced   Healthy
$ oc get pods -n payments-notifications-dev
NAME                             READY   STATUS    RESTARTS   AGE
notifications-6c9d7f9b4-2xk8q    1/1     Running   0          58s
notifications-6c9d7f9b4-q7m2z    1/1     Running   0          58s
$ curl -s https://notifications-dev.apps.ocp-dev.bank.example/healthz/ready
{"status":"ok","version":"1.4.2"}
$ oc get quota,limitrange,networkpolicy,egressfirewall -n payments-notifications-dev --no-headers | wc -l
7

The handover document lists the two project names, the three AD groups, the tier, the Argo CD and Grafana links, the Vault path, the alert channel, the runbook template link, and the sentence "to change any of this, open a PR against tenants/payments-notifications/ or a ticket referencing APP0012345". Ticket opened Monday morning, first green deploy Tuesday afternoon: time-to-first-deploy of 1.3 days, logged.

Likely interview questions

Next up: Amazon EKS Part 1, where the same platform engineer has to reason about VPC CNI IP exhaustion, access entries and Pod Identity instead of OVN, OAuth and SCCs, and learns which OpenShift habits transfer and which do not.