StatefulSets, Jobs & CronJobs — Beyond Stateless Apps

So far, every Pod you've deployed has been a clone — a Deployment can kill any Pod and replace it with an identical twin, and nothing breaks. But some workloads don't work that way. What if Pod-0 is the leader of a database cluster, holding data that Pod-1 and Pod-2 don't have? A Deployment has no concept of "this specific Pod matters." This post covers the three workload types built for exactly that case.

The gap Deployments can't fill

Back in Post 4, a Deployment's whole design rests on one assumption: every replica is identical, and any of them can be thrown away and replaced without anyone noticing. Scale down, and Kubernetes kills whichever Pod is convenient. Scale up, and it creates a new Pod with a random name suffix, like web-7d9f6b8-x4k2p. Move a Pod to a different node, and it comes back with a brand-new name and — unless you attached a PVC yourself — an empty disk. None of that matters for a stateless web server: any replica can answer any request, so it doesn't matter which one you get.

Now picture a 3-node database cluster — Postgres, Cassandra, Kafka, whatever. Node 0 might be the elected leader. Nodes 1 and 2 are replicas, syncing data from node 0 and from each other. Each node stores its own slice of data on disk, and that data is not interchangeable — lose node 1's disk, and you've lost a third of your replicated data, not "just spin up a clone." And the other nodes need to reach "node 1" specifically, by a name that never changes — not whichever random Pod happens to be alive right now. A Deployment can't express any of that, because it's built entirely around sameness. That's the gap a StatefulSet fills.

StatefulSet: identity that survives restarts

A StatefulSet gives up "any Pod will do" and replaces it with three specific guarantees.

1. Stable, predictable names. No random suffixes. A StatefulSet named db always produces Pods named db-0, db-1, db-2. Delete db-1, and Kubernetes recreates a Pod with that exact same name, db-1 — not a new identity, the same one coming back.

2. Stable network identity, via a headless Service. A normal Service (Post 5) load-balances traffic: it hands out one virtual IP, and you never know which backing Pod actually answered a given request. A headless Service is a Service with no virtual IP at all — you get one by setting clusterIP: None. Instead of load-balancing, it gives every Pod its own DNS name. So db-0.db.default.svc.cluster.local always resolves to Pod db-0, specifically. That's exactly what a replica needs when it has to sync from "the leader" by name — not from "whichever Pod happens to answer."

3. Its own disk per Pod, via volumeClaimTemplates. A PersistentVolumeClaim, or PVC (Post 7), is a request for storage that decouples a Pod from the physical disk behind it. A StatefulSet's volumeClaimTemplates block generates one separate PVC per replica: db-0 gets its own claim, db-1 gets a different one, and so on. If db-1 gets rescheduled onto a new node, Kubernetes reattaches that same PVC to the recreated db-1 Pod — its data follows it there. A single Deployment PVC shared across many replicas can't offer that; each replica needs its own.

One more habit to unlearn: a StatefulSet creates, scales, and deletes Pods in order — 0 first, then 1, then 2, and it tears them down in reverse. A Deployment doesn't care what order its Pods start in. A StatefulSet does, because Pod 1 might need Pod 0 already up and running before it can join the cluster correctly.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db          # must match a headless Service
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

$ kubectl get statefulsets
NAME   READY   AGE
db     3/3     4m

$ kubectl get pods -l app=db
NAME    READY   STATUS    RESTARTS   AGE
db-0    1/1     Running   0          4m
db-1    1/1     Running   0          3m
db-2    1/1     Running   0          2m

Look at the -0, -1, -2 suffixes, and the staggered ages: 4m, 3m, 2m. Each Pod came up one at a time, oldest first — that ordering is the StatefulSet doing its job.

Job: run once, finish, stop

Now flip the problem. A Deployment also assumes its Pod should run forever: if the container exits, the Deployment treats that as a crash and restarts it, over and over. That's correct for a web server — it should never stop on its own. It's completely wrong for a one-off script, like a data migration or a batch report, where finishing successfully is the entire point. Point a Deployment at a script like that, and you get an infinite restart loop of a task that already did its job.

A Job is built for exactly this: run-to-completion work. The Job controller starts a Pod, watches it run, and once the container exits successfully, the Job marks itself Complete and stops. No restart, no leftover Pod pretending to still be needed. If the container fails instead, the Job retries it, up to a limit you set with backoffLimit.

apiVersion: batch/v1
kind: Job
metadata:
  name: data-migration
spec:
  backoffLimit: 3
  template:
    spec:
      containers:
        - name: migrate
          image: myapp-migrator:latest
          command: ["python", "migrate.py"]
      restartPolicy: Never

$ kubectl get jobs
NAME             COMPLETIONS   DURATION   AGE
data-migration   1/1           14s        1m

$ kubectl logs job/data-migration
Connecting to database...
Applying migration 0032_add_index...
Migration complete.

CronJob: the same Job, on a schedule

A CronJob is a template that creates Jobs automatically, on a repeating schedule written in standard cron syntax — the same five-field format used by Unix cron. "0 2 * * *" means "at 2:00 AM, every day": minute 0, hour 2, any day of the month, any month, any day of the week. That's exactly what you want for a nightly backup or a scheduled report — no human, and no scheduler outside the cluster, has to trigger it by hand.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: myapp-backup:latest
              command: ["./backup.sh"]
          restartPolicy: OnFailure

$ kubectl get cronjobs
NAME             SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   0 2 * * *   False     0        16h ago         3d

The successfulJobsHistoryLimit and failedJobsHistoryLimit fields matter more than they look. Without them, every completed Job object from every run sticks around forever, cluttering kubectl get jobs with months of leftover history. Set these deliberately instead of trusting whatever default your cluster ships with.

You don't have to wait until 2 AM to find out if your CronJob actually works. Trigger a run right now, on demand:

$ kubectl create job --from=cronjob/nightly-backup manual-run-1
job.batch/manual-run-1 created

$ kubectl logs job/manual-run-1
Starting backup...
Backup uploaded to storage bucket.

Next up: the final post in this series — pulling everything together into real troubleshooting workflows and exam-day strategy for the CKA.