Scheduling, Health Probes & Resource Management

Back in Post 2, we called the kube-scheduler "the dispatcher" and left it at that. Time to open that box: how does it know a node has room, what happens when a container uses more than it's allowed, and how does Kubernetes know a running container is actually okay? Same idea underneath all three: Kubernetes never just trusts your app to be fine — it checks.

Requests and limits: two numbers, two different jobs

Every container in a pod spec can set two numbers, for CPU and for memory: a request and a limit. They sound alike but do different jobs.

A request is a reservation: "hold at least this much aside for me." Set requests.memory: 256Mi and Kubernetes guarantees this container 256Mi. It's the number the scheduler reads when deciding which node a pod lands on — it never looks at limits for that.

A limit is a ceiling: "never let this container use more than this." It plays no part in scheduling — it's enforced later, on the node, by the kubelet, once the pod is already running.

Now the part people mix up constantly, so read this twice: what happens at the ceiling depends on which resource you're talking about. Hit your memory limit and the container gets OOMKilled — "OOM" is short for "out of memory." Linux's OOM killer does exactly what it sounds like: the instant a process asks for more memory than it's allowed, the kernel kills it, no warning, no negotiating — a process either has the bytes or it doesn't. Kubernetes then restarts the container, the same as if it had crashed.

Hit your CPU limit and nothing dies. The container gets throttled instead — the kernel just refuses to give it more CPU time than its limit allows, so it runs slower but keeps running. CPU is like a shared road with a speed limit: go over it and you get held back, not arrested. Memory has no "slow down" option, so the only move left is to kill the process.

Exceed a memory limit and the container dies and restarts. Exceed a CPU limit and the container just slows down. Same ceiling, two very different consequences — know this cold, it's one of the most commonly confused facts on the exam.

How the scheduler actually picks a node

Back to the dispatcher from Post 2. When a new pod needs a home, the scheduler checks each node's unreserved capacity — total CPU and memory, minus whatever every other pod there has already requested — and looks for a node with enough room for this pod's requests. Note that word: requested, not currently used. A node can be idle on real usage and still count as "full" if its pods have requested more than there's room for. If no node clears the bar, the pod doesn't error out — it just sits unscheduled, in a state called Pending.

The simplest way to steer placement yourself is nodeSelector: a plain key-value match against labels on a node. Label a node disktype=ssd, add nodeSelector: {disktype: ssd} to your pod spec, and it only lands there. Blunt — an exact match or nothing — but easy to reason about.

Kubernetes also has affinity/anti-affinity (softer preferences, like "try to keep these two pods apart") and taints and tolerations (nodes that repel pods by default, unless a pod says it tolerates that taint). They're real, and you'll meet them in bigger clusters, but they're more advanced than we need right now — just file away the names for when nodeSelector isn't precise enough.

Health probes: is it actually okay in there?

Requests and limits are about resources. A probe asks something different: is the software inside the container actually working? A container can show status "Running" — process started, hasn't crashed — while the app inside is frozen, deadlocked, or stuck loading a cache. Kubernetes can't see any of that by itself. You have to tell it how to check.

In Kubernetes terms: a failed livenessProbe means kubelet kills the container and restarts it, same as a crash — for genuinely stuck states like a deadlock, where the only fix is a fresh start.

A failed readinessProbe pulls the pod out of the Service's Endpoints list from Post 5 — traffic stops flowing — but the container keeps running, untouched. Picture an app that takes 20 seconds to warm up a cache: it's not broken, just not ready yet, and killing it would be pointless.

startupProbe exists for slow-booting apps — a heavy migration or huge dataset to load before the app is meaningfully alive. Without one, a livenessProbe might start checking (and killing the container) before boot finishes, creating a self-inflicted crash loop on an app that was never broken. startupProbe holds off both checks until it succeeds once, giving slow starters runway first.

Here's a pod using both liveness and readiness together:

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
  - name: web
    image: myapp:1.0
    resources:
      requests:
        memory: "128Mi"
        cpu: "250m"
      limits:
        memory: "256Mi"
        cpu: "500m"
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5

Both hit HTTP endpoints your app exposes: /healthz for "am I fundamentally okay," /ready for "can I serve traffic right now." They needn't be the same endpoint — a database hiccup might reasonably fail readiness without the whole container needing to die.

Your diagnostic reflex: describe, top, describe again

kubectl describe pod is the single most useful command in this post — build the reflex to reach for it first. Scroll to the Events section at the bottom and you'll find both scheduling failures and probe failures logged in plain language:

$ kubectl describe pod web
...
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  2m                 default-scheduler  0/3 nodes are available: 3 Insufficient memory.
  Normal   Scheduled         90s                default-scheduler  Successfully assigned default/web to node-2
  Normal   Pulled            88s                kubelet            Successfully pulled image "myapp:1.0"
  Warning  Unhealthy         30s (x3 over 50s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  BackOff           10s                kubelet            Back-off restarting failed container

That's the scheduler struggling to place the pod, and later the kubelet struggling to keep it healthy — both in the exact same place.

For resource usage itself, kubectl top gives a live snapshot:

$ kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   410m         10%    1823Mi          46%

$ kubectl top pods
NAME   CPU(cores)   MEMORY(bytes)
web    12m          98Mi

kubectl top requires the metrics-server add-on — it's not installed by default. On minikube: minikube addons enable metrics-server, then give it a minute before numbers show up.

Next up: Ingress & Load Balancing — giving the outside world one clean door into all the Services running in your cluster.