Module 04 · Keep it healthy · 45 min · You are: Platform and product

Probes and disruption budgets: staying up while things change

In this module

You learn: What the three probes are for (startup: has it started; readiness: should it get traffic; liveness: should it be restarted), and how a PodDisruptionBudget tells the platform team how many pods it may take away at once.
You do: Add the three probes to checkout and switch one pod's readiness off to see traffic move away. Spread the pods over nodes and add a budget. Then drain a node as the platform team while a loop of requests keeps running.
You end with: checkout has startup, readiness and liveness probes, a spread constraint and a PodDisruptionBudget checkout with maxUnavailable: 1. Both workers are schedulable again and the pods are on different nodes.

A platform team replaces nodes all the time: upgrades, patches, scaling down. Each time, every pod on the node is evicted. Whether users notice depends on things the product team writes: probes that tell Kubernetes when a pod can take traffic, and a budget that tells the platform how many pods may be missing. Written badly, the same objects do harm: a liveness probe can kill a healthy app, and a budget can block a node upgrade for ever.

Open this module in the interactive course to track progress.

Concepts

Readiness probe: "Send me traffic?" While it fails, the pod is taken out of the Service's endpoints. It is never restarted for it.
Liveness probe: "Am I beyond repair?" When it fails failureThreshold times in a row, the kubelet kills the container and starts it again. It must be cheap and depend on nothing outside the process.
Startup probe: "Have I started yet?" Until it succeeds, the other two are not run. Its failureThreshold times periodSeconds is the time the app is given to start.
Voluntary disruption: An eviction someone chose: a drain, a scale-down. A node dying is involuntary; no budget can stop that.
PodDisruptionBudget: For pods matching a selector, how many may be unavailable at once (maxUnavailable) or must stay (minAvailable). kubectl drain evicts through an API that respects it.

Three probes

Probe	Question	On failure	Typical mistake
Startup	Started yet?	Keep waiting, then restart	Replaced by a long `initialDelaySeconds` on liveness
Readiness	Send traffic?	Removed from the endpoints	Checks a dependency every pod shares, so all pods go unready together
Liveness	Restart?	Container killed and started again	Points at a slow or dependency-checking endpoint

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

apps/checkout/deployment.yaml (reference, your repo): The probes and the spread, in their final form.
1. Prefer one pod per node L27–33
2. Out of the Service after two failures, ten seconds L65–70
3. /healthz answers from memory: no database, no other service L71–77
4. Up to 60 seconds to start L78–83
apps/checkout/pdb.yaml (reference, your repo): team-checkout's promise to the platform team.
1. The platform may take one pod away at a time L8
2. Which pods it covers L9–11

Steps

1. Readiness: should this pod get traffic?

Add a readiness probe. Then tell one pod to pretend it is not ready, and watch traffic move away from it.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, after the resources: block

          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
            failureThreshold: 2

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: readiness probe" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

Keep the first pod's name in a variable:

POD=$(kubectl -n checkout-dev get pod -l app.kubernetes.io/name=checkout -o jsonpath='{.items[0].metadata.name}'); echo $POD

podinfo can pretend to be unready. Tell this one pod to:

kubectl -n checkout-dev exec $POD -- wget -qO- --post-data='' http://localhost:9898/readyz/disable

After ten seconds, one pod shows READY 0/1, STATUS still Running, RESTARTS 0. Not ready is not broken.

kubectl -n checkout-dev get pods

The Service dropped it. Run this five times: only the other pod's hostname answers.

curl -s http://checkout.localtest.me:8090/api/info | grep hostname

Make it ready again, and expect 1/1 on both after ten seconds:

kubectl -n checkout-dev exec $POD -- wget -qO- --post-data='' http://localhost:9898/readyz/enable

kubectl -n checkout-dev get pods

2. Liveness and startup

Liveness restarts a container that is stuck; startup gives it time to start before liveness counts. Both point at /healthz, which podinfo answers from memory.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, after the readinessProbe: block

          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 2
            failureThreshold: 30

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: liveness and startup probes" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

How the kubelet reads them: three lines, each with delay, timeout, period and #failure.

kubectl -n checkout-dev describe pod -l app.kubernetes.io/name=checkout | grep -E '^\s+(Liveness|Readiness|Startup):' | head -n 3

A container that dies is restarted in place. Crash one (the command itself fails, because the process is gone):

kubectl -n checkout-dev exec deployment/checkout -- wget -qO- http://localhost:9898/panic

Expect the same pod names, one with RESTARTS 1. A restart is the kubelet's job and keeps the pod; a replacement is the ReplicaSet's.

kubectl -n checkout-dev get pods

3. Spread the pods

Two pods on the same node are one node failure from zero. Ask the scheduler to put them on different nodes when it can.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the pod spec:, above containers:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: checkout

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: spread over nodes" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

The NODE column should show k8s-worker and k8s-worker2:

kubectl -n checkout-dev get pods -o wide

4. The budget, written by you

team-checkout tells the platform team: you may take one of my pods away at a time.

Create ~/platformengr/k8s-platform/apps/checkout/pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
  namespace: checkout-dev
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: disruption budget" && git push -q

kubectl apply -f apps/checkout/pdb.yaml

Expect ALLOWED DISRUPTIONS 1: two healthy pods, one may go.

kubectl -n checkout-dev get poddisruptionbudget checkout

5. Platform team: drain a node under load

Every node upgrade starts with a drain: mark the node unschedulable and evict its pods. Do it while requests keep coming.

In a second terminal, send a request every half second and leave it running. It prints one status code per request; you want an unbroken row of 200.

while true; do curl -s -o /dev/null -m 2 -w '%{http_code} ' http://checkout.localtest.me:8090/healthz; sleep 0.5; done

Drain a worker, in the first terminal:

kubectl drain k8s-worker --ignore-daemonsets --delete-emptydir-data

The evicted pod was recreated on the other worker. Expect both checkout pods on k8s-worker2:

kubectl -n checkout-dev get pods -o wide

And the node is Ready,SchedulingDisabled:

kubectl get nodes

Look at the second terminal: still 200. Put the node back:

kubectl uncordon k8s-worker

Nothing moves a pod back when a node returns. Spread them again with a restart, then check NODE again. Stop the loop with Ctrl+C afterwards.

kubectl -n checkout-dev rollout restart deployment/checkout

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

kubectl -n checkout-dev get pods -o wide

Simplified for the course

podinfo starts in under a second, so the startup probe never has to wait here. It matters for apps that take a minute to warm up: give them a startup probe with a generous budget, never a long liveness delay.
A graceful rollout also needs the app to finish open requests on SIGTERM, and often a short preStop sleep so the endpoint is removed before the process stops. podinfo handles SIGTERM itself.
Nothing rebalances pods after a node comes back. The descheduler does that in production.
A drain stands in for a node upgrade. Upgrading the control plane and replacing nodes is done by your cloud or by Cluster API, one node at a time, with exactly this eviction.

Break it on purpose: We cannot afford to lose a pod

After an outage, team-checkout decides both pods must always be up and changes the budget to minAvailable: 2. It reads as a sensible safety measure. That night the platform team starts a node upgrade.

You practise: reading a drain that hangs, finding which budget holds it, and fixing it without the shortcut that causes the outage the budget was meant to prevent.

1. Break it

As team-checkout:

Change in ~/platformengr/k8s-platform/apps/checkout/pdb.yaml

-   maxUnavailable: 1
+   minAvailable: 2

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: never below 2 pods" && git push -q

kubectl apply -f apps/checkout/pdb.yaml

That night, as the platform team. Leave it running:

kubectl drain k8s-worker2 --ignore-daemonsets --delete-emptydir-data

2. What you see

Expected: node/k8s-worker2 drained within a minute. You get: the same two lines every five seconds, for as long as you wait: evicting pod checkout-dev/checkout-... and error when evicting pods/"checkout-..." -n "checkout-dev" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

The node is Ready,SchedulingDisabled. The site is up. Leave the drain running and open another terminal.

3. Fix it yourself

Let the drain finish, with checkout answering the whole time. kubectl delete pod would work and is not allowed: it bypasses the budget, which is exactly what the team was afraid of. Done when the drain prints drained, and after kubectl uncordon k8s-worker2 both workers are Ready.

Hint 1

kubectl -n checkout-dev get poddisruptionbudget: read ALLOWED DISRUPTIONS. How many pods are there, and how many must stay?

Hint 2

With 2 replicas and minAvailable: 2, zero pods may ever be evicted. The budget has to leave room for one: maxUnavailable: 1. It is team-checkout's file.

The fix

Change in ~/platformengr/k8s-platform/apps/checkout/pdb.yaml

-   minAvailable: 2
+   maxUnavailable: 1

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: budget allows one pod away" && git push -q

kubectl apply -f apps/checkout/pdb.yaml

Look at the terminal with the drain: its next retry succeeds, the pod is recreated on the other worker, and it prints drained. Then put the node back and spread the pods again:

kubectl uncordon k8s-worker2

kubectl -n checkout-dev rollout restart deployment/checkout

After you fixed it: the five questions

What failed?: The node drain: a budget that allowed zero disruptions made every eviction fail.
How was it detected?: By the platform team, as a drain that never ends. On a managed cluster it shows as a node upgrade that stalls and is then forced after a timeout, which evicts the pods anyway.
Who responds?: The platform team notices; team-checkout owns the budget and has to change it. The platform team does not edit a team's budget at 2 a.m.
How is it corrected?: A budget that allows one pod to be away.
How is it prevented?: A platform rule that rejects budgets with zero allowed disruptions (an admission policy), an alert on budgets whose allowed disruptions stay at 0, and a drain timeout in the upgrade runbook with a named owner to call. Teams that need "never below 2" run 3 replicas.

Quiz

A pod's readiness probe fails. What does Kubernetes do?
- It restarts the container after failureThreshold failures in a row
- It stops sending the Service's traffic to that pod and leaves it running
- It deletes the pod, so that the ReplicaSet creates a fresh one in its place
Why should a liveness probe not check the database?
- A slow database would make every pod fail the probe and be restarted at once, turning a slowdown into an outage
- Probes run on the node, and the node has no network route to a database
- The kubelet caches probe results, so a database check would always show stale data
A Deployment has 2 replicas and a budget with minAvailable: 2. What happens when its node is drained?
- The drain evicts the pod after the budget's grace period has passed
- The budget is ignored, because it asks for more than the Deployment can give
- The drain cannot evict the pod and retries without end

Answers

It stops sending the Service's traffic to that pod and leaves it running Readiness controls traffic; only liveness restarts.
A slow database would make every pod fail the probe and be restarted at once, turning a slowdown into an outage Restarting the app cannot fix its dependency.
The drain cannot evict the pod and retries without end Zero allowed disruptions means no voluntary eviction, ever.

Field note

A liveness probe is a loaded gun pointed at your own app. I ask every team the same question about theirs: name a failure that a restart fixes and that the process would not exit from on its own. If there is none, a readiness probe is enough.

Report a problem with this module