Cheat notes · Module 04 · Keep it healthy

Probes and disruption budgets: staying up while things change

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

Readiness decides traffic, liveness decides restarts, startup gives time to start. A PodDisruptionBudget tells drains how many pods may be away at once. Written well they keep the app up through every node upgrade; written badly they kill a healthy app or block the upgrade.

Remember

Readiness fails: out of the Service's endpoints, not restarted.
Liveness fails failureThreshold times: the kubelet kills and restarts the container (same pod, RESTARTS +1).
Startup: holds the other two until it passes; failureThreshold × periodSeconds is the start budget.
Liveness checks the process only, never a database or another service.
kubectl drain evicts through the eviction API, which respects budgets. A dying node is involuntary: no budget helps.
maxUnavailable: 1 lets a drain through; minAvailable equal to the replica count blocks it for ever.
Spread: topologySpreadConstraints on kubernetes.io/hostname. Nothing moves pods back after a node returns.

Pitfalls

minAvailable: 2 with 2 replicas: every eviction fails and the drain retries without end.
Getting a stuck drain through with kubectl delete pod: it bypasses the budget, the outage the budget was meant to stop.
A long initialDelaySeconds on liveness instead of a startup probe.

Commands

`kubectl drain NODE --ignore-daemonsets --delete-emptydir-data`	Evict a node's pods, respecting budgets
`kubectl uncordon NODE`	Schedulable again
`kubectl -n NS get poddisruptionbudget`	ALLOWED DISRUPTIONS
`kubectl -n NS rollout restart deployment/NAME`	Replace every pod (respreads them)

apps/checkout/pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
  namespace: checkout-dev
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout

Test yourself

Answer in your head first, then open the answer.

A pod's readiness probe fails. What does Kubernetes do?

It stops sending the Service's traffic to that pod and leaves it running.

Why should liveness not check the database?

A slow database would restart every pod at once, turning a slowdown into an outage.

2 replicas and minAvailable: 2: what happens on a drain?

The eviction is refused and the drain retries without end.

Who changes the budget when a drain hangs?

The team that owns it; the platform team calls them rather than editing it.

Where module 04 leaves you: checkout has startup, readiness and liveness probes, a spread constraint and a PodDisruptionBudget checkout with maxUnavailable: 1. Both workers are schedulable again and the pods are on different nodes.