Module 07 · Share the cluster · 60 min · You are: Platform and product

Debugging labs: five ways a pod does not run

Product teams come to the platform team with one sentence: "my pod is not running". There are only a handful of stages a pod can be stuck at, and each has exactly one place where the reason is written. Guessing is slow; walking the stages is fast. Try each lab before you open the hints: the skill is knowing where to look next.

Open this module in the interactive course to track progress.

Concepts

Find the stage first

No pod at all: admission said no, look at the ReplicaSet. Pending: the scheduler, look at the pod's events. CreateContainerConfigError or ErrImagePull: the kubelet or the registry, pod events. CrashLoopBackOff: the process, its previous logs. Restarting while healthy: the probes, events.

Events expire

Events are kept for about an hour. Look while it is failing, or you will find a healthy-looking pod with a restart count and no story.

--previous

After a crash, kubectl logs shows the new container, which has not failed yet. The reason is in the one before: kubectl logs --previous.

CrashLoopBackOff is not a cause

It means "it exited, and I wait longer each time before I try again". The cause is the exit: the code, the last log line, or OOMKilled.

Old pods are your safety net

A rolling update keeps the working pods until new ones are ready. Most of these failures leave the site up, which is why they are found late.

Symptom to cause

The full table, with usual causes, is docs/debugging.md in the reference repo.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• docs/debugging.md (reference, your repo): Symptom, stage, who refused, where to read, usual causes. Keep it open during the labs.
  1. One row per stage L5–13

Steps

1. Lab 1: CrashLoopBackOff (Debug lab)

team-checkout wants debug logging and adds a flag from memory.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, above ports:

          command: ["./podinfo", "--port=9898", "--log-level=debug"]

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: debug logging" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

Wait 45 seconds, then look:

sleep 45; kubectl -n checkout-dev get pods

What you see.

Two old pods Running, one new pod in CrashLoopBackOff (or Error) with RESTARTS at 2 or more. The site answers. kubectl -n checkout-dev logs <new pod> may show nothing useful: you are looking at a container that has only just started.

Fix it yourself. Quote the line the process printed before it died, and its exit code. Then fix it through Git.

cd ~/platformengr/k8s-platform && git revert --no-edit HEAD && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

2. Lab 2: pods that never appear, and a pod that stays Pending (Debug lab)

Two changes by team-checkout, one after the other. Both end with fewer pods than asked for. Notice where each one stops. (a) A background worker, as a second Deployment:

Create ~/platformengr/k8s-platform/apps/checkout/worker.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-worker
  namespace: checkout-dev
spec:
  replicas: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout-worker
  template:
    metadata:
      labels:
        app.kubernetes.io/name: checkout-worker
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 100
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: checkout
      containers:
        - name: worker
          image: ghcr.io/stefanprodan/podinfo:6.17.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              memory: 128Mi

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: background worker" && git push -q

kubectl apply -f apps/checkout/worker.yaml

Look at the Deployments:

kubectl -n checkout-dev get deployment

(b) Once you have found and fixed (a): a node selector copied from the manifest of a cloud cluster.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the pod spec:, above securityContext:

      nodeSelector:
        disktype: ssd

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: run on SSD nodes" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev get pods

What you see.

(a) checkout-worker shows 3/4. kubectl -n checkout-dev get pods lists three worker pods and no fourth: nothing Pending, nothing failing.

(b) The two old checkout pods, and one new pod in Pending for as long as you wait.

Fix it yourself. For each: say who refused (admission or the scheduler), where you read it, and quote the message. Then get back to two healthy checkout pods and no worker.

cd ~/platformengr/k8s-platform && git rm -q apps/checkout/worker.yaml && git commit -qm "checkout: no worker for now" && git push -q

kubectl -n checkout-dev delete deployment checkout-worker

-       nodeSelector:
-         disktype: ssd

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: no node selector" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

3. Lab 3: a probe that kills a healthy app (Debug lab)

team-checkout wants the liveness probe to "really test the app", and points it at an endpoint that does some work. Here that is podinfo's /delay/3, which answers after three seconds: a stand-in for a health check that calls the database.

Change in ~/platformengr/k8s-platform/apps/checkout/deployment.yaml

-           livenessProbe:
-             httpGet:
-               path: /healthz
-               port: http
-             periodSeconds: 10
-             timeoutSeconds: 2
+           livenessProbe:
+             httpGet:
+               path: /delay/3
+               port: http
+             periodSeconds: 10
+             timeoutSeconds: 1

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: liveness really tests the app" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

The rollout succeeded. Wait a minute and a half:

sleep 90; kubectl -n checkout-dev get pods

What you see.

Expected: two pods, RESTARTS 0. You get: both pods restarting, RESTARTS 2 and climbing, later CrashLoopBackOff. kubectl -n checkout-dev logs <pod> --previous shows a normal start and a clean shutdown: no error, no panic. The app says it is fine.

Fix it yourself. Find what is stopping the containers, quote the event, and say why the rollout itself did not catch it. Fix through Git.

cd ~/platformengr/k8s-platform && git revert --no-edit HEAD && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

4. Lab 4: an image that will not pull (Debug lab)

team-checkout promotes a version that was never built.

Change in ~/platformengr/k8s-platform/apps/checkout/deployment.yaml

-           image: ghcr.io/stefanprodan/podinfo:6.17.0
+           image: ghcr.io/stefanprodan/podinfo:6.17.99

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: 6.17.99" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

sleep 30; kubectl -n checkout-dev get pods

What you see.

Two old pods Running, one new pod ErrImagePull, then ImagePullBackOff. RESTARTS is 0: the container never started, so there are no logs at all.

Fix it yourself. Quote what the registry answered. Say what else produces the same status. Fix through Git.

cd ~/platformengr/k8s-platform && git revert --no-edit HEAD && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

5. Lab 5: a Deployment with no new pods (Debug lab)

To look around inside the pod, team-checkout adds a helper container with a shell.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under containers:, after the whole checkout container (just above volumes:)

        - name: debug
          image: busybox:1.38.0
          command: ["sleep", "86400"]

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: debug container" && git push -q

Apply it, and read everything it prints:

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=30s

What you see.

The apply says deployment.apps/checkout configured. The rollout times out at 0 out of 2 new replicas have been updated. kubectl -n checkout-dev get pods shows only the two old pods: nothing new, nothing Pending, nothing crashing.

Fix it yourself. Find who refused the new pods and why. The cluster told you in two places: name both. Fix through Git, then get a shell next to checkout the supported way.

cd ~/platformengr/k8s-platform && git revert --no-edit HEAD && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

kubectl -n checkout-dev debug -it "$(kubectl -n checkout-dev get pod -l app.kubernetes.io/name=checkout -o name | head -n 1)" --image=busybox:1.38.0 --target=checkout --profile=restricted

6. Write one runbook entry

For the lab that surprised you most, write the entry you would want at 2 a.m. In your own words: what you saw, where the reason was, the cause, the fix, and who owns it.

Done when:

• docs/runbook.md has at least one entry with five parts: symptom, where the reason was, cause, fix, owner
• The symptom is what kubectl get showed, not the cause

Create ~/platformengr/k8s-platform/docs/runbook.md

# Runbook

## <the symptom, in one line>

- **Symptom:** what `kubectl get` showed.
- **Where the reason was:** the command, and the line that explained it.
- **Cause:**
- **Fix:**
- **Who owns it:**

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "docs: first runbook entry" && git push -q

Simplified for the course

You fixed everything by reverting. Real incidents also need: tell people, restore service first, write it up. The five questions under each lab are the write-up.

Events live about an hour on most clusters. Production platforms ship them, with logs, to a central store, so the story of a 3 a.m. failure is still there at 9.

Quiz

1. A Deployment shows 2/3 and there is no third pod in any state. Where is the reason?
   • In the logs of the third pod's previous container, with kubectl logs --previous
   • In the events of the ReplicaSet, which tried to create the pod and was refused at admission
   • In the scheduler's events on the third pod, which is waiting in Pending
2. A pod is in CrashLoopBackOff and kubectl logs shows a clean start. What next?
   • Read kubectl logs --previous and the Last State in kubectl describe pod: the failed container is the one before
   • Delete the pod, because the back-off timer only resets when a new pod is created
   • Raise the memory limit, because CrashLoopBackOff means the container ran out of memory
3. Both pods restart every half minute, their logs show a clean shutdown, and the rollout had succeeded. What stopped them?
   • The ReplicaSet, which replaces pods that run longer than their progress deadline
   • The scheduler, which moves pods to emptier nodes when the cluster is busy
   • The kubelet, because the liveness probe failed: look at the Unhealthy events

Field note

When I am handed a broken pod I say the stage out loud before I touch anything: not created, not scheduled, not started, crashing, not ready. It sounds silly, and it stops me reading the logs of a container that never ran.

Report a problem with this module