Cheat notes · Module 07 · Share the cluster

Debugging labs: five ways a pod does not run

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

A pod that is not running is stuck at one stage: not created, not scheduled, not started, crashing, or killed from outside. Say the stage first, then read the one place that explains it.

Remember

No pod at all: admission refused it (quota, LimitRange, Pod Security). Read the ReplicaSet's FailedCreate events.
Pending: the scheduler. Read the pod's events (FailedScheduling).
ErrImagePull, CreateContainerConfigError: the registry or the kubelet. Pod events.
CrashLoopBackOff: the process exited. kubectl logs --previous and Last State.
Restarts with clean logs: something stopped it, usually liveness. Unhealthy and Killing events.
Events last about an hour. Look while it is failing.
Old pods keep serving during a failed rollout, which is why these are found late.

Pitfalls

Reading kubectl logs of a container that just restarted: the failed one is --previous.
Missing the Warning: line that kubectl apply printed: Pod Security told you before the rollout hung.
Deleting a file from Git and forgetting the object: delete both.

Commands

`kubectl -n NS get events --field-selector reason=FailedCreate`	Refused at admission
`kubectl -n NS describe pod POD`	Events, state, last state
`kubectl -n NS logs POD --previous`	Logs of the container that died
`kubectl -n NS get events --field-selector reason=Unhealthy`	Probe failures
`kubectl -n NS debug -it POD --image=busybox:1.38.0 --target=checkout --profile=restricted`	A temporary shell next to a container

Test yourself

Answer in your head first, then open the answer.

2/3 and no third pod in any state: where is the reason?

In the ReplicaSet's events: admission refused the pod.

CrashLoopBackOff and kubectl logs shows a clean start. Next?

kubectl logs --previous and the Last State in describe pod.

Pods restart every half minute with clean shutdowns after a successful rollout. Why?

The liveness probe fails; the kubelet kills them. Look at Unhealthy events.

ImagePullBackOff, message ends in not found. Cause?

The tag does not exist. unauthorized means credentials; no such host means the registry name.

Where module 07 leaves you: checkout is healthy again: 2 pods, the probes and image from module 06, nothing extra. Your repo has docs/runbook.md with one real entry.