Cheat notes · Module 08 · Operate and grow

Operate the fleet: failure drills

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

You learn a platform by watching it fail safely. For each failure answer five questions: what failed, how it was detected, who responds, how it is corrected, how it is prevented. Most failures that restore desired state need no human.

Remember

Detection: CI for proposed changes, Argo CD status for drift and sync failures, Kyverno at admission, terraform plan for infrastructure drift, logs and health for the rest.
Ownership: product failures go to product teams, platform failures to the platform team. Mixing them up is the most expensive incident pattern.
Correction: a fix that changes desired state is a pull request. A fix that restores it (self-heal, retries, rebuilds) needs nobody.
The five drills: a deleted namespace heals itself (retry); hand-scaled prod goes back to Git; a team cannot change its baseline (Kyverno, RBAC); an unreachable cluster shows Unknown and recovers; a deleted hub is rebuilt by terraform apply.
The hub is stateless: desired state lives in Git, registrations in Terraform. Nothing about the fleet lives only on the hub.
Prevention: each drill ends with the control that stops it. If there is none, that is the follow-up issue.

Pitfalls

Treating Unknown on a whole cluster's Applications as a Git problem: Argo CD cannot reach the cluster, so it cannot compare.
Never rehearsing the hub rebuild. The team that has done it on a quiet day handles the real one calmly.
Skipping the write-up. The incident note, written while you remember what you saw first, is what makes the operating model real.

Commands

`kubectl --context kind-prod -n checkout delete networkpolicy default-deny-ingress --as=dev-user --as-group=team-checkout`	Act as the team: refused by `protect-platform-baseline`
`docker stop dev-control-plane`	Take a spoke offline (then `docker start`)
`kind delete cluster --name platform`	Lose the hub
`cd infrastructure/kind && terraform apply`	Rebuild it from code; Argo CD re-adopts every Application

Test yourself

Answer in your head first, then open the answer.

Who responds when a product's pod uses :latest and is rejected in prod?

The product team. The platform provided the guardrail and the message.

What does Argo CD need to rebuild the fleet after losing the hub?

Git and the cluster registrations, which Terraform recreates.

Why is the namespace a separate Application from the product workload?

Different owners and lifecycles: the baseline must exist and stay protected whatever the product config says.

Where module 08 leaves you: Everything is back to normal: the fleet healed itself or was rebuilt from code. You have an incident note in your repo, and you know who responds to each kind of failure.