Cheat notes · Module 05 · Releases judged by metrics
Canary with automatic rollback in prod
A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.
The idea
A canary in prod gives the new version a share of the pods in steps, with background analysis running from the first pause. A good release walks through the steps alone. A bad one fails the check and the Rollout aborts itself, with nobody watching; the team then reverts in Git and writes down what happened.
Remember
- Four replicas: one canary pod is 25% of the traffic, so the steps are 25, 50 and 75%.
- Background analysis (
canary.analysis) runs alongside the steps fromstartingStep, and aborts at any failed measurement limit. - An abort scales the canary down and leaves stable serving everyone within seconds.
- Then a revert in
checkout-config, so Git asks for what runs. Argo CD showsHealthyagain. - An incident note in
docs/incidents/: what failed, how it was detected, who responds, how it was corrected, how it is prevented.
Pitfalls
- Moving
analysis:to the wrong indentation: the release runs with no check at all, and the bad version goes out. - Using
retryafter an abort without fixing anything. The same release fails the same way. - Weights that the replica count cannot make: with 4 pods, 5% is still one pod.
Commands
kubectl argo rollouts get rollout checkout -n checkout-prod --watch | Watch the steps and the analysis |
kubectl -n checkout-prod get analysisrun | Which run failed |
kubectl argo rollouts retry rollout checkout -n checkout-prod | Start an aborted release again |
git revert --no-edit HEAD && git push -q | Put Git back on the version that runs |
checkout-config helm/prod.yaml (module 05)
replicas: 4 # one canary pod is 25%
strategy: canary
canary:
steps:
- setWeight: 25
- pause: { duration: 2m }
- setWeight: 50
- pause: { duration: 2m }
- setWeight: 75
- pause: { duration: 1m }
analysis:
templates: [success-rate]
startingStep: 1
Test yourself
Answer in your head first, then open the answer.
A canary aborted at 25%. What do users see?
The stable version, within seconds; the canary pods are scaled down.
Why revert in Git after an automatic abort?
Git still asks for the bad version; until it is reverted, Argo CD shows Degraded.
When does background analysis start?
At startingStep, here the first pause, and it runs until the release ends.
Where module 05 leaves you: Prod runs 6.11.0 on 4 pods as a canary judged by success-rate. A bad release aborts at 25% without a person, and checkout-config/docs/incidents/ has one note.