Module 05 · Releases judged by metrics · 35 min · You are: Product team
Canary with automatic rollback in prod
In this module
- You learn: Background analysis during a canary,
startingStep, and why a canary by pod count is coarse. - You do: Give prod four pods and canary steps of 25, 50 and 75%, run the platform's success-rate check from the first pause, ship 6.11.0, then ship a bad release and watch it roll back, and write the first incident note.
- You end with: Prod runs 6.11.0 on 4 pods as a canary judged by
success-rate. A bad release aborts at 25% without a person, andcheckout-config/docs/incidents/has one note.
Blue-green tests a copy; a canary tests the real thing with real users, a few at a time. The analysis runs in the background while the steps go on, and stops everything as soon as the new pods do worse than the limit. In prod that is the difference between "a quarter of users saw errors for a minute" and "everyone did until someone noticed".
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Background analysis
canary.analysis: one AnalysisRun for the whole release, running while the steps go on. A failure aborts at whatever step it is.- startingStep
- The step index (from 0) where the background analysis starts. Here 1: the first pause, once the canary has pods.
- Weight by pods
- Without traffic routing, 25% of 4 pods is 1 pod. 10% of 4 pods is also 1 pod: the weight is only as fine as the replica count.
- Incident note
- One file per failed release with the five answers. The analysis caught it; the note makes sure the team learns from it.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
checkout-config/helm/stages/prod-05-canary.yaml(reference): Prod's values after this module.checkout-app/chart/templates/rollout.yaml(reference): How the chart writes the analysis.- Only rendered when values name templates L60–63
checkout-config/docs/incidents/2026-11-15-random-errors.md(reference): An example incident note.- The five answers, one line each L3–11
Steps
1. Make prod a canary
Four pods, three steps, and the platform's check. Changing the strategy and the replica count releases nothing.
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- replicas: 2
+ replicas: 4 # four pods: one canary pod is 25%
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
strategy: canary
canary:
steps:
- setWeight: 25 # one pod of four
- pause: { duration: 2m } # analysis watches while it waits
- setWeight: 50
- pause: { duration: 2m }
- setWeight: 75
- pause: { duration: 1m }
analysis:
templates: [success-rate]
startingStep: 1 # from the first pause on
git add -A && git commit -qm "checkout prod: canary judged by success-rate" && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300
Expect Step: 6/6, four pods, ✔ Healthy:
kubectl argo rollouts get rollout checkout -n checkout-prod
2. A good release walks through alone
Ship 6.11.0 to prod, the version dev already runs. Nobody needs to promote: the pauses are timed and the analysis is happy.
In a second terminal, watch:
kubectl argo rollouts get rollout checkout -n checkout-prod --watch
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- tag: "6.7.0"
+ tag: "6.11.0"
git add -A && git commit -qm "checkout prod: 6.11.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
In the watch: SetWeight: 25, 1 canary pod and 3 stable, an AnalysisRun ◌ Running from the first pause, then 50% and 75%, and after about six minutes ✔ Healthy with 4 pods on 6.11.0. Then the run:
kubectl -n checkout-prod get analysisrun
3. A bad release rolls itself back
The same bad release as in dev, now in prod. Nobody is watching. This is what the setup is for.
cd ~/work/checkout-config && git switch -q main && git pull -q
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
faults:
randomError: true # a bad release, on purpose
git add -A && git commit -qm "checkout prod: random errors (a bad release)" && git push -q
argocd app get checkout-prod --refresh >/dev/null
Two to three minutes later, expect ✖ Degraded, RolloutAborted with the failed analysis named in the message, and all four pods back on the stable revision:
kubectl argo rollouts get rollout checkout -n checkout-prod
How many requests failed while it ran? podinfo counted them. Expect a small number next to the thousands of successes:
curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (status) (increase(http_requests_total{namespace="checkout-prod"}[10m]))' | jq -r '.data.result[] | "\(.metric) \(.value[1])"'
Revert in Git, so Argo CD shows Healthy again:
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300
4. Write the incident note
Copy the example and make it true for your release: your times, your numbers from the previous step.
mkdir -p docs/incidents
cp ~/work/progressive-delivery/checkout-config/docs/incidents/2026-11-15-random-errors.md docs/incidents/$(date +%F)-random-errors.md
Edit the file in your editor: what failed, how it was detected, who responds, how it was corrected, how it is prevented. Then commit it:
git add -A && git commit -qm "Incident: random errors in prod, aborted by analysis" && git push -q
Simplified for the course
Timed pauses of one or two minutes keep the course moving. In production, pick pause lengths from your traffic: long enough for the analysis to see a few hundred requests to the canary at each step.
Break it on purpose: The analysis that never ran
A teammate adds the analysis to prod's values by copying from dev, and keeps dev's indentation. The YAML is valid. The review passes.
You practise: checking that a safety net is actually there, from the rendered manifest and the Rollout, not from the values file.
1. Break it
cd ~/work/checkout-config && git switch -q main && git pull -q
Move the analysis: block out from under canary: to the top level (no indentation):
Change in ~/work/checkout-config/helm/prod.yaml
- analysis:
- templates: [success-rate]
- startingStep: 1 # from the first pause on
+ analysis:
+ templates: [success-rate]
+ startingStep: 1 # from the first pause on
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
faults:
randomError: true
git add -A && git commit -qm "checkout prod: analysis from dev; random errors" && git push -q
argocd app get checkout-prod --refresh >/dev/null
2. What you see
Expected: the bad release aborted at 25%, as before. You get: it walks through 25, 50 and 75% and finishes ✔ Healthy. Prod now answers one request in three with an error. No AnalysisRun was created for this release.
Look at the Rollout's strategy in the cluster and at what the chart renders from prod's values.
3. Fix it yourself
Find why nothing judged the release, fix the values, and prove the check is back by shipping the bad release again and watching it abort. Then revert the bad release. Done when prod runs without random errors, Healthy, and the newest AnalysisRun for the bad release is Failed.
Hint 1
kubectl -n checkout-prod get rollout checkout -o yaml shows strategy.canary without analysis. The chart only reads canary.analysis.
Hint 2
Indent analysis: back under canary: (two spaces). To release the bad version again after fixing it, the pod template must change: a fresh faults change or a retry will do.
The fix
First stop the damage: revert the commit, which also puts analysis: back where it was. Prod goes back to a version without random errors, through the canary steps:
cd ~/work/checkout-config && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300
Now prove the check works: ship the bad release with the analysis in the right place. Expect it to abort at 25%:
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
faults:
randomError: true
git add -A && git commit -qm "checkout prod: random errors (prove the check)" && git push -q
argocd app get checkout-prod --refresh >/dev/null
kubectl -n checkout-prod get analysisrun
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
After you fixed it: the five questions
- What failed?
- The release check was not part of the Rollout: the values put it where the chart does not look.
- How was it detected?
- Late: by users or by the error rate in Prometheus. The tools showed no AnalysisRun, which nobody checked.
- Who responds?
- team-checkout, who own prod's values.
- How is it corrected?
- The right indentation; then a fresh bad release to prove the check catches it.
- How is it prevented?
- A values schema in the chart (
values.schema.jsonwithadditionalProperties: falseat the top) rejects an unknown top-levelanalysiskey at render time.
Quiz
- With 4 pods and no traffic routing, what does
setWeight: 10give you?- 10% of the requests, measured by the mesh
- One canary pod: about 25% of the requests
- No canary pod until the weight reaches 25
- The background analysis fails at the 50% step. What happens?
- The release finishes the 50% step, then pauses
- The release aborts: all traffic goes back to the stable pods
- The analysis restarts from step 0
- Why
startingStep: 1and not 0?- Step 0 is reserved for Argo CD
- At step 0 the canary pod is still starting; from the first pause it has traffic to measure
- Analysis cannot start before a pause
Answers
- One canary pod: about 25% of the requests Without traffic routing the weight is reached by pod count, rounded up.
- The release aborts: all traffic goes back to the stable pods A failed background analysis aborts at once, whatever the step.
- At step 0 the canary pod is still starting; from the first pause it has traffic to measure Measure once the canary is serving, or the first measurements have no data.
Field note
Every team I have seen lose trust in automatic rollback lost it the same way: a check that was supposed to be there was not, and a bad release went out "even though we have canaries". Look for the AnalysisRun on every release until it is a habit.