PlatformEngr® Progressive delivery course

Cheat notes · Module 05 · Releases judged by metrics

Canary with automatic rollback in prod

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

A canary in prod gives the new version a share of the pods in steps, with background analysis running from the first pause. A good release walks through the steps alone. A bad one fails the check and the Rollout aborts itself, with nobody watching; the team then reverts in Git and writes down what happened.

Remember

  • Four replicas: one canary pod is 25% of the traffic, so the steps are 25, 50 and 75%.
  • Background analysis (canary.analysis) runs alongside the steps from startingStep, and aborts at any failed measurement limit.
  • An abort scales the canary down and leaves stable serving everyone within seconds.
  • Then a revert in checkout-config, so Git asks for what runs. Argo CD shows Healthy again.
  • An incident note in docs/incidents/: what failed, how it was detected, who responds, how it was corrected, how it is prevented.

Pitfalls

  • Moving analysis: to the wrong indentation: the release runs with no check at all, and the bad version goes out.
  • Using retry after an abort without fixing anything. The same release fails the same way.
  • Weights that the replica count cannot make: with 4 pods, 5% is still one pod.

Commands

kubectl argo rollouts get rollout checkout -n checkout-prod --watchWatch the steps and the analysis
kubectl -n checkout-prod get analysisrunWhich run failed
kubectl argo rollouts retry rollout checkout -n checkout-prodStart an aborted release again
git revert --no-edit HEAD && git push -qPut Git back on the version that runs

checkout-config helm/prod.yaml (module 05)

replicas: 4                  # one canary pod is 25%
strategy: canary
canary:
  steps:
    - setWeight: 25
    - pause: { duration: 2m }
    - setWeight: 50
    - pause: { duration: 2m }
    - setWeight: 75
    - pause: { duration: 1m }
  analysis:
    templates: [success-rate]
    startingStep: 1

Test yourself

Answer in your head first, then open the answer.

A canary aborted at 25%. What do users see?

The stable version, within seconds; the canary pods are scaled down.

Why revert in Git after an automatic abort?

Git still asks for the bad version; until it is reverted, Argo CD shows Degraded.

When does background analysis start?

At startingStep, here the first pause, and it runs until the release ends.

Where module 05 leaves you: Prod runs 6.11.0 on 4 pods as a canary judged by success-rate. A bad release aborts at 25% without a person, and checkout-config/docs/incidents/ has one note.