Module 09 · Operate · 45 min · You are: Platform and product
Capstone: the failed-release drill
In this module
- You learn: That a release check only sees what it measures, how to roll back a completed release quickly, and how to turn an incident into a check.
- You do: Ship a slow release to prod, notice it from latency, roll it back, write a latency ClusterAnalysisTemplate, add it to prod, re-ship the slow release and watch it abort, write the incident, and give catalog the same release setup.
- You end with: Prod is judged by
success-rateandlatency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.
Every automatic rollback setup fails the same way once: a release goes wrong in a way nobody measured. The skill is not to have the perfect check on day one; it is to notice, recover quickly, and add the check so the same failure never needs a person again.
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Measure what users feel
- Errors are one signal. Latency is the other users notice first. The RED method: rate, errors, duration.
- A completed release
- Once a Rollout is at 100%, abort does nothing: the new version is stable. Going back is a revert in Git, which is a new release.
- promote-full
- Skip all remaining steps and analysis. Right for a rollback to a version that already ran in prod; wrong for anything new.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
platform/rollouts/latency.yaml(reference): One answer to the drill: a p95 latency check.checkout-config/helm/prod.yaml(reference): Prod after the drill.- Both checks on every release L37
Steps
1. The slow release (Challenge)
A teammate ships 6.14.0 to prod with a "performance experiment" that makes every request wait between 300 and 900 ms. Ship it as they did, handle the release as team-checkout would, and decide when you notice.
Done when:
- You saw the release pass
success-rate(the AnalysisRun saysSuccessful, or is still running with every measurement passing) - You measured what users get:
curl -s -o /dev/null -w "%{time_total}"through the gateway, with and without the header route, before and after promotion - You decided, and wrote down when and why
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- tag: "6.13.0"
+ tag: "6.14.0"
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
faults:
randomDelay: true # the "performance experiment"
delayMinMs: 300
delayMaxMs: 900
git add -A && git commit -qm "checkout prod: 6.14.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
Hint 1
At the header-route pause, time a request with x-canary: true and one without. Compare.
Hint 2
The success rate stays at 100%: slow answers are still answers.
2. Roll it back fast (Challenge)
Users are on the slow version (or you aborted it in time). Get prod back to 6.13.0 quickly, through Git, without skipping checks for anything new.
Done when:
- Prod runs 6.13.0 on every pod,
Healthyin Argo CD - Git says 6.13.0 with no
faults - If the slow release had completed: you used a revert and
promote --full(or thepromote-fullaction as alice), and you can say why that is safe here and nowhere else
Hint 1
If it is still running: abort, then revert. If it completed: abort does nothing.
Hint 2
A revert of a completed release goes through every step again, pauses included. kubectl argo rollouts promote --full checkout -n checkout-prod skips them: fine for a version that ran in prod an hour ago.
One solution
cd ~/work/checkout-config && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
kubectl argo rollouts promote --full checkout -n checkout-prod
argocd app wait checkout-prod --sync --health --timeout 300
3. Add the missing check (Challenge)
As the platform team, write a second shared check, latency, that fails a release whose new pods have a 95th-percentile request duration above 0.5 seconds. Then, as team-checkout, use it in prod.
Done when:
rollouts/latency.yamlin your platform repo: aClusterAnalysisTemplatenamedlatencywith the same three arguments assuccess-rate, queryinghttp_request_duration_seconds_bucketfor the new pods only- The query returns a number for the running prod revision in Prometheus (try it with the real hash first)
helm/prod.yamlliststemplates: [success-rate, latency]
Hint 1
Start from a copy of success-rate.yaml. histogram_quantile(0.95, sum by (le) (rate(..._bucket{...}[1m]))) gives the p95 in seconds.
Hint 2
The reference has one answer in platform/rollouts/latency.yaml.
One solution
cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/progressive-delivery/platform/rollouts/latency.yaml rollouts/
git add -A && git commit -qm "Shared release check: latency (p95 under 0.5 s)" && git push -q
argocd app get rollouts-templates --refresh >/dev/null
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- templates: [success-rate]
- startingStep: 4 # once real users get the canary
+ templates: [success-rate, latency] # module 09: latency added after the drill
+ startingStep: 4 # once real users get the canary
git add -A && git commit -qm "checkout prod: judged by latency too" && git push -q
argocd app get checkout-prod --refresh >/dev/null
4. Prove it catches the next one (Challenge)
Ship the slow release again. This time nobody should need to notice.
Done when:
- At 5% the background analysis fails on
p95-latencyand the release aborts without a person - The release log shows an
abortedline for checkout-prod with image 6.14.0 - You reverted the slow release; prod runs 6.13.0,
Healthy
Hint 1
Undo your revert from the rollback (git revert of it), resume at the header-route pause, and watch.
Hint 2
The AnalysisRun lists two metrics; one Successful measurement after another for success-rate, failures for p95-latency.
5. Write the incident (Challenge)
One note in checkout-config/docs/incidents/ with the five answers, as in module 05, for the slow release.
Done when:
- What failed: the version and what users felt, with numbers
- How it was detected the first time (a person) and the second time (the check)
- Who responds, how it was corrected, how it is prevented (the new check, and who owns it)
6. Put catalog on a canary (Challenge)
team-catalog (from the Argo CD capstone) uses the same chart. Give it what checkout has, without copying commands from this course.
Done when:
apps/catalog-dev.yamlandapps/catalog-prod.yamluse chartv2.0.0- catalog's prod values: a canary by pod count (or through the gateway) with
success-rateandlatency - One release of catalog in prod walks through its steps, with an AnalysisRun
Successful - team-catalog's account can resume, abort and retry catalog's Rollouts, and nothing more
Hint 1
catalog's namespaces are not in the mesh: use a canary by pod count with at least 4 pods, or the Gateway API with a catalog host (and a team: catalog label on its namespaces, plus the Gateway's allowedRoutes).
Hint 2
The RBAC lines are the checkout ones with role:catalog-team and catalog/*.
7. Tear down
Your three (four) repos keep everything. Recreating the cluster with module 00's rebuild step brings this setup back, Istio and Prometheus included.
Simplified for the course
The latency check uses one limit for everyone. In production, compare the canary with the stable version (the same query for both hashes, a ratio in the condition), so a service that is always at 400 ms is not judged like one at 40 ms.
Quiz
- A release is at 100% and turns out to be bad. What does
kubectl argo rollouts abortdo?- Switches back to the previous version
- Nothing useful: the new version is already the stable one; revert in Git
- Deletes the Rollout
- Why did the slow release pass
success-rate?- Because the analysis had a bug
- Because slow requests still succeed: the check did not measure duration
- Because the load generator times out before 300 ms
- When is
promote --fullthe right tool?- For any urgent release
- To roll back to a version that ran in prod recently, when waiting through the steps only prolongs the incident
- Never: it skips the analysis
Answers
- Nothing useful: the new version is already the stable one; revert in Git Abort goes back to stable. After completion, stable is the new version.
- Because slow requests still succeed: the check did not measure duration A check sees only what it measures.
- To roll back to a version that ran in prod recently, when waiting through the steps only prolongs the incident Skipping checks is safe only for a version that already passed them in prod.
Field note
If you did this without the solutions, you can run progressive delivery for a team: you know where releases go wrong and how to make the next one cheaper. If you needed them, run the drill again next week from a fresh cluster.