Cheat notes · Module 09 · Operate
Capstone: the failed-release drill
A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.
The idea
A check only catches what it measures. A slow release passes the success-rate check while users wait, so the team rolls it back fast, adds the missing latency check, proves it catches the same release, writes the incident, and puts the next app on the same path. That is the loop: every failure leaves the platform with one more check.
Remember
- Roll back first, investigate second: revert in Git, then
promote --fullif the steps only prolong the incident. promote --fullskips the remaining steps and analysis. Only for a version that already ran in prod.- The latency template: p95 of
http_request_duration_seconds_bucketfor the new hash, at most 0.5 s. - Prove the check: ship the slow release again and watch it abort on
latency, then revert. - Same path for catalog: the chart already supports it; values in
checkout-configare all it takes.
Pitfalls
promote --fullon the new, unproven version: it skips every check you built.- A latency limit tighter than the app's normal p95: every release aborts, and the team learns to ignore the check.
Commands
git revert --no-edit HEAD && git push -q | Back to the last good version |
kubectl argo rollouts promote --full checkout -n checkout-prod | Skip the steps for a known-good version |
kubectl get clusteranalysistemplates | success-rate and latency |
kubectl -n checkout-prod get analysisrun | Which metric failed |
argocd-platform platform/rollouts/latency.yaml (excerpt)
metrics:
- name: p95-latency
interval: 30s
count: 6
failureLimit: 1
successCondition: result[0] <= 0.5 # seconds
provider:
prometheus:
query: |
histogram_quantile(0.95, sum by (le) (rate(
http_request_duration_seconds_bucket{..., rollouts_pod_template_hash="{{args.new-hash}}"}[1m])))
Test yourself
Answer in your head first, then open the answer.
Why did the slow release pass?
The only check was success rate; slow answers are still successful answers.
When is promote --full the right tool?
To roll forward to a version that already ran in prod, when the steps only prolong the incident.
What should every release incident leave behind?
A note with the five answers, and a check that catches the same failure next time.
Where module 09 leaves you: Prod is judged by success-rate and latency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.