PlatformEngr® Progressive delivery course

Module 09 · Operate · 45 min · You are: Platform and product

Capstone: the failed-release drill

In this module

  • You learn: That a release check only sees what it measures, how to roll back a completed release quickly, and how to turn an incident into a check.
  • You do: Ship a slow release to prod, notice it from latency, roll it back, write a latency ClusterAnalysisTemplate, add it to prod, re-ship the slow release and watch it abort, write the incident, and give catalog the same release setup.
  • You end with: Prod is judged by success-rate and latency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.

Every automatic rollback setup fails the same way once: a release goes wrong in a way nobody measured. The skill is not to have the perfect check on day one; it is to notice, recover quickly, and add the check so the same failure never needs a person again.

Open this module in the interactive course to track progress and switch tracks.

Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back

Concepts

Measure what users feel
Errors are one signal. Latency is the other users notice first. The RED method: rate, errors, duration.
A completed release
Once a Rollout is at 100%, abort does nothing: the new version is stable. Going back is a revert in Git, which is a new release.
promote-full
Skip all remaining steps and analysis. Right for a rollback to a version that already ran in prod; wrong for anything new.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. The slow release (Challenge)

A teammate ships 6.14.0 to prod with a "performance experiment" that makes every request wait between 300 and 900 ms. Ship it as they did, handle the release as team-checkout would, and decide when you notice.

Done when:

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/prod.yaml

- tag: "6.13.0"
+ tag: "6.14.0"

Add to ~/work/checkout-config/helm/prod.yaml at the end of the file

faults:
  randomDelay: true              # the "performance experiment"
  delayMinMs: 300
  delayMaxMs: 900
git add -A && git commit -qm "checkout prod: 6.14.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
Hint 1

At the header-route pause, time a request with x-canary: true and one without. Compare.

Hint 2

The success rate stays at 100%: slow answers are still answers.

2. Roll it back fast (Challenge)

Users are on the slow version (or you aborted it in time). Get prod back to 6.13.0 quickly, through Git, without skipping checks for anything new.

Done when:

Hint 1

If it is still running: abort, then revert. If it completed: abort does nothing.

Hint 2

A revert of a completed release goes through every step again, pauses included. kubectl argo rollouts promote --full checkout -n checkout-prod skips them: fine for a version that ran in prod an hour ago.

One solution
cd ~/work/checkout-config && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
kubectl argo rollouts promote --full checkout -n checkout-prod
argocd app wait checkout-prod --sync --health --timeout 300

3. Add the missing check (Challenge)

As the platform team, write a second shared check, latency, that fails a release whose new pods have a 95th-percentile request duration above 0.5 seconds. Then, as team-checkout, use it in prod.

Done when:

Hint 1

Start from a copy of success-rate.yaml. histogram_quantile(0.95, sum by (le) (rate(..._bucket{...}[1m]))) gives the p95 in seconds.

Hint 2

The reference has one answer in platform/rollouts/latency.yaml.

One solution
cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/progressive-delivery/platform/rollouts/latency.yaml rollouts/
git add -A && git commit -qm "Shared release check: latency (p95 under 0.5 s)" && git push -q
argocd app get rollouts-templates --refresh >/dev/null
cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/prod.yaml

-     templates: [success-rate]
-     startingStep: 4                          # once real users get the canary
+     templates: [success-rate, latency]       # module 09: latency added after the drill
+     startingStep: 4                          # once real users get the canary
git add -A && git commit -qm "checkout prod: judged by latency too" && git push -q
argocd app get checkout-prod --refresh >/dev/null

4. Prove it catches the next one (Challenge)

Ship the slow release again. This time nobody should need to notice.

Done when:

Hint 1

Undo your revert from the rollback (git revert of it), resume at the header-route pause, and watch.

Hint 2

The AnalysisRun lists two metrics; one Successful measurement after another for success-rate, failures for p95-latency.

5. Write the incident (Challenge)

One note in checkout-config/docs/incidents/ with the five answers, as in module 05, for the slow release.

Done when:

6. Put catalog on a canary (Challenge)

team-catalog (from the Argo CD capstone) uses the same chart. Give it what checkout has, without copying commands from this course.

Done when:

Hint 1

catalog's namespaces are not in the mesh: use a canary by pod count with at least 4 pods, or the Gateway API with a catalog host (and a team: catalog label on its namespaces, plus the Gateway's allowedRoutes).

Hint 2

The RBAC lines are the checkout ones with role:catalog-team and catalog/*.

7. Tear down

Your three (four) repos keep everything. Recreating the cluster with module 00's rebuild step brings this setup back, Istio and Prometheus included.

Simplified for the course

The latency check uses one limit for everyone. In production, compare the canary with the stable version (the same query for both hashes, a ratio in the condition), so a service that is always at 400 ms is not judged like one at 40 ms.

Quiz

  1. A release is at 100% and turns out to be bad. What does kubectl argo rollouts abort do?
    • Switches back to the previous version
    • Nothing useful: the new version is already the stable one; revert in Git
    • Deletes the Rollout
  2. Why did the slow release pass success-rate?
    • Because the analysis had a bug
    • Because slow requests still succeed: the check did not measure duration
    • Because the load generator times out before 300 ms
  3. When is promote --full the right tool?
    • For any urgent release
    • To roll back to a version that ran in prod recently, when waiting through the steps only prolongs the incident
    • Never: it skips the analysis
Answers
  1. Nothing useful: the new version is already the stable one; revert in Git Abort goes back to stable. After completion, stable is the new version.
  2. Because slow requests still succeed: the check did not measure duration A check sees only what it measures.
  3. To roll back to a version that ran in prod recently, when waiting through the steps only prolongs the incident Skipping checks is safe only for a version that already passed them in prod.

Field note

If you did this without the solutions, you can run progressive delivery for a team: you know where releases go wrong and how to make the next one cheaper. If you needed them, run the drill again next week from a fresh cluster.

Report a problem with this module