PlatformEngr® Progressive delivery course

Cheat notes · Module 09 · Operate

Capstone: the failed-release drill

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

A check only catches what it measures. A slow release passes the success-rate check while users wait, so the team rolls it back fast, adds the missing latency check, proves it catches the same release, writes the incident, and puts the next app on the same path. That is the loop: every failure leaves the platform with one more check.

Remember

  • Roll back first, investigate second: revert in Git, then promote --full if the steps only prolong the incident.
  • promote --full skips the remaining steps and analysis. Only for a version that already ran in prod.
  • The latency template: p95 of http_request_duration_seconds_bucket for the new hash, at most 0.5 s.
  • Prove the check: ship the slow release again and watch it abort on latency, then revert.
  • Same path for catalog: the chart already supports it; values in checkout-config are all it takes.

Pitfalls

  • promote --full on the new, unproven version: it skips every check you built.
  • A latency limit tighter than the app's normal p95: every release aborts, and the team learns to ignore the check.

Commands

git revert --no-edit HEAD && git push -qBack to the last good version
kubectl argo rollouts promote --full checkout -n checkout-prodSkip the steps for a known-good version
kubectl get clusteranalysistemplatessuccess-rate and latency
kubectl -n checkout-prod get analysisrunWhich metric failed

argocd-platform platform/rollouts/latency.yaml (excerpt)

metrics:
  - name: p95-latency
    interval: 30s
    count: 6
    failureLimit: 1
    successCondition: result[0] <= 0.5      # seconds
    provider:
      prometheus:
        query: |
          histogram_quantile(0.95, sum by (le) (rate(
            http_request_duration_seconds_bucket{..., rollouts_pod_template_hash="{{args.new-hash}}"}[1m])))

Test yourself

Answer in your head first, then open the answer.

Why did the slow release pass?

The only check was success rate; slow answers are still successful answers.

When is promote --full the right tool?

To roll forward to a version that already ran in prod, when the steps only prolong the incident.

What should every release incident leave behind?

A note with the five answers, and a check that catches the same failure next time.

Where module 09 leaves you: Prod is judged by success-rate and latency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.