PlatformEngr® Progressive delivery course

Cheat notes · Progressive delivery

Progressive delivery: cheat notes

Every module on one page, cut to what you need to remember: the idea, the commands, the pitfalls and a few questions to test yourself. Made for review after you have done the course, not instead of it. Back to the course.

00 Orientation: deploy is not release

The idea

Deploying is not releasing. Argo CD makes the cluster match Git, and a Deployment then replaces every pod as soon as each one is ready, so every user meets a bad version at once. Progressive delivery runs the new version first and gives it users a share at a time, while metrics or a person decide whether it goes on.

Remember

  • Deploy: new pods run. Release: users reach them. Progressive delivery puts time and a decision between the two.
  • Blue-green: a full copy of the new version behind a preview, then one switch for everyone. Fast to undo, double the pods for a while.
  • Canary: a share of the users first, more in steps, judged at each step. Fewer extra pods, slower to finish.
  • Owners: the platform team runs the controller, Prometheus, the mesh and the shared checks. The product team owns its release steps in checkout-config.
  • Same setup as the Argo CD course: kind cluster, argocd-platform, checkout-app, checkout-config, alice for the platform.

Pitfalls

  • Starting on a broken Argo CD platform. Fix or rebuild it first: every module assumes it is Synced and Healthy.
  • Thinking a readiness probe judges a release. It only says the pod answers, not that its answers are right.

Commands

kubectl argo rollouts versionIs the plugin installed?
argocd app listEvery Application, Synced and Healthy before you start
kubectl -n checkout-dev get deployment checkout -o jsonpath='{.spec.strategy}'How a release goes out today (RollingUpdate)

Test yourself

Answer in your head first, then open the answer.

What is the difference between deploy and release?

Deploy runs the new pods; release sends users to them. Progressive delivery separates the two.

When does blue-green fit better than canary?

When you want one clean switch and a fast way back, and can afford a full second copy for a while.

Who owns the release steps of checkout?

The product team, in checkout-config. The platform team owns the controller and the shared checks.

Where module 00 leaves you: Nothing new is installed. Your three repos and the cluster are as the Argo CD course left them, every Application is Synced and Healthy, and the reference repo is cloned next to your repos.

01 Install Argo Rollouts from Git

The idea

Argo Rollouts is a controller with its own resource, the Rollout: a Deployment's pod template plus a strategy. It is installed like every other add-on in the platform: one Application in apps/, a pinned chart version and values in Git, never a kubectl apply of a release URL.

Remember

  • Pinned: chart argo-rollouts 2.43.12, controller v1.10.2. Upgrades are a pull request changing that line.
  • ServerSideApply: the Rollouts CRDs are too large for a client-side apply's annotation.
  • Sync wave -1: the CRDs exist before any app that uses a Rollout.
  • Dashboard read-only: it shows every Rollout; promote and abort go through Argo CD or the CLI, so they leave a trace.
  • kubectl plugin: kubectl-argo-rollouts on your PATH gives kubectl argo rollouts ....

Pitfalls

  • Installing from the latest manifest URL. The version changes under you; pin the chart.
  • A client-side sync of the CRDs fails with metadata.annotations: Too long. Use ServerSideApply=true.

Commands

argocd app wait argo-rollouts --sync --healthWait for the controller
kubectl get crd -o name | grep argoproj.ioRollout, AnalysisTemplate, AnalysisRun and the rest
kubectl argo rollouts list rollouts -AEvery Rollout (none yet)
kubectl -n argo-rollouts port-forward svc/argo-rollouts-dashboard 3100:3100The dashboard at localhost:3100/rollouts

Test yourself

Answer in your head first, then open the answer.

Why does the Rollouts Application run in wave -1?

So its CRDs exist before any app that contains a Rollout is synced.

Why is the dashboard read-only?

Releases are changed through Argo CD or the CLI, which have RBAC and leave a trace.

How do you upgrade Argo Rollouts?

A pull request that changes the pinned chart version in its Application.

Where module 01 leaves you: Argo Rollouts v1.10.2 runs in the argo-rollouts namespace, installed from your platform repo. Nothing uses it yet: checkout is still a Deployment.

02 From Deployment to Rollout

The idea

A Rollout replaces the Deployment with the same pod template, so the migration is a chart release: v2.0.0 renders a Rollout instead of a Deployment, and Argo CD prunes the old one. From then on, a new image tag in checkout-config starts a release that follows the Rollout's steps instead of replacing every pod.

Remember

  • Chart v2.0.0: templates/rollout.yaml replaces deployment.yaml; the HPA now targets kind: Rollout.
  • Default strategy: a canary of setWeight: 50 then pause: {}, which waits for a person.
  • Argo CD health: a paused Rollout is Suspended, an aborted one Degraded, a finished one Healthy.
  • Promote leaves a pause. Abort goes back to the stable version, but Git still asks for the new one.
  • After an abort, revert in Git. Until then the app stays Degraded, because Git and the cluster disagree on purpose.

Pitfalls

  • Leaving a Rollout aborted without a revert. Argo CD shows Degraded until Git matches what runs.
  • Reading setWeight: 50 with one replica as half the traffic. Without a traffic router the weight is a pod count.

Commands

kubectl argo rollouts get rollout checkout -n checkout-dev --watchSteps, ReplicaSets and status, live
kubectl argo rollouts promote checkout -n checkout-devLeave the current pause
kubectl argo rollouts abort checkout -n checkout-devBack to stable, release stopped
argocd app resources checkout-devRollout in, Deployment gone

checkout-app chart/values.yaml (v2.0.0)

strategy: canary
canary:
  steps:
    - setWeight: 50
    - pause: {}        # a person promotes
  analysis: {}
  trafficRouting: ""

Test yourself

Answer in your head first, then open the answer.

What does Argo CD show for a Rollout waiting at pause: {}?

Suspended health.

You aborted a release. What is left to do?

Revert the tag change in Git, so Git asks for the version that runs.

Without a traffic router, what does setWeight change?

The number of canary pods compared with stable ones.

Where module 02 leaves you: checkout runs as a Rollout in dev and prod from chart v2.0.0. Dev runs 6.8.0 with 2 pods; prod still runs 6.7.0. The Deployment is gone.

03 Blue-green in dev

The idea

Blue-green brings the new version up as a full copy behind a preview Service and host, while every user stays on the old one. A person tries the preview and promotes, and the active Service switches to the new pods in one step. The old pods wait for scaleDownDelaySeconds so the switch can be undone fast.

Remember

  • Two Services: checkout (active) and checkout-preview. The controller sets their selectors to a pod-template hash.
  • Two hosts: checkout-dev.localtest.me (users) and preview-checkout-dev.localtest.me (the next version).
  • autoPromotionEnabled: false: waits for a person. Analysis replaces the person in module 04.
  • scaleDownDelaySeconds: 30: the old pods keep running after the switch, for a fast way back.
  • Cost: two full sets of pods while a release waits.

Pitfalls

  • A typo in the image tag leaves the preview in ImagePullBackOff; users are fine, but the release never becomes promotable. Fix the tag in Git.
  • Writing the Services' selectors by hand. The controller owns them; Argo CD must not fight it.

Commands

curl -sk https://checkout-dev.localtest.me:8443/ | jq -r .versionWhat users get
curl -sk https://preview-checkout-dev.localtest.me:8443/ | jq -r .versionWhat is waiting
kubectl argo rollouts promote checkout -n checkout-devSwitch everyone to the preview

checkout-config helm/dev.yaml (module 03)

strategy: blueGreen
blueGreen:
  autoPromotionEnabled: false    # a person promotes
  scaleDownDelaySeconds: 30
ingress:
  host: checkout-dev.localtest.me
  previewHost: preview-checkout-dev.localtest.me

Test yourself

Answer in your head first, then open the answer.

What decides which pods the active Service selects?

The Rollout controller, through the pod-template hash it writes to the selector.

Why keep the old pods for 30 seconds after the switch?

So an abort right after the switch is instant, with no new pods to start.

What does blue-green cost compared with canary?

A full second copy of the app while a release waits.

Where module 03 leaves you: Dev runs blue-green: 6.9.0 on 2 pods behind checkout-dev.localtest.me, and preview-checkout-dev.localtest.me shows the next version whenever one is waiting.

04 Metrics and analysis templates

The idea

Analysis lets metrics decide instead of a person. The platform team writes the checks once as ClusterAnalysisTemplates; each one asks Prometheus a question about the new pods only, by their pod-template hash, and says what a good answer is. A Rollout runs them and promotes, waits or aborts on their verdict.

Remember

  • Prometheus scrapes every pod with prometheus.io/scrape: "true", keeping the labels app_kubernetes_io_name and rollouts_pod_template_hash.
  • The load generator sends a few requests a second, so a run has data when nobody uses the app.
  • ClusterAnalysisTemplate: shared by every namespace; a Rollout uses it with clusterScope: true.
  • new-hash comes from podTemplateHashValue: Latest: the check measures only the new version.
  • Phases: Successful promotes, Failed aborts, Error (no data, query broken) aborts after consecutiveErrorLimit, Inconclusive pauses.
  • prePromotionAnalysis in blue-green: the preview must pass before the switch.

Pitfalls

  • A query that returns nothing is not a pass: it is an Error, and the release aborts. Check the labels in Prometheus first.
  • Measuring the whole app instead of the new pods. Without the hash, the stable pods hide a bad canary.
  • No traffic, no data. A quiet app needs the load generator or a longer interval.

Commands

kubectl -n monitoring port-forward svc/prometheus-server 9090:80Prometheus at localhost:9090
kubectl get clusteranalysistemplatesThe platform's shared checks
kubectl -n checkout-dev get analysisrunRuns and their phase
kubectl describe analysisrun RUN -n checkout-devEvery measurement and why it passed or failed

argocd-platform platform/rollouts/success-rate.yaml (excerpt)

kind: ClusterAnalysisTemplate
metadata:
  name: success-rate
spec:
  args: [{ name: namespace }, { name: app }, { name: new-hash }]
  metrics:
    - name: success-rate
      interval: 30s
      count: 6
      failureLimit: 1
      successCondition: result[0] >= 0.95
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{..., status=~"2.."}[1m]))
            /
            sum(rate(http_requests_total{...}[1m]))

Test yourself

Answer in your head first, then open the answer.

Why filter on rollouts_pod_template_hash?

So the check measures only the new version, not the stable pods next to it.

The query returns an empty result. What happens?

Each measurement is an error; after the error limit the run is Error and the release aborts.

Why a ClusterAnalysisTemplate instead of an AnalysisTemplate?

One check, written once by the platform team, used by every team's namespace.

Where module 04 leaves you: Prometheus scrapes checkout, a load generator sends it traffic, the success-rate ClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.

05 Canary with automatic rollback in prod

The idea

A canary in prod gives the new version a share of the pods in steps, with background analysis running from the first pause. A good release walks through the steps alone. A bad one fails the check and the Rollout aborts itself, with nobody watching; the team then reverts in Git and writes down what happened.

Remember

  • Four replicas: one canary pod is 25% of the traffic, so the steps are 25, 50 and 75%.
  • Background analysis (canary.analysis) runs alongside the steps from startingStep, and aborts at any failed measurement limit.
  • An abort scales the canary down and leaves stable serving everyone within seconds.
  • Then a revert in checkout-config, so Git asks for what runs. Argo CD shows Healthy again.
  • An incident note in docs/incidents/: what failed, how it was detected, who responds, how it was corrected, how it is prevented.

Pitfalls

  • Moving analysis: to the wrong indentation: the release runs with no check at all, and the bad version goes out.
  • Using retry after an abort without fixing anything. The same release fails the same way.
  • Weights that the replica count cannot make: with 4 pods, 5% is still one pod.

Commands

kubectl argo rollouts get rollout checkout -n checkout-prod --watchWatch the steps and the analysis
kubectl -n checkout-prod get analysisrunWhich run failed
kubectl argo rollouts retry rollout checkout -n checkout-prodStart an aborted release again
git revert --no-edit HEAD && git push -qPut Git back on the version that runs

checkout-config helm/prod.yaml (module 05)

replicas: 4                  # one canary pod is 25%
strategy: canary
canary:
  steps:
    - setWeight: 25
    - pause: { duration: 2m }
    - setWeight: 50
    - pause: { duration: 2m }
    - setWeight: 75
    - pause: { duration: 1m }
  analysis:
    templates: [success-rate]
    startingStep: 1

Test yourself

Answer in your head first, then open the answer.

A canary aborted at 25%. What do users see?

The stable version, within seconds; the canary pods are scaled down.

Why revert in Git after an automatic abort?

Git still asks for the bad version; until it is reverted, Argo CD shows Degraded.

When does background analysis start?

At startingStep, here the first pause, and it runs until the release ends.

Where module 05 leaves you: Prod runs 6.11.0 on 4 pods as a canary judged by success-rate. A bad release aborts at 25% without a person, and checkout-config/docs/incidents/ has one note.

06 Istio: exact weights and header routes

The idea

Without a traffic router, a weight is a pod count. With Istio, the Rollout writes the weights into a VirtualService and the mesh sends exactly that share of requests, however many pods there are. It can also route a test header to the canary before any user gets it, and the sidecars give mutual TLS between pods.

Remember

  • Three charts, pinned: base (CRDs), istiod, gateway (release istio-ingress), Istio 1.30.8, sidecar mode.
  • istio-injection: enabled on checkout-prod and loadgen; existing pods need a restart to get a sidecar.
  • Two Services: checkout (stable) and checkout-canary; the VirtualService route primary has both with weights.
  • setCanaryScale runs one canary pod with no traffic; setHeaderRoute sends x-canary: true to it.
  • Argo CD ignores the weights (ignoreDifferences plus RespectIgnoreDifferences=true), because the controller changes them.
  • PeerAuthentication STRICT in prod: only pods with a sidecar can call checkout.

Pitfalls

  • Argo CD resetting the weights to 100/0 in the middle of a release. Ignore .spec.http[].route[].weight and respect it on sync.
  • STRICT mTLS before the callers have sidecars: the load generator gets connection resets.
  • Starting analysis at the header step: nobody but testers sends requests, so there is no data.

Commands

kubectl argo rollouts restart checkout -n checkout-prodRestart the pods so they get a sidecar
kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[0].route[*].weight}'The weights right now
kubectl -n istio-ingress port-forward svc/istio-ingress 8090:80The mesh's front door on localhost:8090
curl -s -H 'x-canary: true' http://checkout.localtest.me:8090/ | jq -r .versionWhat testers get

checkout-config helm/prod.yaml (module 06, steps)

canary:
  trafficRouting: istio
  steps:
    - setCanaryScale: { replicas: 1 }
    - setHeaderRoute:
        name: canary-header
        match:
          - headerName: x-canary
            headerValue: { exact: "true" }
    - pause: {}
    - setWeight: 5
    - pause: { duration: 2m }
    - setWeight: 25
    - pause: { duration: 2m }
    - setWeight: 50
    - pause: { duration: 2m }
    - setHeaderRoute: { name: canary-header }
  analysis:
    templates: [success-rate]
    startingStep: 4

Test yourself

Answer in your head first, then open the answer.

What does setWeight: 5 mean with Istio?

Exactly 5% of requests go to the canary, whatever the pod count.

How do testers reach a canary before users do?

A header route: requests with x-canary: true go to the canary Service.

Why does the checkout-prod Application ignore VirtualService weights?

The Rollout controller changes them during a release; Argo CD must not reset them.

Where module 06 leaves you: Istio 1.30.8 runs; checkout-prod and the load generator have sidecars. Prod releases through a VirtualService: one canary pod, a tester route on the header x-canary: true, then 5/25/50%. Prod runs 6.12.0 with mTLS STRICT.

07 Gateway API: the lighter option

The idea

A mesh is not the only way to split traffic exactly. The Gateway API is a standard set of routing resources that many ingress controllers implement, Traefik included. The Argo Rollouts Gateway API plugin writes the canary weights into an HTTPRoute, so the edge splits the traffic and no pod needs a sidecar.

Remember

  • Gateway API CRDs v1.5.1 (standard channel), installed from Git before Traefik: the chart does not bundle them.
  • Traefik with providers.kubernetesGateway.enabled, and its Gateway traefik-gateway accepting routes from namespaces labelled team: checkout.
  • The plugin argoproj-labs/gatewayAPI v0.17.0, named in the Rollouts values; the controller loads it at start.
  • HTTPRoute checkout with two backends, stable and canary; the controller sets their weights.
  • Mesh or edge: Istio adds header routes inside the cluster and mTLS; the Gateway API is lighter and only sees traffic that enters through the gateway.

Pitfalls

  • An HTTPRoute that is not Accepted: the Gateway's namespace policy does not admit the route's namespace.
  • Traffic that skips the gateway (pod to pod) is not split. Only requests through the edge follow the weights.

Commands

kubectl get gatewayclass,gateway -ATraefik's GatewayClass and Gateway
kubectl -n argo-rollouts logs deploy/argo-rollouts | grep -i pluginDid the plugin load?
kubectl -n checkout-dev get httproute checkout -o jsonpath='{.spec.rules[0].backendRefs[*].weight}'The weights right now
curl -s http://checkout-dev.localtest.me:8088/ | jq -r .versionWhat users get

checkout-config helm/dev.yaml (module 07)

replicas: 2
strategy: canary
canary:
  trafficRouting: gatewayapi
  steps:
    - setWeight: 20            # 20% even with two pods
    - pause: { duration: 1m }
    - setWeight: 50
    - pause: { duration: 1m }
  analysis:
    templates: [success-rate]
    startingStep: 1
gatewayApi:
  host: checkout-dev.localtest.me

Test yourself

Answer in your head first, then open the answer.

Why install the Gateway API CRDs as their own Application?

Traefik's chart does not include them, and they must exist before the Gateway and any HTTPRoute.

What does the plugin change during a release?

The weights of the stable and canary backends in the HTTPRoute.

When is Istio worth its weight over the Gateway API?

When you need header routes or exact splits inside the cluster, or mTLS between pods.

Where module 07 leaves you: Dev releases through HTTPRoute/checkout on Traefik's traefik-gateway: 20% and 50% steps with two pods, judged by success-rate. Dev is served at http://checkout-dev.localtest.me:8088 and runs 6.12.0.

08 Operate releases with teams

The idea

The product team owns its releases, so it needs the buttons for them and nothing more. Argo CD resource actions give resume, abort and retry on Rollouts under Argo CD's RBAC, with a trace. Notifications tell the team when a release finishes or aborts, a deadline aborts a release that is stuck, and a runbook says who does what.

Remember

  • RBAC: action/argoproj.io/Rollout/resume, abort and retry for role:checkout-team on checkout/*, instead of action/*.
  • argocd app actions run APP resume --kind Rollout is promote through Argo CD; restart is denied for the team.
  • Notifications in the Rollouts values: a webhook service release-log, templates and triggers on-rollout-completed and on-rollout-aborted.
  • Subscriptions are annotations on the Rollout: notifications.argoproj.io/subscribe.on-rollout-aborted.release-log.
  • progressDeadlineAbort: true: a release that makes no progress for the deadline aborts instead of hanging.
  • docs/releases.md: who promotes, who aborts, how to revert, where the checks are.

Pitfalls

  • Giving the team action/*: it can then restart or delete pods of any kind in its apps.
  • A notification that never arrives because the Rollout has no subscription annotation. The trigger alone sends nothing.

Commands

argocd app actions list checkout-prod --kind RolloutWhat the team may run
argocd app actions run checkout-prod resume --kind RolloutPromote through Argo CD
argocd app actions run checkout-prod abort --kind RolloutAbort through Argo CD
kubectl -n release-log logs deploy/release-log -fEvery notification, as it arrives

argocd-platform platform/argocd/values.yaml (RBAC excerpt)

p, role:checkout-team, applications, action/argoproj.io/Rollout/resume, checkout/*, allow
p, role:checkout-team, applications, action/argoproj.io/Rollout/abort, checkout/*, allow
p, role:checkout-team, applications, action/argoproj.io/Rollout/retry, checkout/*, allow

Test yourself

Answer in your head first, then open the answer.

How does the product team promote without kubectl access?

argocd app actions run checkout-prod resume --kind Rollout, allowed by its RBAC role.

What makes a notification go to the release log?

A trigger and template in the Rollouts values, and a subscription annotation on the Rollout.

What does progressDeadlineAbort prevent?

A release that hangs forever half-done; it aborts after the progress deadline.

Where module 08 leaves you: The checkout-dev account can resume, abort and retry checkout's Rollouts and nothing else. Every completed or aborted release writes a line to the release log. Prod runs 6.13.0. docs/releases.md is in your platform repo.

09 Capstone: the failed-release drill

The idea

A check only catches what it measures. A slow release passes the success-rate check while users wait, so the team rolls it back fast, adds the missing latency check, proves it catches the same release, writes the incident, and puts the next app on the same path. That is the loop: every failure leaves the platform with one more check.

Remember

  • Roll back first, investigate second: revert in Git, then promote --full if the steps only prolong the incident.
  • promote --full skips the remaining steps and analysis. Only for a version that already ran in prod.
  • The latency template: p95 of http_request_duration_seconds_bucket for the new hash, at most 0.5 s.
  • Prove the check: ship the slow release again and watch it abort on latency, then revert.
  • Same path for catalog: the chart already supports it; values in checkout-config are all it takes.

Pitfalls

  • promote --full on the new, unproven version: it skips every check you built.
  • A latency limit tighter than the app's normal p95: every release aborts, and the team learns to ignore the check.

Commands

git revert --no-edit HEAD && git push -qBack to the last good version
kubectl argo rollouts promote --full checkout -n checkout-prodSkip the steps for a known-good version
kubectl get clusteranalysistemplatessuccess-rate and latency
kubectl -n checkout-prod get analysisrunWhich metric failed

argocd-platform platform/rollouts/latency.yaml (excerpt)

metrics:
  - name: p95-latency
    interval: 30s
    count: 6
    failureLimit: 1
    successCondition: result[0] <= 0.5      # seconds
    provider:
      prometheus:
        query: |
          histogram_quantile(0.95, sum by (le) (rate(
            http_request_duration_seconds_bucket{..., rollouts_pod_template_hash="{{args.new-hash}}"}[1m])))

Test yourself

Answer in your head first, then open the answer.

Why did the slow release pass?

The only check was success rate; slow answers are still successful answers.

When is promote --full the right tool?

To roll forward to a version that already ran in prod, when the steps only prolong the incident.

What should every release incident leave behind?

A note with the five answers, and a check that catches the same failure next time.

Where module 09 leaves you: Prod is judged by success-rate and latency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.