Cheat notes · Progressive delivery
Progressive delivery: cheat notes
Every module on one page, cut to what you need to remember: the idea, the commands, the pitfalls and a few questions to test yourself. Made for review after you have done the course, not instead of it. Back to the course.
00 Orientation: deploy is not release
This module only · Full module
The idea
Deploying is not releasing. Argo CD makes the cluster match Git, and a Deployment then replaces every pod as soon as each one is ready, so every user meets a bad version at once. Progressive delivery runs the new version first and gives it users a share at a time, while metrics or a person decide whether it goes on.
Remember
- Deploy: new pods run. Release: users reach them. Progressive delivery puts time and a decision between the two.
- Blue-green: a full copy of the new version behind a preview, then one switch for everyone. Fast to undo, double the pods for a while.
- Canary: a share of the users first, more in steps, judged at each step. Fewer extra pods, slower to finish.
- Owners: the platform team runs the controller, Prometheus, the mesh and the shared checks. The product team owns its release steps in
checkout-config. - Same setup as the Argo CD course: kind cluster,
argocd-platform,checkout-app,checkout-config, alice for the platform.
Pitfalls
- Starting on a broken Argo CD platform. Fix or rebuild it first: every module assumes it is Synced and Healthy.
- Thinking a readiness probe judges a release. It only says the pod answers, not that its answers are right.
Commands
kubectl argo rollouts version | Is the plugin installed? |
argocd app list | Every Application, Synced and Healthy before you start |
kubectl -n checkout-dev get deployment checkout -o jsonpath='{.spec.strategy}' | How a release goes out today (RollingUpdate) |
Test yourself
Answer in your head first, then open the answer.
What is the difference between deploy and release?
Deploy runs the new pods; release sends users to them. Progressive delivery separates the two.
When does blue-green fit better than canary?
When you want one clean switch and a fast way back, and can afford a full second copy for a while.
Who owns the release steps of checkout?
The product team, in checkout-config. The platform team owns the controller and the shared checks.
Where module 00 leaves you: Nothing new is installed. Your three repos and the cluster are as the Argo CD course left them, every Application is Synced and Healthy, and the reference repo is cloned next to your repos.
01 Install Argo Rollouts from Git
This module only · Full module
The idea
Argo Rollouts is a controller with its own resource, the Rollout: a Deployment's pod template plus a strategy. It is installed like every other add-on in the platform: one Application in apps/, a pinned chart version and values in Git, never a kubectl apply of a release URL.
Remember
- Pinned: chart
argo-rollouts2.43.12, controller v1.10.2. Upgrades are a pull request changing that line. - ServerSideApply: the Rollouts CRDs are too large for a client-side apply's annotation.
- Sync wave -1: the CRDs exist before any app that uses a Rollout.
- Dashboard read-only: it shows every Rollout; promote and abort go through Argo CD or the CLI, so they leave a trace.
- kubectl plugin:
kubectl-argo-rolloutson your PATH giveskubectl argo rollouts ....
Pitfalls
- Installing from the latest manifest URL. The version changes under you; pin the chart.
- A client-side sync of the CRDs fails with
metadata.annotations: Too long. UseServerSideApply=true.
Commands
argocd app wait argo-rollouts --sync --health | Wait for the controller |
kubectl get crd -o name | grep argoproj.io | Rollout, AnalysisTemplate, AnalysisRun and the rest |
kubectl argo rollouts list rollouts -A | Every Rollout (none yet) |
kubectl -n argo-rollouts port-forward svc/argo-rollouts-dashboard 3100:3100 | The dashboard at localhost:3100/rollouts |
Test yourself
Answer in your head first, then open the answer.
Why does the Rollouts Application run in wave -1?
So its CRDs exist before any app that contains a Rollout is synced.
Why is the dashboard read-only?
Releases are changed through Argo CD or the CLI, which have RBAC and leave a trace.
How do you upgrade Argo Rollouts?
A pull request that changes the pinned chart version in its Application.
Where module 01 leaves you: Argo Rollouts v1.10.2 runs in the argo-rollouts namespace, installed from your platform repo. Nothing uses it yet: checkout is still a Deployment.
02 From Deployment to Rollout
This module only · Full module
The idea
A Rollout replaces the Deployment with the same pod template, so the migration is a chart release: v2.0.0 renders a Rollout instead of a Deployment, and Argo CD prunes the old one. From then on, a new image tag in checkout-config starts a release that follows the Rollout's steps instead of replacing every pod.
Remember
- Chart v2.0.0:
templates/rollout.yamlreplacesdeployment.yaml; the HPA now targetskind: Rollout. - Default strategy: a canary of
setWeight: 50thenpause: {}, which waits for a person. - Argo CD health: a paused Rollout is
Suspended, an aborted oneDegraded, a finished oneHealthy. - Promote leaves a pause. Abort goes back to the stable version, but Git still asks for the new one.
- After an abort, revert in Git. Until then the app stays
Degraded, because Git and the cluster disagree on purpose.
Pitfalls
- Leaving a Rollout aborted without a revert. Argo CD shows
Degradeduntil Git matches what runs. - Reading
setWeight: 50with one replica as half the traffic. Without a traffic router the weight is a pod count.
Commands
kubectl argo rollouts get rollout checkout -n checkout-dev --watch | Steps, ReplicaSets and status, live |
kubectl argo rollouts promote checkout -n checkout-dev | Leave the current pause |
kubectl argo rollouts abort checkout -n checkout-dev | Back to stable, release stopped |
argocd app resources checkout-dev | Rollout in, Deployment gone |
checkout-app chart/values.yaml (v2.0.0)
strategy: canary
canary:
steps:
- setWeight: 50
- pause: {} # a person promotes
analysis: {}
trafficRouting: ""
Test yourself
Answer in your head first, then open the answer.
What does Argo CD show for a Rollout waiting at pause: {}?
Suspended health.
You aborted a release. What is left to do?
Revert the tag change in Git, so Git asks for the version that runs.
Without a traffic router, what does setWeight change?
The number of canary pods compared with stable ones.
Where module 02 leaves you: checkout runs as a Rollout in dev and prod from chart v2.0.0. Dev runs 6.8.0 with 2 pods; prod still runs 6.7.0. The Deployment is gone.
03 Blue-green in dev
This module only · Full module
The idea
Blue-green brings the new version up as a full copy behind a preview Service and host, while every user stays on the old one. A person tries the preview and promotes, and the active Service switches to the new pods in one step. The old pods wait for scaleDownDelaySeconds so the switch can be undone fast.
Remember
- Two Services:
checkout(active) andcheckout-preview. The controller sets their selectors to a pod-template hash. - Two hosts:
checkout-dev.localtest.me(users) andpreview-checkout-dev.localtest.me(the next version). autoPromotionEnabled: false: waits for a person. Analysis replaces the person in module 04.scaleDownDelaySeconds: 30: the old pods keep running after the switch, for a fast way back.- Cost: two full sets of pods while a release waits.
Pitfalls
- A typo in the image tag leaves the preview in
ImagePullBackOff; users are fine, but the release never becomes promotable. Fix the tag in Git. - Writing the Services' selectors by hand. The controller owns them; Argo CD must not fight it.
Commands
curl -sk https://checkout-dev.localtest.me:8443/ | jq -r .version | What users get |
curl -sk https://preview-checkout-dev.localtest.me:8443/ | jq -r .version | What is waiting |
kubectl argo rollouts promote checkout -n checkout-dev | Switch everyone to the preview |
checkout-config helm/dev.yaml (module 03)
strategy: blueGreen
blueGreen:
autoPromotionEnabled: false # a person promotes
scaleDownDelaySeconds: 30
ingress:
host: checkout-dev.localtest.me
previewHost: preview-checkout-dev.localtest.me
Test yourself
Answer in your head first, then open the answer.
What decides which pods the active Service selects?
The Rollout controller, through the pod-template hash it writes to the selector.
Why keep the old pods for 30 seconds after the switch?
So an abort right after the switch is instant, with no new pods to start.
What does blue-green cost compared with canary?
A full second copy of the app while a release waits.
Where module 03 leaves you: Dev runs blue-green: 6.9.0 on 2 pods behind checkout-dev.localtest.me, and preview-checkout-dev.localtest.me shows the next version whenever one is waiting.
04 Metrics and analysis templates
This module only · Full module
The idea
Analysis lets metrics decide instead of a person. The platform team writes the checks once as ClusterAnalysisTemplates; each one asks Prometheus a question about the new pods only, by their pod-template hash, and says what a good answer is. A Rollout runs them and promotes, waits or aborts on their verdict.
Remember
- Prometheus scrapes every pod with
prometheus.io/scrape: "true", keeping the labelsapp_kubernetes_io_nameandrollouts_pod_template_hash. - The load generator sends a few requests a second, so a run has data when nobody uses the app.
- ClusterAnalysisTemplate: shared by every namespace; a Rollout uses it with
clusterScope: true. new-hashcomes frompodTemplateHashValue: Latest: the check measures only the new version.- Phases:
Successfulpromotes,Failedaborts,Error(no data, query broken) aborts afterconsecutiveErrorLimit,Inconclusivepauses. prePromotionAnalysisin blue-green: the preview must pass before the switch.
Pitfalls
- A query that returns nothing is not a pass: it is an
Error, and the release aborts. Check the labels in Prometheus first. - Measuring the whole app instead of the new pods. Without the hash, the stable pods hide a bad canary.
- No traffic, no data. A quiet app needs the load generator or a longer interval.
Commands
kubectl -n monitoring port-forward svc/prometheus-server 9090:80 | Prometheus at localhost:9090 |
kubectl get clusteranalysistemplates | The platform's shared checks |
kubectl -n checkout-dev get analysisrun | Runs and their phase |
kubectl describe analysisrun RUN -n checkout-dev | Every measurement and why it passed or failed |
argocd-platform platform/rollouts/success-rate.yaml (excerpt)
kind: ClusterAnalysisTemplate
metadata:
name: success-rate
spec:
args: [{ name: namespace }, { name: app }, { name: new-hash }]
metrics:
- name: success-rate
interval: 30s
count: 6
failureLimit: 1
successCondition: result[0] >= 0.95
provider:
prometheus:
address: http://prometheus-server.monitoring.svc.cluster.local
query: |
sum(rate(http_requests_total{..., status=~"2.."}[1m]))
/
sum(rate(http_requests_total{...}[1m]))
Test yourself
Answer in your head first, then open the answer.
Why filter on rollouts_pod_template_hash?
So the check measures only the new version, not the stable pods next to it.
The query returns an empty result. What happens?
Each measurement is an error; after the error limit the run is Error and the release aborts.
Why a ClusterAnalysisTemplate instead of an AnalysisTemplate?
One check, written once by the platform team, used by every team's namespace.
Where module 04 leaves you: Prometheus scrapes checkout, a load generator sends it traffic, the success-rate ClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.
05 Canary with automatic rollback in prod
This module only · Full module
The idea
A canary in prod gives the new version a share of the pods in steps, with background analysis running from the first pause. A good release walks through the steps alone. A bad one fails the check and the Rollout aborts itself, with nobody watching; the team then reverts in Git and writes down what happened.
Remember
- Four replicas: one canary pod is 25% of the traffic, so the steps are 25, 50 and 75%.
- Background analysis (
canary.analysis) runs alongside the steps fromstartingStep, and aborts at any failed measurement limit. - An abort scales the canary down and leaves stable serving everyone within seconds.
- Then a revert in
checkout-config, so Git asks for what runs. Argo CD showsHealthyagain. - An incident note in
docs/incidents/: what failed, how it was detected, who responds, how it was corrected, how it is prevented.
Pitfalls
- Moving
analysis:to the wrong indentation: the release runs with no check at all, and the bad version goes out. - Using
retryafter an abort without fixing anything. The same release fails the same way. - Weights that the replica count cannot make: with 4 pods, 5% is still one pod.
Commands
kubectl argo rollouts get rollout checkout -n checkout-prod --watch | Watch the steps and the analysis |
kubectl -n checkout-prod get analysisrun | Which run failed |
kubectl argo rollouts retry rollout checkout -n checkout-prod | Start an aborted release again |
git revert --no-edit HEAD && git push -q | Put Git back on the version that runs |
checkout-config helm/prod.yaml (module 05)
replicas: 4 # one canary pod is 25%
strategy: canary
canary:
steps:
- setWeight: 25
- pause: { duration: 2m }
- setWeight: 50
- pause: { duration: 2m }
- setWeight: 75
- pause: { duration: 1m }
analysis:
templates: [success-rate]
startingStep: 1
Test yourself
Answer in your head first, then open the answer.
A canary aborted at 25%. What do users see?
The stable version, within seconds; the canary pods are scaled down.
Why revert in Git after an automatic abort?
Git still asks for the bad version; until it is reverted, Argo CD shows Degraded.
When does background analysis start?
At startingStep, here the first pause, and it runs until the release ends.
Where module 05 leaves you: Prod runs 6.11.0 on 4 pods as a canary judged by success-rate. A bad release aborts at 25% without a person, and checkout-config/docs/incidents/ has one note.
06 Istio: exact weights and header routes
This module only · Full module
The idea
Without a traffic router, a weight is a pod count. With Istio, the Rollout writes the weights into a VirtualService and the mesh sends exactly that share of requests, however many pods there are. It can also route a test header to the canary before any user gets it, and the sidecars give mutual TLS between pods.
Remember
- Three charts, pinned:
base(CRDs),istiod,gateway(releaseistio-ingress), Istio 1.30.8, sidecar mode. istio-injection: enabledoncheckout-prodandloadgen; existing pods need a restart to get a sidecar.- Two Services:
checkout(stable) andcheckout-canary; the VirtualService routeprimaryhas both with weights. setCanaryScaleruns one canary pod with no traffic;setHeaderRoutesendsx-canary: trueto it.- Argo CD ignores the weights (
ignoreDifferencesplusRespectIgnoreDifferences=true), because the controller changes them. - PeerAuthentication STRICT in prod: only pods with a sidecar can call checkout.
Pitfalls
- Argo CD resetting the weights to 100/0 in the middle of a release. Ignore
.spec.http[].route[].weightand respect it on sync. - STRICT mTLS before the callers have sidecars: the load generator gets connection resets.
- Starting analysis at the header step: nobody but testers sends requests, so there is no data.
Commands
kubectl argo rollouts restart checkout -n checkout-prod | Restart the pods so they get a sidecar |
kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[0].route[*].weight}' | The weights right now |
kubectl -n istio-ingress port-forward svc/istio-ingress 8090:80 | The mesh's front door on localhost:8090 |
curl -s -H 'x-canary: true' http://checkout.localtest.me:8090/ | jq -r .version | What testers get |
checkout-config helm/prod.yaml (module 06, steps)
canary:
trafficRouting: istio
steps:
- setCanaryScale: { replicas: 1 }
- setHeaderRoute:
name: canary-header
match:
- headerName: x-canary
headerValue: { exact: "true" }
- pause: {}
- setWeight: 5
- pause: { duration: 2m }
- setWeight: 25
- pause: { duration: 2m }
- setWeight: 50
- pause: { duration: 2m }
- setHeaderRoute: { name: canary-header }
analysis:
templates: [success-rate]
startingStep: 4
Test yourself
Answer in your head first, then open the answer.
What does setWeight: 5 mean with Istio?
Exactly 5% of requests go to the canary, whatever the pod count.
How do testers reach a canary before users do?
A header route: requests with x-canary: true go to the canary Service.
Why does the checkout-prod Application ignore VirtualService weights?
The Rollout controller changes them during a release; Argo CD must not reset them.
Where module 06 leaves you: Istio 1.30.8 runs; checkout-prod and the load generator have sidecars. Prod releases through a VirtualService: one canary pod, a tester route on the header x-canary: true, then 5/25/50%. Prod runs 6.12.0 with mTLS STRICT.
07 Gateway API: the lighter option
This module only · Full module
The idea
A mesh is not the only way to split traffic exactly. The Gateway API is a standard set of routing resources that many ingress controllers implement, Traefik included. The Argo Rollouts Gateway API plugin writes the canary weights into an HTTPRoute, so the edge splits the traffic and no pod needs a sidecar.
Remember
- Gateway API CRDs v1.5.1 (standard channel), installed from Git before Traefik: the chart does not bundle them.
- Traefik with
providers.kubernetesGateway.enabled, and its Gatewaytraefik-gatewayaccepting routes from namespaces labelledteam: checkout. - The plugin
argoproj-labs/gatewayAPIv0.17.0, named in the Rollouts values; the controller loads it at start. - HTTPRoute
checkoutwith two backends, stable and canary; the controller sets their weights. - Mesh or edge: Istio adds header routes inside the cluster and mTLS; the Gateway API is lighter and only sees traffic that enters through the gateway.
Pitfalls
- An HTTPRoute that is not
Accepted: the Gateway's namespace policy does not admit the route's namespace. - Traffic that skips the gateway (pod to pod) is not split. Only requests through the edge follow the weights.
Commands
kubectl get gatewayclass,gateway -A | Traefik's GatewayClass and Gateway |
kubectl -n argo-rollouts logs deploy/argo-rollouts | grep -i plugin | Did the plugin load? |
kubectl -n checkout-dev get httproute checkout -o jsonpath='{.spec.rules[0].backendRefs[*].weight}' | The weights right now |
curl -s http://checkout-dev.localtest.me:8088/ | jq -r .version | What users get |
checkout-config helm/dev.yaml (module 07)
replicas: 2
strategy: canary
canary:
trafficRouting: gatewayapi
steps:
- setWeight: 20 # 20% even with two pods
- pause: { duration: 1m }
- setWeight: 50
- pause: { duration: 1m }
analysis:
templates: [success-rate]
startingStep: 1
gatewayApi:
host: checkout-dev.localtest.me
Test yourself
Answer in your head first, then open the answer.
Why install the Gateway API CRDs as their own Application?
Traefik's chart does not include them, and they must exist before the Gateway and any HTTPRoute.
What does the plugin change during a release?
The weights of the stable and canary backends in the HTTPRoute.
When is Istio worth its weight over the Gateway API?
When you need header routes or exact splits inside the cluster, or mTLS between pods.
Where module 07 leaves you: Dev releases through HTTPRoute/checkout on Traefik's traefik-gateway: 20% and 50% steps with two pods, judged by success-rate. Dev is served at http://checkout-dev.localtest.me:8088 and runs 6.12.0.
08 Operate releases with teams
This module only · Full module
The idea
The product team owns its releases, so it needs the buttons for them and nothing more. Argo CD resource actions give resume, abort and retry on Rollouts under Argo CD's RBAC, with a trace. Notifications tell the team when a release finishes or aborts, a deadline aborts a release that is stuck, and a runbook says who does what.
Remember
- RBAC:
action/argoproj.io/Rollout/resume,abortandretryforrole:checkout-teamoncheckout/*, instead ofaction/*. argocd app actions run APP resume --kind Rolloutis promote through Argo CD;restartis denied for the team.- Notifications in the Rollouts values: a webhook service
release-log, templates and triggerson-rollout-completedandon-rollout-aborted. - Subscriptions are annotations on the Rollout:
notifications.argoproj.io/subscribe.on-rollout-aborted.release-log. progressDeadlineAbort: true: a release that makes no progress for the deadline aborts instead of hanging.docs/releases.md: who promotes, who aborts, how to revert, where the checks are.
Pitfalls
- Giving the team
action/*: it can then restart or delete pods of any kind in its apps. - A notification that never arrives because the Rollout has no subscription annotation. The trigger alone sends nothing.
Commands
argocd app actions list checkout-prod --kind Rollout | What the team may run |
argocd app actions run checkout-prod resume --kind Rollout | Promote through Argo CD |
argocd app actions run checkout-prod abort --kind Rollout | Abort through Argo CD |
kubectl -n release-log logs deploy/release-log -f | Every notification, as it arrives |
argocd-platform platform/argocd/values.yaml (RBAC excerpt)
p, role:checkout-team, applications, action/argoproj.io/Rollout/resume, checkout/*, allow
p, role:checkout-team, applications, action/argoproj.io/Rollout/abort, checkout/*, allow
p, role:checkout-team, applications, action/argoproj.io/Rollout/retry, checkout/*, allow
Test yourself
Answer in your head first, then open the answer.
How does the product team promote without kubectl access?
argocd app actions run checkout-prod resume --kind Rollout, allowed by its RBAC role.
What makes a notification go to the release log?
A trigger and template in the Rollouts values, and a subscription annotation on the Rollout.
What does progressDeadlineAbort prevent?
A release that hangs forever half-done; it aborts after the progress deadline.
Where module 08 leaves you: The checkout-dev account can resume, abort and retry checkout's Rollouts and nothing else. Every completed or aborted release writes a line to the release log. Prod runs 6.13.0. docs/releases.md is in your platform repo.
09 Capstone: the failed-release drill
This module only · Full module
The idea
A check only catches what it measures. A slow release passes the success-rate check while users wait, so the team rolls it back fast, adds the missing latency check, proves it catches the same release, writes the incident, and puts the next app on the same path. That is the loop: every failure leaves the platform with one more check.
Remember
- Roll back first, investigate second: revert in Git, then
promote --fullif the steps only prolong the incident. promote --fullskips the remaining steps and analysis. Only for a version that already ran in prod.- The latency template: p95 of
http_request_duration_seconds_bucketfor the new hash, at most 0.5 s. - Prove the check: ship the slow release again and watch it abort on
latency, then revert. - Same path for catalog: the chart already supports it; values in
checkout-configare all it takes.
Pitfalls
promote --fullon the new, unproven version: it skips every check you built.- A latency limit tighter than the app's normal p95: every release aborts, and the team learns to ignore the check.
Commands
git revert --no-edit HEAD && git push -q | Back to the last good version |
kubectl argo rollouts promote --full checkout -n checkout-prod | Skip the steps for a known-good version |
kubectl get clusteranalysistemplates | success-rate and latency |
kubectl -n checkout-prod get analysisrun | Which metric failed |
argocd-platform platform/rollouts/latency.yaml (excerpt)
metrics:
- name: p95-latency
interval: 30s
count: 6
failureLimit: 1
successCondition: result[0] <= 0.5 # seconds
provider:
prometheus:
query: |
histogram_quantile(0.95, sum by (le) (rate(
http_request_duration_seconds_bucket{..., rollouts_pod_template_hash="{{args.new-hash}}"}[1m])))
Test yourself
Answer in your head first, then open the answer.
Why did the slow release pass?
The only check was success rate; slow answers are still successful answers.
When is promote --full the right tool?
To roll forward to a version that already ran in prod, when the steps only prolong the incident.
What should every release incident leave behind?
A note with the five answers, and a check that catches the same failure next time.
Where module 09 leaves you: Prod is judged by success-rate and latency, runs 6.13.0, and has an incident note with the five answers. catalog releases as a canary with both checks. You can run progressive delivery for a team.