PlatformEngr® Progressive delivery course

Module 02 · Foundations · 40 min · You are: Product team

From Deployment to Rollout

In this module

  • You learn: How a Rollout differs from a Deployment, what a canary step and a pause do, how Argo CD shows a paused or aborted release, and why "abort" and "revert" are different things.
  • You do: Copy chart v2.0.0 into checkout-app and tag it, move both environments to it, ship podinfo 6.8.0 to dev, watch it stop at 50%, promote it, and then practise an abort.
  • You end with: checkout runs as a Rollout in dev and prod from chart v2.0.0. Dev runs 6.8.0 with 2 pods; prod still runs 6.7.0. The Deployment is gone.

The Rollout is the product team's: it is in the chart, next to the pod template, so the team decides how its releases go out. A canary that pauses at 50% is the smallest useful change: half the pods run the new version, and nothing goes further until someone has looked.

Open this module in the interactive course to track progress and switch tracks.

Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back

Concepts

Revision
Every change to the pod template is a new revision with its own ReplicaSet. The running one is stable; the one being released is the canary.
Steps
setWeight: 50 moves half of the pods (without traffic routing) to the new version. pause: {} waits for a person; pause: { duration: 2m } waits two minutes.
Promote and abort
Promote: skip the current pause. Abort: send everything back to the stable version. Neither changes Git.
Suspended and Degraded
Argo CD's health for a Rollout: Suspended while it waits at a pause, Degraded after an abort, Healthy when a release is complete.

Deployment vs Rollout

DeploymentRollout
Pod template and selectoryesthe same fields
How a new version goes outrolling update, all the waythe steps you write: weights, pauses, analysis
Going backa new rolling updateabort: instant to the stable ReplicaSet
Who runs itkube-controller-managerthe Argo Rollouts controller
HPA, PDB, Servicesworkwork (the HPA targets kind: Rollout)

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. Copy chart v2.0.0 into checkout-app

The new chart: the Deployment template goes, a Rollout and the templates later modules switch on come in. All are off until values turn them on.

cd ~/work/checkout-app && git switch -q main && git pull -q
git rm -q chart/templates/deployment.yaml
cp ~/work/progressive-delivery/checkout-app/chart/Chart.yaml ~/work/progressive-delivery/checkout-app/chart/values.yaml chart/
cp ~/work/progressive-delivery/checkout-app/chart/templates/* chart/templates/

Expect D for deployment.yaml, M for Chart.yaml, values.yaml, _helpers.tpl and hpa.yaml, and new files: rollout.yaml, ingress.yaml, two Services, a VirtualService and an HTTPRoute:

git status --short

Render it as Argo CD will. Expect Rollout where Deployment was, and nothing new besides it: the other templates stay empty with dev's values.

helm template checkout chart -n checkout-dev -f ~/work/checkout-config/helm/dev.yaml | grep '^kind:'

2. Release the chart

A release of checkout-app is a tag. Argo CD deploys tags, never a branch.

git add -A && git commit -qm "Chart 2.0.0: checkout as an Argo Rollouts Rollout" && git push -q

Tag the release and push the tag:

git tag v2.0.0 && git push -q origin v2.0.0

3. Two pods in dev

A canary by pod count needs at least two pods to be one of each. team-checkout changes its dev values.

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

- replicas: 1
+ replicas: 2
git add -A && git commit -qm "checkout dev: 2 replicas" && git push -q

4. Move dev to chart v2.0.0

The platform repo pins the chart release per environment. Change dev first.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Change in ~/work/argocd-platform/apps/checkout-dev.yaml

-       targetRevision: v1.0.0        # a release of the chart, owned by team-checkout
+       targetRevision: v2.0.0        # a release of the chart, owned by team-checkout
git add -A && git commit -qm "checkout-dev: chart v2.0.0" && git push -q
argocd app get root --refresh >/dev/null; argocd app get checkout-dev --refresh >/dev/null

Wait for the switch. Argo CD deletes the Deployment (prune) and creates the Rollout. The first version of a Rollout goes straight to 100%: there is no stable version to protect yet.

argocd app wait checkout-dev --sync --health --timeout 300

Expect a Rollout, no Deployment, and Synced and Healthy on every line:

argocd app resources checkout-dev

5. Read the Rollout

The plugin draws the Rollout: its strategy, its step, and a tree of revisions.

Expect Status: ✔ Healthy, Strategy: Canary, Step: 2/2, one image (podinfo:6.7.1) marked stable, and one ReplicaSet revision:1 with two pods:

kubectl argo rollouts get rollout checkout -n checkout-dev

6. Ship 6.8.0 to dev

A new image is a new revision. Watch it stop at the pause.

In a second terminal, watch the Rollout and leave it running:

kubectl argo rollouts get rollout checkout -n checkout-dev --watch

In the first terminal, change the version:

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

- tag: "6.7.1"                 # promote by changing this line
+ tag: "6.8.0"                 # promote by changing this line
git add -A && git commit -qm "checkout dev: 6.8.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null

In the watch: Step: 1/2, SetWeight: 50, Status: ॥ Paused, two images (6.8.0 as canary, 6.7.1 as stable) and two ReplicaSets with one pod each.

7. What Argo CD and the dashboard show

The release is paused on purpose. Both tools say so.

Expect Health Status: Suspended: the Rollout waits for a person. Sync is Synced: Git says 6.8.0 and the Rollout says 6.8.0. How far it has gone is not in Git.

argocd app get checkout-dev | grep -E 'Sync Status|Health Status'

One pod per version, by the version label:

kubectl -n checkout-dev get pods -L app.kubernetes.io/version

In the dashboard (http://localhost:3100/rollouts, namespace checkout-dev), open checkout. Find: 1. the steps on the left with the pause highlighted, 2. the two revisions with their images, 3. the canary and stable markers.

8. Promote it

You looked, it is fine: let it go on. Promote changes the Rollout's status in the cluster, not Git.

Promote: skip the pause. The watch shows Step: 2/2, two pods on 6.8.0, and ✔ Healthy; the old ReplicaSet scales to zero.

kubectl argo rollouts promote checkout -n checkout-dev

Argo CD says Healthy again:

argocd app wait checkout-dev --sync --health --timeout 300

9. Move prod to chart v2.0.0

Prod keeps its Deployment-like default: a canary that waits for a person at 50%. It becomes a full canary in module 05.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Change in ~/work/argocd-platform/apps/checkout-prod.yaml

-       targetRevision: v1.0.0        # a release of the chart, owned by team-checkout
+       targetRevision: v2.0.0        # a release of the chart, owned by team-checkout
git add -A && git commit -qm "checkout-prod: chart v2.0.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300

Expect ✔ Healthy and podinfo:6.7.0:

kubectl argo rollouts get rollout checkout -n checkout-prod

Simplified for the course

Moving from a Deployment to a Rollout here deletes the Deployment and starts new pods: a few seconds without pods. In production, migrate without downtime with workloadRef: the Rollout points at the existing Deployment and scales it down as its own pods come up (scaleDown: progressively). Then remove the Deployment.

A canary by pod count is as fine as your replicas: with 2 pods the only steps are 50% and 100%. Modules 06 and 07 split traffic instead of pods.

Break it on purpose: Abort, and walk away

team-checkout ships 6.9.0 to dev. At the pause someone sees a stack trace in a log, aborts, and goes to lunch. Nobody touches Git.

You practise: reading an aborted release, and understanding that Git still says the new version until someone changes it.

1. Break it

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

- tag: "6.8.0"
+ tag: "6.9.0"
git add -A && git commit -qm "checkout dev: 6.9.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null; sleep 30
kubectl argo rollouts abort checkout -n checkout-dev

2. What you see

Expected after the abort: checkout back to normal. You get: both pods run 6.8.0 (users are safe), but argocd app get checkout-dev shows Health Status: Degraded, and kubectl argo rollouts get rollout checkout -n checkout-dev shows ✖ Degraded with a message about the abort. Git still says 6.9.0.

Look at the Rollout's status and at helm/dev.yaml in Git.

3. Fix it yourself

Get dev back to Healthy with the version you decide is right. Assume the stack trace was real: 6.9.0 must not go out. Done when checkout-dev is Healthy and both pods run 6.8.0, with Git saying 6.8.0.

Hint 1

An abort is not a decision about the version. Git still asks for 6.9.0, so the Rollout stays Degraded until it gets it or Git changes.

Hint 2

Two ways out: kubectl argo rollouts retry rollout releases 6.9.0 again, or a revert in checkout-config makes Git say 6.8.0. Only one of them keeps 6.9.0 away.

The fix
cd ~/work/checkout-config && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300
kubectl argo rollouts get rollout checkout -n checkout-dev
After you fixed it: the five questions
What failed?
A release was aborted by hand; Git and the cluster disagreed about which version should run.
How was it detected?
Argo CD: Degraded. The plugin: RolloutAborted and the stable image.
Who responds?
team-checkout: the version is theirs to decide.
How is it corrected?
A revert in checkout-config (or, if the version was fine, retry).
How is it prevented?
Make it a rule in the runbook: every abort is followed by a revert or a retry within the hour, and the person who aborts writes which.

Quiz

  1. A Rollout waits at pause: {}. What does Argo CD show?
    • OutOfSync, because the release is not finished
    • Synced and Suspended
    • Synced and Degraded
  2. You abort a release. What does Git say?
    • The new version, until someone changes it
    • The stable version: the abort writes a commit
    • Nothing changes in Git, and Argo CD reverts the abort
  3. Why did the first version of the Rollout go straight to 100%?
    • Because the steps only apply in prod
    • Because there was no stable version yet to protect
    • Because Argo CD skipped the steps on the first sync
Answers
  1. Synced and Suspended Git and the Rollout agree; the Rollout is waiting, which Argo CD calls Suspended.
  2. The new version, until someone changes it Abort is an operation in the cluster. Only a commit changes Git.
  3. Because there was no stable version yet to protect Steps run between a stable and a new revision. The very first revision has nothing to compare with.

Field note

The most common question after an abort: "why is Argo CD red, I fixed it?" The users are fine, but Git still describes the bad version. Make the revert part of the abort, every time, and Argo CD goes green on its own.

Report a problem with this module