Module 03 · Releases judged by metrics · 30 min · You are: Product team
Blue-green in dev
In this module
- You learn: The active and preview Services, why the switch is instant in both directions, and what
scaleDownDelaySecondsbuys you. - You do: Turn on blue-green and two hosts on Traefik in dev's values, ship 6.9.0, compare the live and the preview host, promote, and watch the old copy go 30 seconds later.
- You end with: Dev runs blue-green: 6.9.0 on 2 pods behind
checkout-dev.localtest.me, andpreview-checkout-dev.localtest.meshows the next version whenever one is waiting.
Some releases you want to test whole before anyone sees them: a new API, a schema change behind a flag, a big refactor. Blue-green gives testers a real copy of the next version on its own address, and users get it only when someone promotes. Going back is the same switch in reverse, as long as the old copy still runs.
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Active Service
- What users reach (
checkout). Argo Rollouts points its selector at the pods of the active version. - Preview Service
checkout-preview: the version waiting for promotion. Between releases it points at the active version too.- Promotion
- Swap the selectors: the active Service now selects the new pods. Users move in one step.
- Scale-down delay
scaleDownDelaySeconds: how long the old pods keep running after the switch. During that time an abort switches back instantly.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
checkout-config/helm/stages/dev-03-blue-green.yaml(reference): Dev's values after this module.checkout-app/chart/templates/rollout.yaml(reference): The blue-green half of the Rollout.- The two Services and the two settings, all from values L30–34
checkout-app/chart/templates/ingress.yaml(reference): One Ingress per host, each to its own Service.- The live host goes to checkout, the preview host to checkout-preview L2
Steps
1. Turn on blue-green in dev
Only values change: the chart already supports it. Changing the strategy does not change the pods, so nothing is released yet.
cd ~/work/checkout-config && git switch -q main && git pull -q
Add to ~/work/checkout-config/helm/dev.yaml at the end of the file
strategy: blueGreen
blueGreen:
autoPromotionEnabled: false # a person promotes
scaleDownDelaySeconds: 30
ingress:
host: checkout-dev.localtest.me # the live version
previewHost: preview-checkout-dev.localtest.me # the next one, before promotion
git add -A && git commit -qm "checkout dev: blue-green with a preview host" && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300
Expect Strategy: BlueGreen, one revision marked stable, active:
kubectl argo rollouts get rollout checkout -n checkout-dev
2. Open the live version
Traefik now serves dev on its own host, with the same self-signed certificate as Argo CD.
podinfo answers curl with JSON. Expect "version": "6.8.0" and "message": "checkout (dev)":
curl -sk https://checkout-dev.localtest.me:8443/ | jq '{version, message, hostname}'
In a browser, https://checkout-dev.localtest.me:8443 shows the podinfo page with the same version. The certificate warning is the one you met with Argo CD: continue.
3. Ship 6.9.0
A full second copy comes up behind the preview Service. Users stay on 6.8.0.
In a second terminal, watch:
kubectl argo rollouts get rollout checkout -n checkout-dev --watch
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- tag: "6.8.0"
+ tag: "6.9.0"
git add -A && git commit -qm "checkout dev: 6.9.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null
In the watch: Status: ॥ Paused, BlueGreenPause, a new ReplicaSet with 2 pods marked preview, the old one stable, active. Four pods for a while: that is the price of blue-green.
4. Compare live and preview
Two hosts, two versions, at the same time.
Expect 6.8.0 from the live host and 6.9.0 from the preview host:
curl -sk https://checkout-dev.localtest.me:8443/ | jq -r .version
curl -sk https://preview-checkout-dev.localtest.me:8443/ | jq -r .version
Argo Rollouts did this by changing the Services' selectors. Expect a rollouts-pod-template-hash in both, with different values:
kubectl -n checkout-dev get svc checkout checkout-preview -o custom-columns=NAME:.metadata.name,HASH:.spec.selector.rollouts-pod-template-hash
5. Promote: the switch
Users move in one step. The old pods stay for 30 seconds.
kubectl argo rollouts promote checkout -n checkout-dev
Expect 6.9.0 from the live host right away:
curl -sk https://checkout-dev.localtest.me:8443/ | jq -r .version
In the watch, the old ReplicaSet is marked delay with a countdown, then scales to zero. Until then, an abort would switch back instantly.
Simplified for the course
The preview host is open to anyone who can reach Traefik. In production, put it behind your internal network or an authentication middleware, so only testers see unreleased versions.
Break it on purpose: A typo in the version
team-checkout ships 6.10.0 to dev, typed from a chat message with the letter O instead of a zero.
You practise: finding why a new version never becomes ready, and seeing that blue-green kept users on the old version the whole time.
1. Break it
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- tag: "6.9.0"
+ tag: "6.10.O"
git add -A && git commit -qm "checkout dev: 6.10.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null
2. What you see
Expected: a preview with 6.10.0 to test. You get: the preview host answers with 6.9.0, the Rollout stays ◌ Progressing and, after two minutes, Argo CD shows checkout-dev Degraded. The live host answers 6.9.0 all along.
Look at the new ReplicaSet's pods in kubectl argo rollouts get rollout checkout -n checkout-dev.
3. Fix it yourself
Find why the preview never got the new version, fix it through Git, and get 6.10.0 to the preview. Done when the preview host answers 6.10.0 and the live host still answers 6.9.0 (do not promote yet).
Hint 1
The new pods are not running. What does their status column say?
Hint 2
ImagePullBackOff: the image podinfo:6.10.O does not exist. Correct the tag in helm/dev.yaml.
The fix
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- tag: "6.10.O"
+ tag: "6.10.0"
git add -A && git commit -qm "checkout dev: 6.10.0 (fix the tag)" && git push -q
argocd app get checkout-dev --refresh >/dev/null
Once the watch shows the preview pods running, expect 6.10.0 and 6.9.0:
curl -sk https://preview-checkout-dev.localtest.me:8443/ | jq -r .version
curl -sk https://checkout-dev.localtest.me:8443/ | jq -r .version
Then promote it, so dev runs 6.10.0 for the next module:
kubectl argo rollouts promote checkout -n checkout-dev
After you fixed it: the five questions
- What failed?
- A release with an image that does not exist.
- How was it detected?
- The plugin: the new pods in
ImagePullBackOff. Argo CD:DegradedafterprogressDeadlineSeconds(120 s). - Who responds?
- team-checkout, who typed the tag.
- How is it corrected?
- The right tag in checkout-config; the next revision replaces the broken one.
- How is it prevented?
- CI on checkout-config that checks every image tag exists in the registry before merge. And blue-green itself: users never saw it.
Quiz
- During a blue-green release, how many pods run in dev with
replicas: 2?- Two: one of each version
- Four: two active, two preview
- Three: the preview gets one pod until promotion
- What does promotion change?
- The image of the active pods
- The selector of the active Service, to the new pods
- The Ingress, to a new host
- Why keep the old pods for
scaleDownDelaySecondsafter the switch?- So an abort can switch back instantly, without starting pods
- So Prometheus can scrape them once more
- Because Kubernetes needs 30 seconds to delete a ReplicaSet
Answers
- Four: two active, two preview Blue-green runs a full copy of the new version next to the old one.
- The selector of the active Service, to the new pods Same pods, same hosts: only which pods the active Service selects.
- So an abort can switch back instantly, without starting pods While the old copy runs, going back is a selector change.
Field note
Blue-green is the strategy teams understand fastest, because it matches how they tested before: "there is a staging copy, we look at it, then we switch". Its cost is the second copy. For a service with 40 pods that is 40 more pods during every release.