Module 06 · Traffic · 50 min · You are: Platform and product
Istio: exact weights and header routes
In this module
- You learn: How a mesh splits requests instead of pods, how Argo Rollouts drives a VirtualService, why Argo CD must ignore the weights, and what STRICT mutual TLS changes.
- You do: Install Istio (base, istiod, gateway) as three Applications, label prod and the load generator for sidecars, switch prod to
trafficRouting: istio, ship 6.12.0, test it with a header at 0%, promote through 5, 25 and 50%, and require mTLS. - You end with: Istio 1.30.8 runs; checkout-prod and the load generator have sidecars. Prod releases through a VirtualService: one canary pod, a tester route on the header
x-canary: true, then 5/25/50%. Prod runs 6.12.0 with mTLS STRICT.
With pods as the unit, 5% needs twenty pods. A mesh routes each request, so 5% is 5% with one canary pod, and a header can send testers to the canary before any user gets it. Istio is the platform team's; the VirtualService that says how checkout is routed is team-checkout's, in its chart.
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Sidecar
- An Envoy proxy Istio adds to every pod in a labelled namespace (
istio-injection: enabled). It routes outgoing requests and secures incoming ones. - VirtualService
- Istio's routing rule for a host: here two destinations,
checkout(stable) andcheckout-canary, with weights Argo Rollouts changes during a release. - Header route
setHeaderRoute: a route added in front of the weights, so requests withx-canary: truego to the canary whatever the weight.- ignoreDifferences
- Git says 100/0; during a release the cluster says 95/5. Argo CD must not call that drift, or self-heal would fight Argo Rollouts.
- mTLS STRICT
- Sidecars only accept mutually authenticated TLS. A caller outside the mesh is refused.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
platform/apps/istiod.yaml(reference): Istio's control plane, from the official chart, with values from Git.platform/istio/config/gateway.yaml(reference): The platform's front door into the mesh.checkout-app/chart/templates/virtualservice.yaml(reference): How checkout is routed: team-checkout's.checkout-config/helm/prod.yaml(reference): Prod's values after this module (module 09 addslatency).platform/apps/checkout-prod.yaml(reference): The prod Application, now ignoring the weights.
Steps
1. Install Istio from Git
Three Applications in the documented order (base, istiod, gateway), the gateway's config, and the chart repository allowed in the platform project.
cd ~/work/argocd-platform && git switch -q main && git pull -q
Add to ~/work/argocd-platform/projects/platform.yaml under sourceRepos:, after the prometheus line
- https://istio-release.storage.googleapis.com/charts # module 06
mkdir -p istio/config
cp ~/work/progressive-delivery/platform/apps/istio-base.yaml ~/work/progressive-delivery/platform/apps/istiod.yaml ~/work/progressive-delivery/platform/apps/istio-gateway.yaml apps/
cp ~/work/progressive-delivery/platform/istio/istiod-values.yaml ~/work/progressive-delivery/platform/istio/gateway-values.yaml istio/
cp ~/work/progressive-delivery/platform/istio/config/gateway.yaml istio/config/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/istiod.yaml apps/istio-gateway.yaml
scripts/validate.sh
git add -A && git commit -qm "Istio 1.30.8: base, istiod, ingress gateway" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait istio-base --sync --health --timeout 300
argocd app wait istiod --sync --health --timeout 300
argocd app wait istio-gateway --sync --health --timeout 300
Expect istiod in istio-system and istio-ingress in istio-ingress, all 1/1:
kubectl get pods -n istio-system -n istio-ingress 2>/dev/null; kubectl get pods -A -l 'app in (istiod,istio-ingress)'
2. Put prod and the load generator in the mesh
Namespaces are the platform team's. A label tells istiod to add a sidecar to new pods there.
Add to ~/work/argocd-platform/namespaces/checkout.yaml under the checkout-prod labels, after env: prod
istio-injection: enabled # module 06: every pod gets an Istio sidecar
Add to ~/work/argocd-platform/namespaces/platform-tools.yaml at the end of the loadgen Namespace, after name: loadgen
labels:
istio-injection: enabled # module 06: its requests go through the mesh
git add -A && git commit -qm "Sidecars for checkout-prod and loadgen" && git push -q
argocd app get namespaces --refresh >/dev/null
Existing pods keep running without a sidecar until they restart. Restart both; the Rollout restart replaces checkout's pods one at a time:
kubectl argo rollouts restart checkout -n checkout-prod
kubectl -n loadgen rollout restart deployment loadgen
A minute later, expect 2/2 under READY: the app and its sidecar:
kubectl -n checkout-prod get pods
3. Route prod through Istio
team-checkout switches prod to traffic routing; the platform team tells Argo CD to leave the weights alone. Both change in the same hour.
Platform: ignore the weights in the prod Application.
Add to ~/work/argocd-platform/apps/checkout-prod.yaml above syncPolicy:, at the same indentation
# Module 06: Argo Rollouts moves the weights during a release. Git keeps
# 100/0; this keeps Argo CD from calling that drift and putting them back.
ignoreDifferences:
- group: networking.istio.io
kind: VirtualService
jqPathExpressions:
- .spec.http[].route[].weight
Add to ~/work/argocd-platform/apps/checkout-prod.yaml at the end of the file, under syncPolicy: (after selfHeal: true)
syncOptions:
- RespectIgnoreDifferences=true
git add -A && git commit -qm "checkout-prod: ignore VirtualService weights" && git push -q
argocd app get root --refresh >/dev/null
team-checkout: replace the canary: block in prod's values with the one below, and add the Istio host.
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- canary:
- steps:
- - setWeight: 25 # one pod of four
- - pause: { duration: 2m } # analysis watches while it waits
- - setWeight: 50
- - pause: { duration: 2m }
- - setWeight: 75
- - pause: { duration: 1m }
- analysis:
- templates: [success-rate]
- startingStep: 1 # from the first pause on
+ canary:
+ trafficRouting: istio
+ steps:
+ - setCanaryScale: { replicas: 1 } # one canary pod, no traffic yet
+ - setHeaderRoute: # testers reach it with a header
+ name: canary-header
+ match:
+ - headerName: x-canary
+ headerValue: { exact: "true" }
+ - pause: {} # a person tries it, then promotes
+ - setWeight: 5 # 5% of real requests
+ - pause: { duration: 2m }
+ - setWeight: 25
+ - pause: { duration: 2m }
+ - setWeight: 50
+ - pause: { duration: 2m }
+ - setHeaderRoute: { name: canary-header } # remove the test route
+ analysis:
+ templates: [success-rate]
+ startingStep: 4 # once real users get the canary
+ istio:
+ host: checkout.localtest.me # http://checkout.localtest.me:8090 (port-forward)
+ headerRoute: true
git add -A && git commit -qm "checkout prod: canary through Istio, header route for testers" && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300
Expect the two weights from Git, 100 and 0, and Synced:
kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[0].route[*].weight}'; echo
4. Reach prod through the gateway
On kind the gateway has no load balancer. A port-forward on 8090 stands in for one.
In a second terminal, leave this running:
kubectl -n istio-ingress port-forward svc/istio-ingress 8090:80
Expect 6.11.0. The request went through the gateway, the VirtualService and a sidecar:
curl -s http://checkout.localtest.me:8090/ | jq -r .version
5. Ship 6.12.0 and test it with a header
One canary pod comes up with no users. Testers send a header.
In a third terminal, watch:
kubectl argo rollouts get rollout checkout -n checkout-prod --watch
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- tag: "6.11.0"
+ tag: "6.12.0"
git add -A && git commit -qm "checkout prod: 6.12.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
At the pause (Step: 2/11, one canary pod), expect 6.11.0 without the header and 6.12.0 with it:
curl -s http://checkout.localtest.me:8090/ | jq -r .version
curl -s -H 'x-canary: true' http://checkout.localtest.me:8090/ | jq -r .version
The route Argo Rollouts added, in front of primary. Expect canary-header first:
kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[*].name}'; echo
6. Promote: 5% with one pod
The tester is happy. Real users start at 5%, and the analysis starts with them.
kubectl argo rollouts promote checkout -n checkout-prod
Expect 95 5, and still one canary pod among five:
kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[-1:].route[*].weight}'; echo
The sidecars count every request. With the Prometheus port-forward from module 04 running, expect about 19 times more requests to checkout than to checkout-canary:
curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (destination_service_name) (rate(istio_requests_total{reporter="destination", destination_service_namespace="checkout-prod"}[1m]))' | jq -r '.data.result[] | "\(.metric) \(.value[1])"'
Argo CD stays Synced while the weights move: the ignore works.
argocd app get checkout-prod | grep 'Sync Status'
Let it finish: 25%, 50%, then 100%. About eight minutes; the watch ends ✔ Healthy on 6.12.0.
7. Require mutual TLS in prod
Every caller of checkout-prod now has a sidecar (the load generator, the gateway). Make that the rule.
cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/progressive-delivery/platform/istio/config/peer-authentication.yaml istio/config/
git add -A && git commit -qm "checkout-prod: mTLS STRICT" && git push -q
argocd app get istio-gateway --refresh >/dev/null
From a pod outside the mesh, expect the call to fail (curl reports a reset connection, exit code 56):
kubectl run nomesh --rm -i --restart=Never --image=curlimages/curl:8.11.1 -- curl -s -m 5 http://checkout.checkout-prod/
Through the gateway it still works. Expect 6.12.0:
curl -s http://checkout.localtest.me:8090/ | jq -r .version
Simplified for the course
Sidecar mode, the long-established Istio mode with the most complete Argo Rollouts support. Istio's ambient mode drops the sidecars and routes through waypoint proxies with the Gateway API; the Rollout side looks like module 07.
Istio is taught here only as far as delivery needs: routing and one security rule. Authorization policies, egress control and multi-cluster are a course of their own.
Break it on purpose: The pod that skipped the mesh
The platform team rebuilds the load generator's namespace "to clean it up" and forgets the label. Nothing errors.
You practise: recognising traffic that bypasses the mesh, from the numbers rather than from an error.
1. Break it
cd ~/work/argocd-platform && git switch -q main && git pull -q
Delete from ~/work/argocd-platform/namespaces/platform-tools.yaml
- labels:
- istio-injection: enabled # module 06: its requests go through the mesh
git add -A && git commit -qm "loadgen: tidy namespace" && git push -q
argocd app get namespaces --refresh >/dev/null
The label goes (prune is off for namespaces, but the label is a field Argo CD owns). Restart the load generator so its pod is recreated:
kubectl -n loadgen rollout restart deployment loadgen
2. What you see
Expected: the load generator's requests to checkout-prod succeed and show up in istio_requests_total. You get: the loadgen pod is 1/1 (no sidecar), and its requests to checkout.checkout-prod fail: the canary analysis in the next release would see no traffic from it. Argo CD is Synced and Healthy everywhere.
Look at the loadgen pod's READY column and at its requests: kubectl -n loadgen exec deploy/loadgen -- curl -s -m 5 -o /dev/null -w "%{http_code}" http://checkout.checkout-prod/.
3. Fix it yourself
Find why the load generator lost its access, fix it in Git, and show that its requests reach prod again. Done when the loadgen pod is 2/2 and the curl above prints 200.
Hint 1
1/1 means no sidecar. Which object decides whether a pod gets one?
Hint 2
Put the istio-injection: enabled label back on the loadgen Namespace in Git, then restart the Deployment. STRICT mTLS refuses callers without a sidecar.
The fix
cd ~/work/argocd-platform && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get namespaces --refresh >/dev/null
kubectl -n loadgen rollout restart deployment loadgen
A minute later, expect 2/2 and 200:
kubectl -n loadgen get pods
kubectl -n loadgen exec deploy/loadgen -c loadgen -- curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://checkout.checkout-prod/
After you fixed it: the five questions
- What failed?
- A namespace without the injection label: its new pods were outside the mesh, and STRICT mTLS refused them.
- How was it detected?
- The pod's READY
1/1, failed calls from it, and a drop in prod'sistio_requests_total. No error in Argo CD. - Who responds?
- The platform team, who own namespaces.
- How is it corrected?
- The label back in Git and a restart of the pods.
- How is it prevented?
- Check the label in CI (
validate.shcan fail a Namespace in a mesh list without it), and alert on a sudden drop in requests to a service.
Quiz
- Prod has 4 pods. What does
setWeight: 5give you with Istio routing?- One canary pod and about 25% of requests
- About 5% of requests to the canary, whatever the pod counts
- No traffic until the canary has 5 pods
- Why does the prod Application ignore the VirtualService weights?
- Because Argo CD cannot read Istio resources
- Because Argo Rollouts changes them during a release; self-heal would put Git's 100/0 back
- Because weights are secrets
- What does the header route let you do?
- Send testers to the canary before any real user gets it
- Encrypt traffic between the gateway and the pods
- Skip the analysis for requests with a header
Answers
- About 5% of requests to the canary, whatever the pod counts The VirtualService splits requests; the pod count only has to carry them.
- Because Argo Rollouts changes them during a release; self-heal would put Git's 100/0 back Two controllers must not own the same field. Argo CD gives it to Argo Rollouts.
- Send testers to the canary before any real user gets it A tester route at 0% weight is the safest first look at a new version in prod.
Field note
Teams often add Istio for the canary and discover the mesh was the bigger change: every pod has a sidecar, every call goes through it, and every namespace label matters. Start with one namespace, as you did here, and add the next only when the first is boring.