PlatformEngr® Progressive delivery course

Module 06 · Traffic · 50 min · You are: Platform and product

Istio: exact weights and header routes

In this module

  • You learn: How a mesh splits requests instead of pods, how Argo Rollouts drives a VirtualService, why Argo CD must ignore the weights, and what STRICT mutual TLS changes.
  • You do: Install Istio (base, istiod, gateway) as three Applications, label prod and the load generator for sidecars, switch prod to trafficRouting: istio, ship 6.12.0, test it with a header at 0%, promote through 5, 25 and 50%, and require mTLS.
  • You end with: Istio 1.30.8 runs; checkout-prod and the load generator have sidecars. Prod releases through a VirtualService: one canary pod, a tester route on the header x-canary: true, then 5/25/50%. Prod runs 6.12.0 with mTLS STRICT.

With pods as the unit, 5% needs twenty pods. A mesh routes each request, so 5% is 5% with one canary pod, and a header can send testers to the canary before any user gets it. Istio is the platform team's; the VirtualService that says how checkout is routed is team-checkout's, in its chart.

Open this module in the interactive course to track progress and switch tracks.

Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back

Concepts

Sidecar
An Envoy proxy Istio adds to every pod in a labelled namespace (istio-injection: enabled). It routes outgoing requests and secures incoming ones.
VirtualService
Istio's routing rule for a host: here two destinations, checkout (stable) and checkout-canary, with weights Argo Rollouts changes during a release.
Header route
setHeaderRoute: a route added in front of the weights, so requests with x-canary: true go to the canary whatever the weight.
ignoreDifferences
Git says 100/0; during a release the cluster says 95/5. Argo CD must not call that drift, or self-heal would fight Argo Rollouts.
mTLS STRICT
Sidecars only accept mutually authenticated TLS. A caller outside the mesh is refused.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. Install Istio from Git

Three Applications in the documented order (base, istiod, gateway), the gateway's config, and the chart repository allowed in the platform project.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Add to ~/work/argocd-platform/projects/platform.yaml under sourceRepos:, after the prometheus line

    - https://istio-release.storage.googleapis.com/charts  # module 06
mkdir -p istio/config
cp ~/work/progressive-delivery/platform/apps/istio-base.yaml ~/work/progressive-delivery/platform/apps/istiod.yaml ~/work/progressive-delivery/platform/apps/istio-gateway.yaml apps/
cp ~/work/progressive-delivery/platform/istio/istiod-values.yaml ~/work/progressive-delivery/platform/istio/gateway-values.yaml istio/
cp ~/work/progressive-delivery/platform/istio/config/gateway.yaml istio/config/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/istiod.yaml apps/istio-gateway.yaml
scripts/validate.sh
git add -A && git commit -qm "Istio 1.30.8: base, istiod, ingress gateway" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait istio-base --sync --health --timeout 300
argocd app wait istiod --sync --health --timeout 300
argocd app wait istio-gateway --sync --health --timeout 300

Expect istiod in istio-system and istio-ingress in istio-ingress, all 1/1:

kubectl get pods -n istio-system -n istio-ingress 2>/dev/null; kubectl get pods -A -l 'app in (istiod,istio-ingress)'

2. Put prod and the load generator in the mesh

Namespaces are the platform team's. A label tells istiod to add a sidecar to new pods there.

Add to ~/work/argocd-platform/namespaces/checkout.yaml under the checkout-prod labels, after env: prod

    istio-injection: enabled   # module 06: every pod gets an Istio sidecar

Add to ~/work/argocd-platform/namespaces/platform-tools.yaml at the end of the loadgen Namespace, after name: loadgen

  labels:
    istio-injection: enabled       # module 06: its requests go through the mesh
git add -A && git commit -qm "Sidecars for checkout-prod and loadgen" && git push -q
argocd app get namespaces --refresh >/dev/null

Existing pods keep running without a sidecar until they restart. Restart both; the Rollout restart replaces checkout's pods one at a time:

kubectl argo rollouts restart checkout -n checkout-prod
kubectl -n loadgen rollout restart deployment loadgen

A minute later, expect 2/2 under READY: the app and its sidecar:

kubectl -n checkout-prod get pods

3. Route prod through Istio

team-checkout switches prod to traffic routing; the platform team tells Argo CD to leave the weights alone. Both change in the same hour.

Platform: ignore the weights in the prod Application.

Add to ~/work/argocd-platform/apps/checkout-prod.yaml above syncPolicy:, at the same indentation

  # Module 06: Argo Rollouts moves the weights during a release. Git keeps
  # 100/0; this keeps Argo CD from calling that drift and putting them back.
  ignoreDifferences:
    - group: networking.istio.io
      kind: VirtualService
      jqPathExpressions:
        - .spec.http[].route[].weight

Add to ~/work/argocd-platform/apps/checkout-prod.yaml at the end of the file, under syncPolicy: (after selfHeal: true)

    syncOptions:
      - RespectIgnoreDifferences=true
git add -A && git commit -qm "checkout-prod: ignore VirtualService weights" && git push -q
argocd app get root --refresh >/dev/null

team-checkout: replace the canary: block in prod's values with the one below, and add the Istio host.

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/prod.yaml

- canary:
-   steps:
-     - setWeight: 25              # one pod of four
-     - pause: { duration: 2m }    # analysis watches while it waits
-     - setWeight: 50
-     - pause: { duration: 2m }
-     - setWeight: 75
-     - pause: { duration: 1m }
-   analysis:
-     templates: [success-rate]
-     startingStep: 1              # from the first pause on
+ canary:
+   trafficRouting: istio
+   steps:
+     - setCanaryScale: { replicas: 1 }       # one canary pod, no traffic yet
+     - setHeaderRoute:                        # testers reach it with a header
+         name: canary-header
+         match:
+           - headerName: x-canary
+             headerValue: { exact: "true" }
+     - pause: {}                              # a person tries it, then promotes
+     - setWeight: 5                           # 5% of real requests
+     - pause: { duration: 2m }
+     - setWeight: 25
+     - pause: { duration: 2m }
+     - setWeight: 50
+     - pause: { duration: 2m }
+     - setHeaderRoute: { name: canary-header }   # remove the test route
+   analysis:
+     templates: [success-rate]
+     startingStep: 4                          # once real users get the canary
+ istio:
+   host: checkout.localtest.me                # http://checkout.localtest.me:8090 (port-forward)
+   headerRoute: true
git add -A && git commit -qm "checkout prod: canary through Istio, header route for testers" && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health --timeout 300

Expect the two weights from Git, 100 and 0, and Synced:

kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[0].route[*].weight}'; echo

4. Reach prod through the gateway

On kind the gateway has no load balancer. A port-forward on 8090 stands in for one.

In a second terminal, leave this running:

kubectl -n istio-ingress port-forward svc/istio-ingress 8090:80

Expect 6.11.0. The request went through the gateway, the VirtualService and a sidecar:

curl -s http://checkout.localtest.me:8090/ | jq -r .version

5. Ship 6.12.0 and test it with a header

One canary pod comes up with no users. Testers send a header.

In a third terminal, watch:

kubectl argo rollouts get rollout checkout -n checkout-prod --watch
cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/prod.yaml

- tag: "6.11.0"
+ tag: "6.12.0"
git add -A && git commit -qm "checkout prod: 6.12.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null

At the pause (Step: 2/11, one canary pod), expect 6.11.0 without the header and 6.12.0 with it:

curl -s http://checkout.localtest.me:8090/ | jq -r .version
curl -s -H 'x-canary: true' http://checkout.localtest.me:8090/ | jq -r .version

The route Argo Rollouts added, in front of primary. Expect canary-header first:

kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[*].name}'; echo

6. Promote: 5% with one pod

The tester is happy. Real users start at 5%, and the analysis starts with them.

kubectl argo rollouts promote checkout -n checkout-prod

Expect 95 5, and still one canary pod among five:

kubectl -n checkout-prod get virtualservice checkout -o jsonpath='{.spec.http[-1:].route[*].weight}'; echo

The sidecars count every request. With the Prometheus port-forward from module 04 running, expect about 19 times more requests to checkout than to checkout-canary:

curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (destination_service_name) (rate(istio_requests_total{reporter="destination", destination_service_namespace="checkout-prod"}[1m]))' | jq -r '.data.result[] | "\(.metric) \(.value[1])"'

Argo CD stays Synced while the weights move: the ignore works.

argocd app get checkout-prod | grep 'Sync Status'

Let it finish: 25%, 50%, then 100%. About eight minutes; the watch ends ✔ Healthy on 6.12.0.

7. Require mutual TLS in prod

Every caller of checkout-prod now has a sidecar (the load generator, the gateway). Make that the rule.

cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/progressive-delivery/platform/istio/config/peer-authentication.yaml istio/config/
git add -A && git commit -qm "checkout-prod: mTLS STRICT" && git push -q
argocd app get istio-gateway --refresh >/dev/null

From a pod outside the mesh, expect the call to fail (curl reports a reset connection, exit code 56):

kubectl run nomesh --rm -i --restart=Never --image=curlimages/curl:8.11.1 -- curl -s -m 5 http://checkout.checkout-prod/

Through the gateway it still works. Expect 6.12.0:

curl -s http://checkout.localtest.me:8090/ | jq -r .version

Simplified for the course

Sidecar mode, the long-established Istio mode with the most complete Argo Rollouts support. Istio's ambient mode drops the sidecars and routes through waypoint proxies with the Gateway API; the Rollout side looks like module 07.

Istio is taught here only as far as delivery needs: routing and one security rule. Authorization policies, egress control and multi-cluster are a course of their own.

Break it on purpose: The pod that skipped the mesh

The platform team rebuilds the load generator's namespace "to clean it up" and forgets the label. Nothing errors.

You practise: recognising traffic that bypasses the mesh, from the numbers rather than from an error.

1. Break it

cd ~/work/argocd-platform && git switch -q main && git pull -q

Delete from ~/work/argocd-platform/namespaces/platform-tools.yaml

-   labels:
-     istio-injection: enabled       # module 06: its requests go through the mesh
git add -A && git commit -qm "loadgen: tidy namespace" && git push -q
argocd app get namespaces --refresh >/dev/null

The label goes (prune is off for namespaces, but the label is a field Argo CD owns). Restart the load generator so its pod is recreated:

kubectl -n loadgen rollout restart deployment loadgen

2. What you see

Expected: the load generator's requests to checkout-prod succeed and show up in istio_requests_total. You get: the loadgen pod is 1/1 (no sidecar), and its requests to checkout.checkout-prod fail: the canary analysis in the next release would see no traffic from it. Argo CD is Synced and Healthy everywhere.

Look at the loadgen pod's READY column and at its requests: kubectl -n loadgen exec deploy/loadgen -- curl -s -m 5 -o /dev/null -w "%{http_code}" http://checkout.checkout-prod/.

3. Fix it yourself

Find why the load generator lost its access, fix it in Git, and show that its requests reach prod again. Done when the loadgen pod is 2/2 and the curl above prints 200.

Hint 1

1/1 means no sidecar. Which object decides whether a pod gets one?

Hint 2

Put the istio-injection: enabled label back on the loadgen Namespace in Git, then restart the Deployment. STRICT mTLS refuses callers without a sidecar.

The fix
cd ~/work/argocd-platform && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get namespaces --refresh >/dev/null
kubectl -n loadgen rollout restart deployment loadgen

A minute later, expect 2/2 and 200:

kubectl -n loadgen get pods
kubectl -n loadgen exec deploy/loadgen -c loadgen -- curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://checkout.checkout-prod/
After you fixed it: the five questions
What failed?
A namespace without the injection label: its new pods were outside the mesh, and STRICT mTLS refused them.
How was it detected?
The pod's READY 1/1, failed calls from it, and a drop in prod's istio_requests_total. No error in Argo CD.
Who responds?
The platform team, who own namespaces.
How is it corrected?
The label back in Git and a restart of the pods.
How is it prevented?
Check the label in CI (validate.sh can fail a Namespace in a mesh list without it), and alert on a sudden drop in requests to a service.

Quiz

  1. Prod has 4 pods. What does setWeight: 5 give you with Istio routing?
    • One canary pod and about 25% of requests
    • About 5% of requests to the canary, whatever the pod counts
    • No traffic until the canary has 5 pods
  2. Why does the prod Application ignore the VirtualService weights?
    • Because Argo CD cannot read Istio resources
    • Because Argo Rollouts changes them during a release; self-heal would put Git's 100/0 back
    • Because weights are secrets
  3. What does the header route let you do?
    • Send testers to the canary before any real user gets it
    • Encrypt traffic between the gateway and the pods
    • Skip the analysis for requests with a header
Answers
  1. About 5% of requests to the canary, whatever the pod counts The VirtualService splits requests; the pod count only has to carry them.
  2. Because Argo Rollouts changes them during a release; self-heal would put Git's 100/0 back Two controllers must not own the same field. Argo CD gives it to Argo Rollouts.
  3. Send testers to the canary before any real user gets it A tester route at 0% weight is the safest first look at a new version in prod.

Field note

Teams often add Istio for the canary and discover the mesh was the bigger change: every pod has a sidecar, every call goes through it, and every namespace label matters. Start with one namespace, as you did here, and add the next only when the first is boring.

Report a problem with this module