Module 04 · Releases judged by metrics · 40 min · You are: Platform team
Metrics and analysis templates
In this module
- You learn: How a Prometheus query becomes a release check, why the check looks only at the new pods, and who owns the definition of "healthy".
- You do: Add Prometheus and a load generator, read podinfo's request counter in Prometheus, ship the
success-ratetemplate, turn on pre-promotion analysis in dev, and watch a good release promote itself and a bad one stop. - You end with: Prometheus scrapes checkout, a load generator sends it traffic, the
success-rateClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.
A person looking at a preview catches what a person notices. A query catches what users would notice, on every release, with the same limits each time. The platform team owns the definition (which metric, which limit) so every team is judged the same way; teams choose when to run it.
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Metric
- One entry in a template: a provider (Prometheus), a query, an interval, a count and a success condition.
- Pre-promotion analysis
- Blue-green: run the analysis against the preview before the switch. Success promotes (with auto-promotion on); failure aborts.
- The new pods only
- The query filters on
rollouts_pod_template_hash, the label every pod of one revision carries. The Rollout passes the new revision's hash as an argument. - failureLimit
- How many failed measurements are allowed.
failureLimit: 1means the second failure fails the run.
How an AnalysisRun ends
| Phase | When | What the release does |
|---|---|---|
Successful | every measurement met the condition (within the limits) | goes on (promotes, with auto-promotion) |
Failed | more failed measurements than failureLimit | aborts |
Error | the query could not be evaluated, more than consecutiveErrorLimit (4) times in a row | aborts |
Inconclusive | neither success nor failure condition matched | pauses for a person |
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
platform/rollouts/success-rate.yaml(reference): The platform's shared check: the share of the new pods' requests that succeed.platform/loadgen/loadgen.yaml(reference): Steady traffic for the checks to judge.- One line per target: URL and Host header. The preview host gets traffic too L9–12
checkout-config/helm/stages/dev-04-analysis.yaml(reference): Dev's values after this module.checkout-app/chart/templates/_helpers.tpl(reference): How the chart passes the arguments.- Every template gets namespace, app and the new revision's hash L14–27
Steps
1. Allow the Prometheus charts
The platform project lists every repo it may deploy from. A new chart repository is a deliberate change.
cd ~/work/argocd-platform && git switch -q main && git pull -q
Add to ~/work/argocd-platform/projects/platform.yaml under sourceRepos:, after the external-secrets line
- https://prometheus-community.github.io/helm-charts # Progressive delivery, module 04
2. Add Prometheus and the load generator
Prometheus with only its server, scraping pods that ask for it. The load generator is a pod that sends five requests a second to each checkout host.
mkdir -p prometheus loadgen
cp ~/work/progressive-delivery/platform/apps/prometheus.yaml ~/work/progressive-delivery/platform/apps/loadgen.yaml apps/
cp ~/work/progressive-delivery/platform/prometheus/values.yaml prometheus/
cp ~/work/progressive-delivery/platform/loadgen/loadgen.yaml loadgen/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/prometheus.yaml apps/loadgen.yaml
Create ~/work/argocd-platform/namespaces/platform-tools.yaml
# Namespaces for platform tooling. The others are created by their
# Application (CreateNamespace=true).
apiVersion: v1
kind: Namespace
metadata:
name: loadgen
scripts/validate.sh
git add -A && git commit -qm "Prometheus and a load generator" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait prometheus --sync --health --timeout 300
argocd app wait loadgen --sync --health --timeout 300
3. Read checkout's metrics
The chart's pods carry prometheus.io/scrape: "true". podinfo counts every request by status code.
In a second terminal, forward Prometheus and leave it running:
kubectl -n monitoring port-forward svc/prometheus-server 9090:80
Requests per second in dev, per revision and status. Expect lines with status: 200 (and 404 for the preview host when nothing waits), each with a rollouts_pod_template_hash:
curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (rollouts_pod_template_hash, status) (rate(http_requests_total{namespace="checkout-dev"}[1m]))' | jq -r '.data.result[] | "\(.metric) \(.value[1])"'
The same in the browser: http://localhost:9090, paste the query, Graph tab. One line per revision and status.
4. Ship the success-rate template
The check as the platform defines it. Read it in Quick look before you commit it.
cd ~/work/argocd-platform && git switch -q main && git pull -q
mkdir -p rollouts
cp ~/work/progressive-delivery/platform/rollouts/success-rate.yaml rollouts/
cp ~/work/progressive-delivery/platform/apps/rollouts-templates.yaml apps/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/rollouts-templates.yaml
scripts/validate.sh
git add -A && git commit -qm "Shared release check: success-rate" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait rollouts-templates --sync --health --timeout 300
Expect one row, success-rate:
kubectl get clusteranalysistemplates
5. Let the analysis promote dev
team-checkout opts in: run the platform's check against the preview, and promote on success.
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- autoPromotionEnabled: false # a person promotes
+ autoPromotionEnabled: true # analysis decides now, not a person
Add to ~/work/checkout-config/helm/dev.yaml under blueGreen:, after scaleDownDelaySeconds: 30
prePromotionAnalysis:
templates: [success-rate] # the platform's ClusterAnalysisTemplate
git add -A && git commit -qm "checkout dev: promote when success-rate passes" && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300
6. A good release promotes itself
Ship 6.11.0. The preview comes up, the load generator reaches it, the analysis measures for three minutes and promotes.
In a second terminal, watch:
kubectl argo rollouts get rollout checkout -n checkout-dev --watch
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- tag: "6.10.0"
+ tag: "6.11.0"
git add -A && git commit -qm "checkout dev: 6.11.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null
In the watch: an AnalysisRun under the new revision, ◌ Running with ✔ marks adding up, then the switch. Afterwards, list the runs. Expect one Successful:
kubectl -n checkout-dev get analysisrun
7. A bad release stops itself
Ship the same version with podinfo's random errors on: one request in three fails. Watch the analysis say no.
cd ~/work/checkout-config && git switch -q main && git pull -q
Add to ~/work/checkout-config/helm/dev.yaml at the end of the file
faults:
randomError: true # a bad release, on purpose
git add -A && git commit -qm "checkout dev: random errors (a bad release)" && git push -q
argocd app get checkout-dev --refresh >/dev/null
Within a minute or two the run fails and the Rollout aborts. Expect the newest run Failed and the live host still healthy:
kubectl -n checkout-dev get analysisrun
curl -sk -o /dev/null -w '%{http_code}\n' https://checkout-dev.localtest.me:8443/
What it measured: the values are near 0.67, below 0.95:
kubectl -n checkout-dev get analysisrun -o jsonpath='{.items[-1:].status.metricResults[0].measurements[*].value}'; echo
Undo the bad release in Git, as module 02 taught:
git revert --no-edit HEAD && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300
Simplified for the course
Prometheus here keeps two days of data in memory and scrapes every pod that asks. The Observability course runs it properly. The analysis only needs the query to answer.
The load generator stands in for users. For a quiet service with little traffic, an analysis has too few requests to judge: use synthetic checks (a
jobmetric that runs tests against the preview) or longer pauses.
Break it on purpose: A label that does not exist
A platform engineer "tidies" the template and writes app= instead of app_kubernetes_io_name=, the way it looks in their other dashboards.
You practise: telling a check that failed from a check that could not run, and reading an AnalysisRun.
1. Break it
cd ~/work/argocd-platform && git switch -q main && git pull -q
In both lines of the query (replace all in your editor):
Change in ~/work/argocd-platform/rollouts/success-rate.yaml
- app_kubernetes_io_name="{{args.app}}"
+ app="{{args.app}}"
git add -A && git commit -qm "success-rate: shorter label" && git push -q
argocd app get rollouts-templates --refresh >/dev/null
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/dev.yaml
- tag: "6.11.0"
+ tag: "6.11.1"
git add -A && git commit -qm "checkout dev: 6.11.1" && git push -q
argocd app get checkout-dev --refresh >/dev/null
2. What you see
Expected: 6.11.1 (a good release) promoted after three minutes. You get: after about two minutes the release aborts, and checkout-dev is Degraded. The new pods were fine: no errors, no restarts.
Look at the newest AnalysisRun: kubectl -n checkout-dev get analysisrun and kubectl -n checkout-dev describe analysisrun <its name>.
3. Fix it yourself
Find why a good release was stopped, fix the check, and release 6.11.1. Done when the newest AnalysisRun is Successful and dev runs 6.11.1.
Hint 1
The run is Error, not Failed. A failed run measured something bad; an errored run could not measure.
Hint 2
The measurements say the condition could not be evaluated on an empty result: the query matched no series. Run the query in Prometheus with app= and with app_kubernetes_io_name=.
The fix
cd ~/work/argocd-platform && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get rollouts-templates --refresh >/dev/null
The template is right again. Release 6.11.1 once more:
kubectl argo rollouts retry rollout checkout -n checkout-dev
After three minutes, expect the newest run Successful:
kubectl -n checkout-dev get analysisrun
After you fixed it: the five questions
- What failed?
- The release check: a query that matched nothing.
- How was it detected?
- The AnalysisRun: phase
Error, measurements that could not evaluateresult[0]. - Who responds?
- The platform team owns the template; team-checkout noticed.
- How is it corrected?
- A revert of the template, then
retryof the release. - How is it prevented?
- Test a template change against a running revision before merge: an AnalysisRun from the template in a scratch namespace, or the query in Prometheus with a real hash.
Quiz
- Why does the query filter on
rollouts_pod_template_hash?- To measure only the new version's pods, not the old ones
- Because Prometheus needs a unique label per pod
- To exclude the load generator's requests
- An AnalysisRun ends
Error. What happened?- The new version answered too many errors
- The query could not be evaluated, several times in a row
- A person aborted the release
- Who should own
success-rateand its 95% limit?- Each product team, in its chart
- The platform team, shared by every team
- Whoever releases, set per release
Answers
- To measure only the new version's pods, not the old ones Mixing old and new pods would hide a bad new version behind a good old one.
- The query could not be evaluated, several times in a row Failed is a bad measurement; Error is no usable measurement.
- The platform team, shared by every team One definition of healthy for everyone; teams choose when to use it.
Field note
The first template every team writes compares error counts, not rates, and the first busy release fails it. Rates over a window, divided by the total, are what make a check fair to a service with ten times more traffic.