PlatformEngr® Progressive delivery course

Module 04 · Releases judged by metrics · 40 min · You are: Platform team

Metrics and analysis templates

In this module

  • You learn: How a Prometheus query becomes a release check, why the check looks only at the new pods, and who owns the definition of "healthy".
  • You do: Add Prometheus and a load generator, read podinfo's request counter in Prometheus, ship the success-rate template, turn on pre-promotion analysis in dev, and watch a good release promote itself and a bad one stop.
  • You end with: Prometheus scrapes checkout, a load generator sends it traffic, the success-rate ClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.

A person looking at a preview catches what a person notices. A query catches what users would notice, on every release, with the same limits each time. The platform team owns the definition (which metric, which limit) so every team is judged the same way; teams choose when to run it.

Open this module in the interactive course to track progress and switch tracks.

Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back

Concepts

Metric
One entry in a template: a provider (Prometheus), a query, an interval, a count and a success condition.
Pre-promotion analysis
Blue-green: run the analysis against the preview before the switch. Success promotes (with auto-promotion on); failure aborts.
The new pods only
The query filters on rollouts_pod_template_hash, the label every pod of one revision carries. The Rollout passes the new revision's hash as an argument.
failureLimit
How many failed measurements are allowed. failureLimit: 1 means the second failure fails the run.

How an AnalysisRun ends

PhaseWhenWhat the release does
Successfulevery measurement met the condition (within the limits)goes on (promotes, with auto-promotion)
Failedmore failed measurements than failureLimitaborts
Errorthe query could not be evaluated, more than consecutiveErrorLimit (4) times in a rowaborts
Inconclusiveneither success nor failure condition matchedpauses for a person

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. Allow the Prometheus charts

The platform project lists every repo it may deploy from. A new chart repository is a deliberate change.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Add to ~/work/argocd-platform/projects/platform.yaml under sourceRepos:, after the external-secrets line

    - https://prometheus-community.github.io/helm-charts   # Progressive delivery, module 04

2. Add Prometheus and the load generator

Prometheus with only its server, scraping pods that ask for it. The load generator is a pod that sends five requests a second to each checkout host.

mkdir -p prometheus loadgen
cp ~/work/progressive-delivery/platform/apps/prometheus.yaml ~/work/progressive-delivery/platform/apps/loadgen.yaml apps/
cp ~/work/progressive-delivery/platform/prometheus/values.yaml prometheus/
cp ~/work/progressive-delivery/platform/loadgen/loadgen.yaml loadgen/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/prometheus.yaml apps/loadgen.yaml

Create ~/work/argocd-platform/namespaces/platform-tools.yaml

# Namespaces for platform tooling. The others are created by their
# Application (CreateNamespace=true).
apiVersion: v1
kind: Namespace
metadata:
  name: loadgen
scripts/validate.sh
git add -A && git commit -qm "Prometheus and a load generator" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait prometheus --sync --health --timeout 300
argocd app wait loadgen --sync --health --timeout 300

3. Read checkout's metrics

The chart's pods carry prometheus.io/scrape: "true". podinfo counts every request by status code.

In a second terminal, forward Prometheus and leave it running:

kubectl -n monitoring port-forward svc/prometheus-server 9090:80

Requests per second in dev, per revision and status. Expect lines with status: 200 (and 404 for the preview host when nothing waits), each with a rollouts_pod_template_hash:

curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (rollouts_pod_template_hash, status) (rate(http_requests_total{namespace="checkout-dev"}[1m]))' | jq -r '.data.result[] | "\(.metric) \(.value[1])"'

The same in the browser: http://localhost:9090, paste the query, Graph tab. One line per revision and status.

4. Ship the success-rate template

The check as the platform defines it. Read it in Quick look before you commit it.

cd ~/work/argocd-platform && git switch -q main && git pull -q
mkdir -p rollouts
cp ~/work/progressive-delivery/platform/rollouts/success-rate.yaml rollouts/
cp ~/work/progressive-delivery/platform/apps/rollouts-templates.yaml apps/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/rollouts-templates.yaml
scripts/validate.sh
git add -A && git commit -qm "Shared release check: success-rate" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait rollouts-templates --sync --health --timeout 300

Expect one row, success-rate:

kubectl get clusteranalysistemplates

5. Let the analysis promote dev

team-checkout opts in: run the platform's check against the preview, and promote on success.

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

-   autoPromotionEnabled: false    # a person promotes
+   autoPromotionEnabled: true     # analysis decides now, not a person

Add to ~/work/checkout-config/helm/dev.yaml under blueGreen:, after scaleDownDelaySeconds: 30

  prePromotionAnalysis:
    templates: [success-rate]    # the platform's ClusterAnalysisTemplate
git add -A && git commit -qm "checkout dev: promote when success-rate passes" && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300

6. A good release promotes itself

Ship 6.11.0. The preview comes up, the load generator reaches it, the analysis measures for three minutes and promotes.

In a second terminal, watch:

kubectl argo rollouts get rollout checkout -n checkout-dev --watch
cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

- tag: "6.10.0"
+ tag: "6.11.0"
git add -A && git commit -qm "checkout dev: 6.11.0" && git push -q
argocd app get checkout-dev --refresh >/dev/null

In the watch: an AnalysisRun under the new revision, ◌ Running with ✔ marks adding up, then the switch. Afterwards, list the runs. Expect one Successful:

kubectl -n checkout-dev get analysisrun

7. A bad release stops itself

Ship the same version with podinfo's random errors on: one request in three fails. Watch the analysis say no.

cd ~/work/checkout-config && git switch -q main && git pull -q

Add to ~/work/checkout-config/helm/dev.yaml at the end of the file

faults:
  randomError: true              # a bad release, on purpose
git add -A && git commit -qm "checkout dev: random errors (a bad release)" && git push -q
argocd app get checkout-dev --refresh >/dev/null

Within a minute or two the run fails and the Rollout aborts. Expect the newest run Failed and the live host still healthy:

kubectl -n checkout-dev get analysisrun
curl -sk -o /dev/null -w '%{http_code}\n' https://checkout-dev.localtest.me:8443/

What it measured: the values are near 0.67, below 0.95:

kubectl -n checkout-dev get analysisrun -o jsonpath='{.items[-1:].status.metricResults[0].measurements[*].value}'; echo

Undo the bad release in Git, as module 02 taught:

git revert --no-edit HEAD && git push -q
argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 300

Simplified for the course

Prometheus here keeps two days of data in memory and scrapes every pod that asks. The Observability course runs it properly. The analysis only needs the query to answer.

The load generator stands in for users. For a quiet service with little traffic, an analysis has too few requests to judge: use synthetic checks (a job metric that runs tests against the preview) or longer pauses.

Break it on purpose: A label that does not exist

A platform engineer "tidies" the template and writes app= instead of app_kubernetes_io_name=, the way it looks in their other dashboards.

You practise: telling a check that failed from a check that could not run, and reading an AnalysisRun.

1. Break it

cd ~/work/argocd-platform && git switch -q main && git pull -q

In both lines of the query (replace all in your editor):

Change in ~/work/argocd-platform/rollouts/success-rate.yaml

- app_kubernetes_io_name="{{args.app}}"
+ app="{{args.app}}"
git add -A && git commit -qm "success-rate: shorter label" && git push -q
argocd app get rollouts-templates --refresh >/dev/null
cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

- tag: "6.11.0"
+ tag: "6.11.1"
git add -A && git commit -qm "checkout dev: 6.11.1" && git push -q
argocd app get checkout-dev --refresh >/dev/null

2. What you see

Expected: 6.11.1 (a good release) promoted after three minutes. You get: after about two minutes the release aborts, and checkout-dev is Degraded. The new pods were fine: no errors, no restarts.

Look at the newest AnalysisRun: kubectl -n checkout-dev get analysisrun and kubectl -n checkout-dev describe analysisrun <its name>.

3. Fix it yourself

Find why a good release was stopped, fix the check, and release 6.11.1. Done when the newest AnalysisRun is Successful and dev runs 6.11.1.

Hint 1

The run is Error, not Failed. A failed run measured something bad; an errored run could not measure.

Hint 2

The measurements say the condition could not be evaluated on an empty result: the query matched no series. Run the query in Prometheus with app= and with app_kubernetes_io_name=.

The fix
cd ~/work/argocd-platform && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get rollouts-templates --refresh >/dev/null

The template is right again. Release 6.11.1 once more:

kubectl argo rollouts retry rollout checkout -n checkout-dev

After three minutes, expect the newest run Successful:

kubectl -n checkout-dev get analysisrun
After you fixed it: the five questions
What failed?
The release check: a query that matched nothing.
How was it detected?
The AnalysisRun: phase Error, measurements that could not evaluate result[0].
Who responds?
The platform team owns the template; team-checkout noticed.
How is it corrected?
A revert of the template, then retry of the release.
How is it prevented?
Test a template change against a running revision before merge: an AnalysisRun from the template in a scratch namespace, or the query in Prometheus with a real hash.

Quiz

  1. Why does the query filter on rollouts_pod_template_hash?
    • To measure only the new version's pods, not the old ones
    • Because Prometheus needs a unique label per pod
    • To exclude the load generator's requests
  2. An AnalysisRun ends Error. What happened?
    • The new version answered too many errors
    • The query could not be evaluated, several times in a row
    • A person aborted the release
  3. Who should own success-rate and its 95% limit?
    • Each product team, in its chart
    • The platform team, shared by every team
    • Whoever releases, set per release
Answers
  1. To measure only the new version's pods, not the old ones Mixing old and new pods would hide a bad new version behind a good old one.
  2. The query could not be evaluated, several times in a row Failed is a bad measurement; Error is no usable measurement.
  3. The platform team, shared by every team One definition of healthy for everyone; teams choose when to use it.

Field note

The first template every team writes compares error counts, not rates, and the first busy release fails it. Rates over a window, divided by the total, are what make a check fair to a service with ten times more traffic.

Report a problem with this module