PlatformEngr® Progressive delivery course

Cheat notes · Module 04 · Releases judged by metrics

Metrics and analysis templates

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

Analysis lets metrics decide instead of a person. The platform team writes the checks once as ClusterAnalysisTemplates; each one asks Prometheus a question about the new pods only, by their pod-template hash, and says what a good answer is. A Rollout runs them and promotes, waits or aborts on their verdict.

Remember

  • Prometheus scrapes every pod with prometheus.io/scrape: "true", keeping the labels app_kubernetes_io_name and rollouts_pod_template_hash.
  • The load generator sends a few requests a second, so a run has data when nobody uses the app.
  • ClusterAnalysisTemplate: shared by every namespace; a Rollout uses it with clusterScope: true.
  • new-hash comes from podTemplateHashValue: Latest: the check measures only the new version.
  • Phases: Successful promotes, Failed aborts, Error (no data, query broken) aborts after consecutiveErrorLimit, Inconclusive pauses.
  • prePromotionAnalysis in blue-green: the preview must pass before the switch.

Pitfalls

  • A query that returns nothing is not a pass: it is an Error, and the release aborts. Check the labels in Prometheus first.
  • Measuring the whole app instead of the new pods. Without the hash, the stable pods hide a bad canary.
  • No traffic, no data. A quiet app needs the load generator or a longer interval.

Commands

kubectl -n monitoring port-forward svc/prometheus-server 9090:80Prometheus at localhost:9090
kubectl get clusteranalysistemplatesThe platform's shared checks
kubectl -n checkout-dev get analysisrunRuns and their phase
kubectl describe analysisrun RUN -n checkout-devEvery measurement and why it passed or failed

argocd-platform platform/rollouts/success-rate.yaml (excerpt)

kind: ClusterAnalysisTemplate
metadata:
  name: success-rate
spec:
  args: [{ name: namespace }, { name: app }, { name: new-hash }]
  metrics:
    - name: success-rate
      interval: 30s
      count: 6
      failureLimit: 1
      successCondition: result[0] >= 0.95
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{..., status=~"2.."}[1m]))
            /
            sum(rate(http_requests_total{...}[1m]))

Test yourself

Answer in your head first, then open the answer.

Why filter on rollouts_pod_template_hash?

So the check measures only the new version, not the stable pods next to it.

The query returns an empty result. What happens?

Each measurement is an error; after the error limit the run is Error and the release aborts.

Why a ClusterAnalysisTemplate instead of an AnalysisTemplate?

One check, written once by the platform team, used by every team's namespace.

Where module 04 leaves you: Prometheus scrapes checkout, a load generator sends it traffic, the success-rate ClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.