Cheat notes · Module 04 · Releases judged by metrics
Metrics and analysis templates
A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.
The idea
Analysis lets metrics decide instead of a person. The platform team writes the checks once as ClusterAnalysisTemplates; each one asks Prometheus a question about the new pods only, by their pod-template hash, and says what a good answer is. A Rollout runs them and promotes, waits or aborts on their verdict.
Remember
- Prometheus scrapes every pod with
prometheus.io/scrape: "true", keeping the labelsapp_kubernetes_io_nameandrollouts_pod_template_hash. - The load generator sends a few requests a second, so a run has data when nobody uses the app.
- ClusterAnalysisTemplate: shared by every namespace; a Rollout uses it with
clusterScope: true. new-hashcomes frompodTemplateHashValue: Latest: the check measures only the new version.- Phases:
Successfulpromotes,Failedaborts,Error(no data, query broken) aborts afterconsecutiveErrorLimit,Inconclusivepauses. prePromotionAnalysisin blue-green: the preview must pass before the switch.
Pitfalls
- A query that returns nothing is not a pass: it is an
Error, and the release aborts. Check the labels in Prometheus first. - Measuring the whole app instead of the new pods. Without the hash, the stable pods hide a bad canary.
- No traffic, no data. A quiet app needs the load generator or a longer interval.
Commands
kubectl -n monitoring port-forward svc/prometheus-server 9090:80 | Prometheus at localhost:9090 |
kubectl get clusteranalysistemplates | The platform's shared checks |
kubectl -n checkout-dev get analysisrun | Runs and their phase |
kubectl describe analysisrun RUN -n checkout-dev | Every measurement and why it passed or failed |
argocd-platform platform/rollouts/success-rate.yaml (excerpt)
kind: ClusterAnalysisTemplate
metadata:
name: success-rate
spec:
args: [{ name: namespace }, { name: app }, { name: new-hash }]
metrics:
- name: success-rate
interval: 30s
count: 6
failureLimit: 1
successCondition: result[0] >= 0.95
provider:
prometheus:
address: http://prometheus-server.monitoring.svc.cluster.local
query: |
sum(rate(http_requests_total{..., status=~"2.."}[1m]))
/
sum(rate(http_requests_total{...}[1m]))
Test yourself
Answer in your head first, then open the answer.
Why filter on rollouts_pod_template_hash?
So the check measures only the new version, not the stable pods next to it.
The query returns an empty result. What happens?
Each measurement is an error; after the error limit the run is Error and the release aborts.
Why a ClusterAnalysisTemplate instead of an AnalysisTemplate?
One check, written once by the platform team, used by every team's namespace.
Where module 04 leaves you: Prometheus scrapes checkout, a load generator sends it traffic, the success-rate ClusterAnalysisTemplate exists, and dev promotes itself when the preview's success rate is at least 95%. Dev runs 6.10.0.