Cheat notes · Module 09 · Operate and grow

Observability and change visibility

A five-minute review for when you have done this module and want it back in your head. Open the full module for the steps and the lab, or the full text to print.

The idea

An incident asks two questions: what is happening, and what changed. Logs from every cluster in one Grafana answer the first; Argo CD links, deployment records and the trace script answer the second.

Remember

Agents per cluster, store on the hub: Alloy runs on every cluster and pushes logs to Loki on the hub; one Grafana for the fleet.
Reachability is a fleet fact: spokes cannot resolve the hub's cluster-local DNS, so the hub exposes the Loki gateway (a fixed NodePort here) and each inventory file sets lokiEndpoint.
Same labels everywhere: logs carry cluster and environment; Applications carry fleet.gitops/product, owner, environment.
From running app to change: Argo CD shows the synced commit and links to product.yaml and the config history; the namespace's fleet.gitops/request annotation links the onboarding issue; GitHub Deployments list who changed what.
CI proves shape, not function: an address can be valid YAML and still lead nowhere. Alert when a cluster stops sending logs.

Pitfalls

A pull request "moves" a cluster's Loki endpoint to a port where nothing listens: every check passes, every Application is Healthy, and that cluster's logs stop. Read the agent's logs; revert.
Naming things differently in Grafana, Argo CD and Git. When all three say checkout, people find things in seconds.

Commands

`kubectl --context kind-platform -n observability port-forward svc/grafana 3000:80`	Open Grafana at localhost:3000
`{cluster="prod", namespace="checkout"}`	LogQL in Grafana Explore: one product in prod (drop `cluster` for the whole fleet)
`kubectl --context kind-prod get namespace checkout -o jsonpath='{.metadata.annotations.fleet\.gitops/request}'`	The issue that asked for this namespace
`kubectl --context kind-dev -n observability-agents logs -l app.kubernetes.io/name=alloy --tail 30`	Debug a log route from the agent's side

Test yourself

Answer in your head first, then open the answer.

Why is lokiEndpoint in the inventory and not in the Alloy template?

How a spoke reaches the hub depends on its network: a per-cluster fact. Templates stay identical.

How do you find the onboarding request for a running namespace?

The fleet.gitops/request annotation on the namespace.

CI passed but logs stopped. What does that teach?

Validation proves a change is well-formed, not that it works. Runtime signals and alerts are the other half.

Where module 09 leaves you: Alloy agents on every cluster ship logs to Loki on the hub, Grafana shows both clusters, and you can trace a running app back to its pull request. dev\'s log route is back after the exercise.