Module 09 · Operate and grow · 40 min · You are: Platform and product

Observability and change visibility

Two questions matter during an incident: what is happening, and what changed. Logs from every cluster in one place answer the first; Argo CD links, deployment records and trace.sh answer the second.

Open this module in the interactive course to track progress and switch tracks.

Change journey practised: Requested → Proposed → Validated → Previewed → Approved → Merged → Reconciled → Healthy

Concepts

Agents per cluster, store on the hub

Alloy runs on every node and pushes to the hub. One Grafana for the fleet.

Reachability is a fleet decision

Spokes cannot resolve the hub's cluster-local DNS. The hub exposes the Loki gateway (a fixed NodePort here, an internal load balancer in a cloud) and each inventory file says where to push: lokiEndpoint.

Labels everywhere

Logs carry cluster and environment; Applications carry fleet.gitops/product, owner, environment. The same words in every tool.

From running to change

Argo CD shows the synced commit and links to product.yaml and the config history; the namespace links to the onboarding issue.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• applicationsets/observability-agents-appset.yaml (reference, your repo): The log agent on every cluster.
  1. Every log line gets the cluster name L68–69
  2. Where to push: taken from the inventory L85
• applicationsets/platform-observability.yaml (reference, your repo): Loki and Grafana on the hub.
  1. Loki is reachable from the spokes on a fixed NodePort L49–51
  2. Grafana reads from Loki L87–95
• clusters/dev.yaml (reference, your repo): Where dev ships its logs.
  1. The hub node by name, on the Loki NodePort L11
• docs/observability-topology.md (reference, your repo): Why spokes cannot use cluster-local DNS.
  1. The reachability decision in one paragraph L56–64

Steps

1. Propose observability

Two parts: Loki and Grafana on the hub (where logs are stored and viewed), and a log agent, Alloy, on every cluster (which sends them).

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c observability

Copy the two ApplicationSets and the docs, and point them at your repo:

git checkout reference/main -- applicationsets/observability-agents-appset.yaml applicationsets/platform-observability.yaml docs
scripts/set-repo-url.sh

List the new Applications. Expect Loki and Grafana for the hub, and an agent Application for each cluster.

python3 scripts/fleet_check.py | grep -E 'observability|loki|grafana'

git add -A && git commit -m "Observability: logs from every cluster to the hub"
git push -u origin HEAD

Open the pull request ↗

2. Open Grafana and query the fleet

After merge and sync, open http://localhost:3000 (user admin). In Explore, query {cluster="prod", namespace="checkout"}, then {namespace="checkout"} to see dev and prod together.

Print the Grafana password and copy it:

kubectl --context kind-platform -n observability get secret grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo

In a new terminal, start the port-forward and leave it running. Then open http://localhost:3000.

kubectl --context kind-platform -n observability port-forward svc/grafana 3000:80

3. Walk back from a running app to its change

In the Argo CD UI open checkout-prod: the commit it runs, and the links product.yaml, config and history. Then the namespace's request annotation and the Deployments page.

The namespace remembers the issue that asked for it. Expect the URL of your onboarding issue:

kubectl --context kind-prod get namespace checkout -o jsonpath='{.metadata.annotations.fleet\.gitops/request}'; echo

And the trace shows which commit each prod Application has applied:

scripts/trace.sh --context kind-platform --selector fleet.gitops/environment=prod

Simplified for the workshop

Grafana uses its generated admin password and a port-forward. In production it sits behind SSO with per-team folders, and alerting (including "no logs from cluster X for 10 minutes") is part of the platform.

Break it on purpose: Cut the log route for dev

A pull request "moves" dev's Loki endpoint to a new port. The checks pass: it is valid YAML pointing somewhere.

You practise: noticing missing data, debugging the log pipeline from the agent's side, and the limits of what CI can check.

1. Break it

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c break-logs

Change in ~/work/fleet-gitops/clusters/dev.yaml

- lokiEndpoint: http://platform-control-plane:31100/loki/api/v1/push
+ lokiEndpoint: http://platform-control-plane:31199/loki/api/v1/push

git add -A && git commit -m "dev: move Loki endpoint"
git push -u origin HEAD

Open the pull request, merge it, and wait a few minutes.

2. What you see

In Grafana Explore, {cluster="prod"} keeps showing new lines, but {cluster="dev"} stops at around the time you merged. No check failed and every Application is Healthy. Expected: logs from both clusters.

3. Fix it yourself

Find the cause from the cluster side first: what does dev's log agent say? Then get dev's logs flowing again.

kubectl --context kind-dev -n observability-agents logs -l app.kubernetes.io/name=alloy --tail 30 | grep -i error

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c restore-logs && git revert --no-edit HEAD && git push -u origin HEAD

Quiz

1. Why is lokiEndpoint in the inventory instead of the Alloy template?
   • How a spoke reaches the hub depends on the cluster's network, so it is a per-cluster fact
   • It is only a YAML style choice; the endpoint could just as well sit inside the Alloy template
   • Alloy requires the endpoint in the inventory, and fails to start if it finds it anywhere else
2. How do you find the onboarding request for a running namespace?
   • Ask around the team until someone remembers who requested it and when it was created
   • The fleet.gitops/request annotation on the namespace
   • Run kubectl logs on the Argo CD controller and search for the namespace name
3. CI passed, but logs stopped. What does that teach?
   • CI is useless, so the checks should be removed and replaced with manual review of every change before it merges to the main branch
   • Grafana is broken, and restarting it brings the missing logs back without any change to the repository or the cluster configuration
   • Validation proves a change is well-formed, not that it works; runtime signals and alerts are the other half

Field note

Name things the same everywhere. When the label in Grafana, the Application label in Argo CD and the folder name in Git are all checkout, people find things in seconds.

Report a problem with this module