Module 08 · Operate and grow · 60 min · You are: Platform and product

Operate the fleet: failure drills

In this module

You learn: What a failure costs on a GitOps platform. For each of five drills you answer: what failed, how it was detected, who responds, how it is corrected and how it is prevented.
You do: Break the fleet five ways: delete a namespace, scale prod by hand, try to change the platform baseline as a product team, take a cluster offline, and delete the hub. Then write up the drill that taught you most.
You end with: Everything is back to normal: the fleet healed itself or was rebuilt from code. You have an incident note in your repo, and you know who responds to each kind of failure.

You learn a platform by watching it fail safely. Each drill below is a failure we have seen in real fleets, chosen because the platform's design (Git, self-heal, policies, retries, Terraform) changes what the failure costs.

Open this module in the interactive course to track progress and switch tracks.

Concepts

Detection: CI for proposed changes, Argo CD status for drift and sync failures, Kyverno for admission, Terraform plan for infrastructure drift, logs and health for the rest.
Ownership: The drill tables name the responder. Product failures go to product teams; platform failures to the platform team. Mixing them up is the most expensive incident pattern.
Correction through Git: Every fix that changes desired state is a pull request. Fixes that restore desired state (self-heal, retries, rebuilds) need no human.
Prevention: Each drill ends with the control that stops it from recurring. If there is none, that is the follow-up issue.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

docs/operating-model.md (reference, your repo): The drills and the first commands to reach for.
1. Every drill: detection, responder, fix, prevention L46–59
2. Commands to reach for first L61–72

Steps

1. Drill 1: a missing dependency heals itself

Delete the checkout namespace on dev. The app needs it; the baseline creates it.

Delete the namespace. --wait=false returns at once instead of waiting for the deletion:

kubectl --context kind-dev delete namespace checkout --wait=false

Watch Argo CD. Expect checkout-dev to go OutOfSync, fail while the namespace is missing, retry, and then return to Synced and Healthy by itself. Press Ctrl-C when it has.

kubectl --context kind-platform -n argocd get applications -l fleet.gitops/product=checkout -w

What failed?: checkout-dev lost its namespace (and everything in it).
How was it detected?: Argo CD: both checkout Applications OutOfSync/Missing.
Who responds?: Nobody needs to act; the platform team investigates who deleted it.
How is it corrected?: Self-heal recreates the namespace baseline; checkout-dev fails, retries with backoff, then syncs.
How is it prevented?: The namespace is a separate, platform-owned Application, and protect-platform-baseline blocks product teams from deleting it.

2. Drill 2: drift in prod

Someone scales prod by hand during an incident and forgets about it.

Scale prod by hand:

kubectl --context kind-prod -n checkout scale deployment checkout --replicas=5

Watch the Deployment. Expect 5 replicas for a few seconds and then back to 2. Press Ctrl-C afterwards.

kubectl --context kind-prod -n checkout get deployment checkout -w

What failed?: Live state differs from Git.
How was it detected?: Argo CD OutOfSync.
Who responds?: Argo CD reverts it; the product team should make a real change if 5 replicas were needed.
How is it corrected?: Self-heal back to Git within seconds.
How is it prevented?: If scaling is legitimately dynamic, use an HPA in Git and let Argo CD ignore replicas.

3. Drill 3: an unauthorized baseline change

Act as team-checkout: delete the default-deny network policy, then try to raise your own quota.

--as and --as-group make kubectl act as a member of team-checkout. First try to delete the default-deny network policy. Expect a refusal from Kyverno, naming protect-platform-baseline.

kubectl --context kind-prod -n checkout delete networkpolicy default-deny-ingress --as=dev-user --as-group=team-checkout

Now try to raise your own quota. Expect forbidden: the team's role cannot change quotas.

kubectl --context kind-prod -n checkout patch resourcequota product-quota --type=merge -p '{"spec":{"hard":{"pods":"500"}}}' --as=dev-user --as-group=team-checkout

The same user can still do its job. Expect a list of pods:

kubectl --context kind-prod -n checkout get pods --as=dev-user --as-group=team-checkout

What failed?: A product team tried to change the platform-owned baseline.
How was it detected?: Kyverno denies the delete (protect-platform-baseline); RBAC denies the quota patch (edit cannot change quotas).
Who responds?: The product team.
How is it corrected?: A pull request changing size or asking the platform for a policy change.
How is it prevented?: Policy plus RBAC plus the AppProject blacklist: three layers, each visible in Git.

4. Drill 4: a cluster becomes unavailable

Stop the dev control plane, look at Argo CD, start it again.

Stop the dev cluster:

docker stop dev-control-plane

Look at Argo CD. After a minute, expect dev's Applications to show Unknown: Argo CD cannot reach the cluster, so it cannot compare.

kubectl --context kind-platform -n argocd get applications | grep -E '^(NAME|dev-|checkout-dev)'

Start it again, and watch the Applications recover by themselves. Press Ctrl-C when they are Synced.

docker start dev-control-plane

kubectl --context kind-platform -n argocd get applications -w

On AKS, EKS or GKE

# Cloud track: scale the dev node pool to zero, or block the API server with an
# authorized-IP rule, then undo it. Watch Argo CD the same way.

What failed?: A spoke API server is unreachable.
How was it detected?: Argo CD cluster connection state; Applications turn Unknown. Alert on it.
Who responds?: The platform team.
How is it corrected?: Restore the cluster; Argo CD resyncs by itself. If it is gone for good: terraform apply (module 02).
How is it prevented?: Nothing on the cluster depends on manual steps, so recovery is a rebuild, not a reconstruction.

5. Drill 5: lose the hub

The big one. Delete the hub and rebuild it. Nothing about the fleet lives only on the hub.

Delete the hub:

kind delete cluster --name platform

Rebuild it from code. Terraform recreates the hub, Argo CD, the cluster registrations and the root app. Type yes when asked.

cd ~/work/fleet-gitops/infrastructure/kind && terraform apply

Watch Argo CD re-adopt every Application from Git. Press Ctrl-C when they are Synced and Healthy.

kubectl --context kind-platform -n argocd get applications -w

On AKS, EKS or GKE

# Cloud track: read along, or recreate the hub with your cloud scripts.

What failed?: The management cluster is gone: no Argo CD, no UI.
How was it detected?: Everyone notices. Workloads on the spokes keep running.
Who responds?: The platform team.
How is it corrected?: terraform apply. Argo CD re-adopts every Application from Git; the spokes see no change.
How is it prevented?: The hub is stateless. Rehearse this drill on a schedule.

6. Write the incident note

Pick the drill that taught you the most and write it up as a short issue in your repo. Link the commits or pull requests. Writing it right after the incident, while you still remember what you saw first, is the habit that makes an operating model real.

Done when:

Five lines: what failed, how it was detected, who responded, how it was corrected, how it is prevented
The detection line names the first signal you actually saw, not the cause you found later
The prevention line links a control that exists, or a follow-up issue

Open a new issue ↗

Quiz

Who responds when a product team's pod uses :latest and is rejected in prod?
- The product team
- The platform team, since it owns the Kyverno policies and the cluster that rejected it
- Argo CD, which retries the sync until the image tag is accepted by the policy
What does Argo CD need to rebuild the fleet after losing the hub?
- A recent backup of etcd from the old hub, restored onto the new one before anything else
- Git and the cluster registrations (Terraform recreates both)
- Nothing can be rebuilt; losing the hub means starting the whole fleet from scratch
Why is the namespace a separate Application from the product workload?
- For performance: one big Application containing both would take too long to compare and sync, so the work is split into two that run faster
- Argo CD requires every namespace to be its own Application, and refuses to sync a product workload that shares an Application with one
- Different owners and lifecycles: the platform baseline must exist and stay protected regardless of the product config

Answers

The product team The workload config is product-owned. The platform provided the guardrail and the error message.
Git and the cluster registrations (Terraform recreates both) Desired state lives in Git; the hub is stateless by design.
Different owners and lifecycles: the platform baseline must exist and stay protected regardless of the product config Separate ownership; retries make the ordering converge.

Field note

Run drill 5 before you need it. The team that has rebuilt its hub on a quiet Tuesday handles the real one calmly.

Report a problem with this module