Module 11 · Operate with teams · 90 min · You are: Platform and product

Debugging labs: seven ways Argo CD goes wrong

Every lab here is a failure I have seen in real teams. Try each one before opening the hints: the skill is reading Argo CD's signals (sync status, health, conditions, operation results, events) and knowing which layer to look at next.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

Start at the Application

argocd app get shows sync, health, conditions and the last operation. Most answers are already there.

One layer down at a time

Application, then the resource tree, then the object's events, then pod logs, then Argo CD's own logs.

Render before you guess

argocd app manifests and argocd app diff show exactly what Argo CD will apply and what differs.

Fix through Git

Revert is the default fix. A kubectl fix is only for restoring service while the real fix goes through Git.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• docs/debugging.md (reference): The runbook: symptoms to causes.
  1. Keep this table open during the labs L17–27
• labs/stuck-finalizer/configmap.yaml (reference): The object lab 6 deploys.
  1. A finalizer whose controller no longer exists L7–8
• checkout-app/chart/templates/deployment.yaml (reference): Why lab 7 has a clean fix.
  1. With autoscaling on, the chart leaves replicas to the HPA L7–9

Steps

1. Lab 1: merged, but prod stays OutOfSync (Debug lab)

The platform team announces a change freeze (in Git, of course). Then team-checkout merges a banner change for prod. It never arrives. Run the commands, then find out why without looking at the first line again.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Add to ~/work/argocd-platform/projects/checkout.yaml under spec:, above orphanedResources:
As the platform team.

  syncWindows:
    - kind: deny
      schedule: "* * * * *"
      duration: 1h
      applications: ["checkout-prod"]

git add -A && git commit -qm "Change freeze for checkout-prod" && git push -q

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/prod.yaml
As team-checkout.

- message: checkout (prod)
+ message: checkout (prod) - new banner

git add -A && git commit -qm "checkout: new prod banner" && git push -q

Wait 30 seconds, then ask Argo CD what it sees:

sleep 30; argocd app get checkout-prod --refresh | grep -E 'Sync Status|Window|Message'

What you see.

The banner change is merged, but checkout-prod stays OutOfSync and never syncs, although auto-sync is on. No error, no failed sync.

Fix it yourself. Find what is holding the change back, without reading the commands above again. Then let the change through the GitOps way.

cd ~/work/argocd-platform && git switch -q main && git pull -q

-   syncWindows:
-     - kind: deny
-       schedule: "* * * * *"
-       duration: 1h
-       applications: ["checkout-prod"]

git add -A && git commit -qm "End change freeze" && git push -q

argocd app wait checkout-prod --sync --health --timeout 180

2. Lab 2: Synced, then Degraded (Debug lab)

team-checkout promotes a version to dev. Argo CD syncs it happily. Two minutes later the app is Degraded.

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

-   tag: "6.7.1"
+   tag: "6.7.99"

git add -A && git commit -qm "checkout: 6.7.99 to dev" && git push -q

Wait two and a half minutes, then look at what Argo CD says:

argocd app get checkout-dev --refresh >/dev/null; sleep 150
argocd app get checkout-dev | grep -E 'Sync Status|Health Status|Deployment'

What you see.

Argo CD applied it: Synced. Two minutes later checkout-dev is Degraded, and the old pods are still serving.

Fix it yourself. Find which resource is unhealthy and why, then roll back.

cd ~/work/checkout-config

git revert --no-edit HEAD && git push -q

argocd app wait checkout-dev --health --timeout 180

3. Lab 3: invalid manifests, two kinds (Debug lab)

Two broken commits to dev values, one after the other. They fail differently: notice where.

cd ~/work/checkout-config && git switch -q main && git pull -q

Add to ~/work/checkout-config/helm/dev.yaml at the end, exactly like this
(a) Not even valid YAML: an unclosed list, and a three-space indent.

resources:
  limits:
   memory: 128Mi
  requests: [

git commit -qam "checkout: dev limits" && git push -q

Wait 20 seconds and look. Note where it failed:

argocd app get checkout-dev --refresh | grep -E 'Sync Status|Error'

Undo (a), then make the second mistake:

git revert --no-edit HEAD && git push -q

Add to ~/work/checkout-config/helm/dev.yaml at the end
(b) Valid YAML, but not a valid Kubernetes quantity.

resources:
  limits:
    memory: lots

git commit -qam "checkout: more memory" && git push -q

Wait 30 seconds and look again. Note where this one failed:

sleep 30; argocd app get checkout-dev --refresh | grep -E 'Sync Status|Health|Message'

What you see.

(a) Sync status Unknown with an error: nothing was applied. (b) Argo CD shows new manifests and tries to sync, but the sync fails.

Fix it yourself. For each mistake, say which part of Argo CD caught it (rendering or applying) and quote the error. Then get checkout-dev healthy again.

cd ~/work/checkout-config && git revert --no-edit HEAD && git push -q

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health

4. Lab 4: missing permissions (Debug lab)

As team-checkout: add a NetworkPolicy to your own namespace through Git, then try to clean up an old app from the CLI.

cd ~/work/checkout-config && git switch -q main && git pull -q

Create ~/work/checkout-config/secrets/dev/allow-all.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
spec:
  podSelector: {}
  ingress: [{}]

git add -A && git commit -qm "checkout: open network" && git push -q

Now work as team-checkout from the CLI:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username checkout-dev --password 'another-long-one'

Wait 20 seconds and see what Argo CD says about your change:

sleep 20; argocd app get checkout-dev --refresh | grep -iE 'permitted|Sync Status'

Then try the clean-up:

argocd app delete checkout-dev --yes

What you see.

Two refusals: checkout-dev reports the NetworkPolicy as not permitted, and the delete command is denied.

Fix it yourself. Say which refusal came from the AppProject and which from Argo CD RBAC, and why each one is right. Then clean up as team-checkout and log back in as alice.

cd ~/work/checkout-config && git rm -q secrets/dev/allow-all.yaml && git commit -qm "Remove NetworkPolicy" && git push -q

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'

5. Lab 5: repository authentication (Debug lab)

Someone tidies up the deploy keys on checkout-config. Nothing changes... until Argo CD next needs to fetch.

The tidy-up: find the deploy key called argocd and delete it.

ID=$(gh repo deploy-key list -R YOUR_GITHUB_USER/checkout-config --json id,title -q '.[] | select(.title=="argocd") | .id')
gh repo deploy-key delete "$ID" -R YOUR_GITHUB_USER/checkout-config

Make Argo CD fetch from Git again, ignoring its cache:

argocd app get checkout-dev --hard-refresh | grep -iE 'Sync Status|Error|auth'

And ask what it thinks of its repositories:

argocd repo list

What you see.

checkout-dev is still Healthy: the pods run. But the hard refresh reports an error fetching the repository, and argocd repo list shows the connection failing.

Fix it yourself. Find which credential broke and restore it, without changing anything in Argo CD.

gh repo deploy-key add ~/work/keys/checkout-config.pub -R YOUR_GITHUB_USER/checkout-config --title argocd

argocd app get checkout-dev --hard-refresh | grep -E 'Sync Status'

argocd repo list

6. Lab 6: an Application that will not delete (Debug lab)

A throwaway Application from the reference repo's labs/ folder (applied by hand: it is a lab, not part of your platform). Delete it and watch it hang.

Create the throwaway Application. Everything between <<'EOF' and EOF is a YAML file passed straight to kubectl apply, so nothing is saved on disk.

cat <<'EOF' | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: lab-finalizer
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/platformengr/argocd-gitops.git
    targetRevision: main
    path: labs/stuck-finalizer
  destination:
    server: https://kubernetes.default.svc
    namespace: lab
  syncPolicy:
    automated: {}
    syncOptions: [CreateNamespace=true]
EOF

Wait until it is healthy:

argocd app wait lab-finalizer --health --timeout 120

Now delete it. --wait=false returns at once instead of waiting for the deletion to finish:

kubectl -n argocd delete application lab-finalizer --wait=false

Wait 20 seconds and look:

sleep 20; argocd app get lab-finalizer | grep -E 'Sync Status|Health'

What you see.

lab-finalizer is still there after 20 seconds, and it stays: deleting, but never gone.

Fix it yourself. Find what the deletion is waiting for and unblock it safely. Do not remove Argo CD's own finalizer.

kubectl -n lab patch configmap legacy-config --type=json -p '[{"op":"remove","path":"/metadata/finalizers"}]'

kubectl -n argocd get application lab-finalizer

kubectl delete namespace lab

7. Lab 7: OutOfSync again, right after every sync (Debug lab)

An on-call engineer adds an autoscaler from a runbook with kubectl. Now checkout-dev flips between Synced and OutOfSync and the replica count jumps.

The runbook fix: add an autoscaler by hand.

kubectl -n checkout-dev autoscale deployment checkout --min=2 --max=4 --cpu-percent=80

Print the replica count every 10 seconds for a minute:

for i in 1 2 3 4 5 6; do kubectl -n checkout-dev get deploy checkout -o jsonpath='{.spec.replicas} '; sleep 10; done; echo

What does Argo CD say is different?

argocd app diff checkout-dev

argocd app get checkout-dev | grep -iE 'orphan|Sync Status'

What you see.

The replica count jumps between values, and checkout-dev flips between Synced and OutOfSync after every sync.

Fix it yourself. Find the two things fighting over one field, decide which should own it, and make that permanent in Git.

kubectl -n checkout-dev delete hpa checkout

cd ~/work/checkout-config && git switch -q main && git pull -q

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 4

git add -A && git commit -qm "checkout: autoscaling in dev" && git push -q

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health

kubectl -n checkout-dev get hpa,deploy

8. Write the runbook entry

Pick the lab that surprised you most and write it up in your platform repo, so the next person finds the answer in a minute. This is how a team's runbook gets written: one incident at a time, right after it.

Done when:

• The symptom is written the way it looked, not the way you now understand it
• Someone who has not done the lab could follow "Where to look" and "Fix"
• The owner is a team, not a person

cd ~/work/argocd-platform && git switch -q main && git pull -q

Create ~/work/argocd-platform/docs/runbook.md
Fill in every line from the lab you picked.

# Runbook

## <the symptom, as you first saw it>

- **Where to look:** <the command or UI screen that showed the cause>
- **Cause:** <one sentence>
- **Fix:** <the commit or pull request that fixed it>
- **Prevention:** <the control that stops it next time, or the issue that tracks it>
- **Owner:** <the team that responds>

git add -A && git commit -qm "docs: first runbook entry" && git push -q

Quiz

1. An Application is Unknown with a ComparisonError. What is true?
   • Argo CD could not render or read the source; nothing was applied, the running workload is untouched
   • The workload is down: Argo CD could not render the source, so it removed the pods that it could no longer compare with Git
   • Argo CD crashed while comparing, so the whole application-controller needs a restart before any Application can be refreshed again
2. An app goes OutOfSync again right after every sync, always on the same field. Likely cause?
   • A slow network between Argo CD and the cluster makes every sync look incomplete
   • Another controller changes that field
   • A bug in Argo CD's diff engine, which a newer version will fix with no change on your side
3. An Application is stuck deleting. What should you not do first?
   • Look for resources with deletionTimestamp and finalizers that block the deletion
   • Read the events and the controller logs to see which object is holding things up
   • Remove resources-finalizer.argocd.argoproj.io from the Application

Field note

When I run these labs live, the fastest teams are not the ones who know the most flags. They are the ones who read the whole argocd app get output before typing anything else.

Report a problem with this module