Module 03 · Foundations · 45 min · You are: Platform and product

Reconciliation: auto-sync, prune and self-heal

Manual sync tells you about drift; automated sync removes it. With prune and self-heal on, Git is not just the record of what should run, it is the only way to change what runs. That is the moment GitOps starts paying off, and the moment it surprises people, so you will provoke both.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

The loop

Every timeout.reconciliation (60 s here, plus jitter) Argo CD re-reads Git and compares. Live changes are noticed sooner, through watches. A Git webhook makes detection instant.

Automated sync

syncPolicy.automated: when Git changes, apply it without waiting for a person.

Prune

Delete live objects that were removed from Git. Off by default even with auto-sync, so a bad commit cannot delete half an app.

Self-heal

When the live state changes outside Git, put it back. Without it, auto-sync only reacts to new commits.

Sync options worth knowing

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• platform/apps/argocd.yaml (reference): A real Application using the options deliberately. Today, read the sync policy.
  1. Self-heal on, prune off: nobody deletes part of Argo CD by deleting a line L30–33
  2. Server-side apply, because Argo CD's CRDs are too big for client-side apply L34–35
  3. Fields Argo CD must not fight over (module 11) L25–29
• platform/apps/checkout-dev.yaml (reference): The finished checkout Application. Today, only its sync policy matters.
  1. Deleting the Application deletes what it deployed L13–14
  2. Prune and self-heal: Git is the only way to change it L33–36

Steps

1. Drift without auto-sync

Scale by hand. Argo CD notices (OutOfSync, the diff shows replicas) and does nothing: no sync policy yet. Sync it back yourself.

Change the running app by hand, outside Git. This is the drift:

kubectl -n checkout-dev scale deployment checkout --replicas=4

Wait a few seconds. Expect Sync Status: OutOfSync (the cluster no longer matches Git), and the Deployment line marked OutOfSync:

argocd app get checkout-dev | grep -E 'Sync Status|Deployment'

Expect one changed line, replicas: 2 in Git, 4 in the cluster. Argo CD reports it and leaves it alone:

argocd app diff checkout-dev

Put Git's version back yourself:

argocd app sync checkout-dev

2. Turn on automated sync with prune and self-heal

Add an automated block to the sync policy. Commit it and apply it.

Add to ~/work/argocd-platform/apps/checkout-dev.yaml under syncPolicy:, above syncOptions:

    automated:
      prune: true       # delete what was deleted from Git
      selfHeal: true    # undo changes made outside Git

Commit and push the change:

cd ~/work/argocd-platform && git commit -qam "checkout-dev: automated sync, prune, self-heal" && git push -q

Update the Application in the cluster. The Application object in the cluster is what Argo CD reads, not the file in Git, so apply the new version:

kubectl apply -n argocd -f apps/checkout-dev.yaml

Check that it took. Expect Sync Policy: Automated (Prune):

argocd app get checkout-dev | grep -E 'Sync Policy'

3. Self-heal: change it by hand again

Do the same hand change as before. This time Argo CD does not just report it: it puts Git's version back.

kubectl -n checkout-dev scale deployment checkout --replicas=4

Watch the Deployment straight away. The READY column goes to 4/4 and then back to 2/2 within seconds. Press Ctrl-C to stop watching.

kubectl -n checkout-dev get deployment checkout -w

4. Prune: delete something from Git

team-checkout removes the Service from Git. With prune on, Argo CD deletes it from the cluster on the next sync. Then put it back.

Delete the Service file from the repo:

cd ~/work/checkout-config && git switch -q main && git pull -q
git rm -q manifests/service.yaml

git commit -qm "checkout: remove the Service" && git push -q

Ask Argo CD to look at Git now (the output is hidden, only the effect matters):

argocd app get checkout-dev --refresh >/dev/null

A few seconds later the Service is gone from the cluster too. Expect NotFound:

kubectl -n checkout-dev get service checkout

Bring it back the same way, with a commit that undoes the last one:

git revert --no-edit HEAD && git push -q

Look at Git again and wait until the app is Synced and Healthy:

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health

The Service is back:

kubectl -n checkout-dev get service checkout

5. Follow the operation

Every sync is recorded: who or what triggered it (automated here), which revision, which resources changed.

The list of syncs: one row per deploy, with the date and the Git revision that was deployed.

argocd app history checkout-dev

The detail of the latest sync: what started it (look for automated), which revision, and the result for each resource. tail -15 keeps the last 15 lines.

argocd app get checkout-dev --show-operation | tail -15

Simplified for the course

Argo CD polls Git every 60 seconds. In production, add a GitHub webhook to Argo CD so changes are picked up the moment they merge, and keep polling as the fallback.

Break it on purpose: Fix production with kubectl at 2 a.m.

During an incident someone changes the banner on the running Deployment with kubectl (a stand-in for any setting). It looks like it worked.

You practise: recognising self-heal at work, finding the evidence in the history, and making an urgent change that stays.

1. Break it

kubectl -n checkout-dev set env deployment/checkout PODINFO_UI_MESSAGE="hotfix"

Run this right away, and again after 20 seconds:

kubectl -n checkout-dev get deployment checkout -o jsonpath='{.spec.template.spec.containers[0].env[0].value}'; echo

2. What you see

Right away: hotfix. 20 seconds later: checkout (dev) again, and nobody touched it. In the UI, checkout-dev has a new entry under History and rollback that you did not start.

3. Fix it yourself

The incident lead wants the banner to say hotfix until the incident is over. Make it stay. When the incident is over, put the old banner back.

cd ~/work/checkout-config && git switch -q main && git pull -q

-               value: checkout (dev)
+               value: hotfix

git commit -qam "checkout: hotfix banner" && git push -q

argocd app get checkout-dev --refresh >/dev/null; sleep 20
kubectl -n checkout-dev get deployment checkout -o jsonpath='{.spec.template.spec.containers[0].env[0].value}'; echo

git revert --no-edit HEAD && git push -q

Quiz

1. Auto-sync is on but prune is off. You delete a manifest from Git. What happens?
   • The object stays and the app shows OutOfSync (the object would be pruned)
   • The object is deleted from the cluster at the next sync, as prune is implied by auto-sync
   • Argo CD stops with a sync error and refuses to run again until the manifest is added back to Git
2. What does self-heal react to?
   • New commits on the tracked branch, which it applies as soon as they arrive
   • Live changes made outside Git
   • Pods that fail their health checks, which it restarts until the workload turns Healthy again
3. Why would you set Prune=false on a single resource?
   • To make syncs finish faster, since pruning is the slowest phase of a large application
   • To turn off health checks for that one resource, so a broken volume claim does not make the app Degraded
   • So a deleted line in Git can never delete it, for example a volume claim holding data

Field note

The most common incident in a team new to self-heal: "Argo CD keeps reverting my fix". That is not a bug, it is the point. The second most common: a controller and Argo CD fighting over one field. You will meet that one in module 11.

Report a problem with this module