Module 10 · Operate with teams · 45 min · You are: Product team

Deployment ordering: hooks, phases and waves

Applying everything at once works until order matters: a schema migration must finish before the new version starts, configuration must exist before what reads it, a project before the Applications in it. Argo CD has phases, hooks and waves for this. Used well, a failed migration means nothing changed instead of half a release.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

Phases

PreSync, Sync, PostSync, and SyncFail when something fails. (PreDelete and PostDelete run around app deletion.)

Hooks

Resources annotated argocd.argoproj.io/hook: <phase>, usually Jobs. Not part of the app's normal state; a delete policy says when to clean them up.

Sync waves

argocd.argoproj.io/sync-wave: "-1": lower waves first. Argo CD waits until a wave is healthy before starting the next.

Waves between Applications

In an app of apps, child Applications have waves too, but only wait if Applications report health. That needs one health check in Argo CD's config.

The order Argo CD applies things in

Hook delete policies: BeforeHookCreation (default: keep the last run until the next), HookSucceeded, HookFailed.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• checkout-app/chart/templates/migration-job.yaml (reference): A PreSync hook: the migration.
  1. Runs before anything else; keeps the last run so you can read its logs L9–10
  2. A switch to make it fail on purpose L24–26
• checkout-app/chart/templates/smoke-test.yaml (reference): PostSync and SyncFail hooks.
  1. After everything is healthy: test the real service L9–10
  2. Only when the sync fails: notify the team L27
• checkout-app/chart/templates/configmap.yaml (reference): A sync wave.
  1. Configuration before the Deployment that reads it L8
• platform/argocd/values.yaml (reference): Health for Applications, so waves between them work.
  1. Report a child Application's own health as its health L23–33
• platform/apps/projects.yaml (reference): A wave in the app of apps.
  1. Projects and namespaces first L8

Steps

1. Platform: let Applications report health

Sync waves can also order whole Applications, for example projects before the apps that use them. That only works if a finished Application reports itself as healthy, and Argo CD needs one small piece of configuration (a Lua health check) for that. Add it to the values file.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Add to ~/work/argocd-platform/argocd/values.yaml under cm:, after the banner lines

    # Module 10: Applications get a health status, so sync waves between
    # Applications (app of apps) wait for the previous wave to be Healthy.
    resource.customizations.health.argoproj.io_Application: |
      hs = {}
      hs.status = "Progressing"
      hs.message = ""
      if obj.status ~= nil and obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
      return hs

Compare your file with the reference. Expect same as the reference (on the kind track). If diff prints lines instead, those lines are where yours differs.

diff argocd/values.yaml ~/work/argocd-gitops/platform/argocd/values.yaml && echo "same as the reference"

git add -A && git commit -qm "Argo CD: health status for Applications" && git push -q

See which Applications already carry a wave number. Expect projects.yaml and namespaces.yaml with a negative wave (they go first):

grep -H 'sync-wave' apps/*.yaml

2. team-checkout: turn on the migration and the smoke test in dev

A hook is a Kubernetes Job that Argo CD runs at a fixed point of a sync. The chart already contains two: a database migration that runs before the deploy, and a smoke test that runs after it. Switch both on in the dev values.

cd ~/work/checkout-config && git switch -q main && git pull -q

Add to ~/work/checkout-config/helm/dev.yaml at the end

migration:
  enabled: true                # module 10
smokeTest:
  enabled: true                # module 10

git add -A && git commit -qm "checkout: migration and smoke test in dev" && git push -q

Expect Sync Status: Synced. Hooks are not part of what Argo CD compares, so adding them changes nothing it compares, and auto-sync has nothing to do.

argocd app get checkout-dev --refresh | grep -E 'Sync Status'

Hooks run when a sync runs, so start one yourself:

argocd app sync checkout-dev --timeout 240

Both hooks left a finished Job behind. Expect checkout-migrate and checkout-smoke-test, each 1/1 complete:

kubectl -n checkout-dev get jobs

Read what the migration did:

kubectl -n checkout-dev logs job/checkout-migrate

3. Read the order of the last sync

Every sync records the order it applied things in. Find three things: the migration in the PreSync phase (first), the ConfigMap in wave -1 (before the Deployment that reads it), and the smoke test in PostSync (last). In the UI: click the sync status, then the operation details.

Print the table of resources from the last sync. sed skips the summary and starts at the table header (GROUP).

argocd app get checkout-dev --show-operation | sed -n '/^GROUP/,$p'

4. Release with a failing migration

Promote 6.7.0 to dev with a migration that fails. The sync stops in PreSync: the Deployment keeps running the old version, and the SyncFail hook runs.

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/helm/dev.yaml

-   tag: "6.7.1"
+   tag: "6.7.0"

Add to ~/work/checkout-config/helm/dev.yaml under migration:, after enabled: true

  fail: true

git add -A && git commit -qm "checkout: 6.7.0 with a broken migration" && git push -q

argocd app get checkout-dev --refresh >/dev/null

Wait about 40 seconds. Expect the sync to have failed: Phase: Failed (or Error) and a message about the PreSync hook.

argocd app get checkout-dev | grep -E 'Phase|Message|Sync Status'

The Deployment was never touched. Expect the image to still end in 6.7.1:

kubectl -n checkout-dev get deploy checkout -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

Both hooks left logs. First the migration that failed, then the SyncFail hook that reported it:

kubectl -n checkout-dev logs job/checkout-migrate
kubectl -n checkout-dev logs job/checkout-sync-failed

5. Recover with a revert

A revert is a change like any other: it goes through a pull request.

cd ~/work/checkout-config

git revert --no-edit HEAD && git push -q

Wait until the sync succeeds and the app is healthy again:

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 240

Simplified for the course

The migration is a sleep. A real one must be safe to run twice and backwards compatible with the running version, because the old pods keep serving while it runs.

Break it on purpose: A smoke test that fails

team-checkout points the smoke test at a new endpoint without trying it first, and syncs to try it. podinfo answers /status/503 with an HTTP 503.

You practise: following a sync through its phases, finding which hook failed and why, and telling "the sync failed" apart from "the app is unhealthy".

1. Break it

cd ~/work/checkout-config && git switch -q main && git pull -q

Add to ~/work/checkout-config/helm/dev.yaml under smokeTest:, after enabled: true

  path: /status/503

git add -A && git commit -qm "checkout: smoke test endpoint" && git push -q

argocd app get checkout-dev --refresh >/dev/null

Start a sync. Hooks are not compared, so only a sync runs them. --async returns at once instead of waiting.

argocd app sync checkout-dev --async

2. What you see

Wait about two minutes. Expected: the sync ends Succeeded, as before. You get: the last sync is Failed, while the app is still Healthy and every pod is running. Something after the deploy failed.

3. Fix it yourself

Find out which phase failed, which hook failed, and the exact error it logged. Check that the SyncFail hook ran. Then fix it so the next sync succeeds.

kubectl -n checkout-dev logs job/checkout-smoke-test | tail -2

kubectl -n checkout-dev logs job/checkout-sync-failed

cd ~/work/checkout-config

git revert --no-edit HEAD && git push -q

argocd app get checkout-dev --refresh >/dev/null
argocd app sync checkout-dev

argocd app get checkout-dev | grep -E 'Phase|Health Status'

Quiz

1. The PreSync migration fails. What happens to the running Deployment?
   • Nothing: the sync stops before the Sync phase
   • The Deployment is updated anyway, because hooks only report and never block the sync
   • The running Deployment is deleted, so that the failed migration can be retried from scratch
2. Two resources have sync-wave -1 and 0. Which goes first, and when does the second start?
   • Both start at once, since waves only order the resources inside a single namespace
   • -1 first; 0 starts once wave -1 is healthy
   • Wave 0 goes first because waves count up from the highest number; -1 starts after it finishes
3. Why does the app of apps need a health check for Applications?
   • Applications cannot be synced at all unless a health check is defined for them, so the sync of the root app fails straight away without one
   • The health check only changes the colours shown in the UI, and the order of the syncs stays exactly the same whether it is defined or not
   • Without it a child Application counts as healthy at once, so waves between Applications do not wait

Field note

Most "we need ordering" requests are really "our app crashes if the database is not ready". Fix that in the app with retries first. Use hooks for things that must happen exactly once per release, like migrations.

Report a problem with this module