PlatformEngr® Argo CD course

Cheat notes · Argo CD

Argo CD: cheat notes

Every module on one page, cut to what you need to remember: the idea, the commands, the pitfalls and a few questions to test yourself. Made for review after you have done the course, not instead of it. Back to the course.

00 Orientation: what Argo CD is and where it fits

This module only · Full module

The idea

Argo CD is a controller inside the cluster that pulls the desired state from Git and keeps the cluster equal to it. CI builds and tests; it never touches the cluster.

Remember

  • Pull, not push: Argo CD reads Git with read-only access. No pipeline holds cluster credentials, and every running object has a commit behind it.
  • Reconcile, not apply once: a CI job running kubectl apply deploys and forgets. Argo CD keeps comparing, shows drift and can correct it.
  • Three repos, three jobs: argocd-platform (platform team: Argo CD, Applications, projects), checkout-app (team-checkout: chart and code, released as tags), checkout-config (team-checkout: what runs in each environment).
  • The environment repo decides what runs where. The app repo produces releases; a change in checkout-config is a deploy.
  • The layers: code and build (product team), desired state in Git, delivery (Argo CD), guardrails (AppProjects, RBAC, platform-owned namespaces), runtime (Kubernetes).
  • The stand-in product: checkout runs the public podinfo image, so the course needs no code and no CI.

Pitfalls

  • Fixing things with kubectl edit "just this once". Argo CD undoes it and nobody knows it happened. Changing Git, even when kubectl is quicker, is the habit the course builds.
  • Treating checkout-config as the product's code. It holds only the Kubernetes files that say how the product runs.

Test yourself

Answer in your head first, then open the answer.

What does Argo CD do that a CI job running kubectl apply does not?

It keeps comparing the cluster with Git after the deploy, shows drift and can correct it.

Why is pull-based delivery safer?

The credentials that can change production stay inside the cluster. Argo CD only needs read access to Git.

Which repository says which version runs in prod?

The environment repository, checkout-config. The app repo only produces releases.

Where module 00 leaves you: Nothing is installed in a cluster yet. You have your details saved in the course, every tool checked, a local copy of the reference repo to copy from, and a clear picture of what you will build.

01 Install Argo CD the reviewable way

This module only · Full module

The idea

Install Argo CD the way you will run it: a pinned Helm chart, its settings in a values file in Git, nothing changed by hand. Argo CD is a few controllers plus new object types (CRDs) in your cluster.

Remember

  • Components: argocd-server (UI and API), argocd-repo-server (clones Git, renders Helm and Kustomize), argocd-application-controller (compares and syncs), Redis (cache), plus the ApplicationSet and notifications controllers.
  • CRDs: Application (what to deploy, from where, to where), AppProject (the fence around Applications), ApplicationSet (generates Applications).
  • Values file in Git: argocd/values.yaml holds Argo CD's settings. The chart renders them into ConfigMaps such as argocd-cm.
  • Pinned chart version: the version is part of the command (later of a manifest), so installs repeat exactly and upgrades are deliberate.
  • admin: the install creates it with a random password in argocd-initial-admin-secret. Fine to start; module 06 replaces it with personal accounts.
  • The port-forward: until module 06 the UI and the CLI only work while kubectl -n argocd port-forward svc/argocd-server 8080:443 runs. The CLI login lasts 24 hours.

Pitfalls

  • + New App in the UI creates an Application that exists nowhere in Git. Use the UI to watch; change things through Git.
  • A hand edit of Argo CD's ConfigMap makes the next helm upgrade fail with conflict ... with "kubectl-patch": Kubernetes records each field's manager. Put the value in values.yaml, then take the field back once with --force-conflicts.
  • connection refused from argocd means the port-forward stopped. Unauthenticated means the login expired: log in again.

Commands

helm install argocd argo/argo-cd --version 10.10.5 -n argocd --create-namespace -f argocd/values.yaml --waitInstall Argo CD: this chart, this version, your values
kubectl get crd | grep argoproj.ioThe three object types Argo CD adds
kubectl -n argocd get deployments,statefulsetsThe components. First list to check when Argo CD misbehaves
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -dThe initial admin password
kubectl -n argocd port-forward svc/argocd-server 8080:443Reach the UI and API. Keep it running in its own terminal
argocd login localhost:8080 --insecure --username admin --password ...Log the CLI in
argocd cluster listClusters Argo CD can deploy to (in-cluster)
argocd proj get defaultThe default project: any repo, any destination

The values file

global:
  domain: argocd.localtest.me
configs:
  cm:
    timeout.reconciliation: 60s   # check Git every minute
dex:
  enabled: false                  # no SSO on a laptop

Test yourself

Answer in your head first, then open the answer.

Which component clones Git and renders Helm charts?

argocd-repo-server. The application controller compares and syncs; the server serves the UI and API.

Why install chart 10.10.5 instead of the latest?

A repeatable, reviewable install: the version is in Git, and upgrades become deliberate changes.

Where does the admin password come from?

The install generates it into the argocd-initial-admin-secret Secret.

helm upgrade fails with a conflict on .data.timeout.reconciliation. What happened?

Someone changed that field by hand. Server-side apply refuses to overwrite another field manager silently.

Where module 01 leaves you: A running cluster with Argo CD installed from your values file, a platform repo with that file committed, and you logged in as admin in both the UI and the CLI. No applications yet.

02 Your first Application, from Git

This module only · Full module

The idea

An Application is a pointer: a source in Git, a destination in a cluster, and a project that allows it. Argo CD reports two independent things about it: sync (does the cluster equal Git?) and health (does it work?).

Remember

  • Application: a Kubernetes object in the argocd namespace with source (repoURL, targetRevision, path), destination (server, namespace), project and an optional syncPolicy.
  • Sync status: Synced, OutOfSync, or Unknown when Argo CD cannot even render or read the source.
  • Health status: Healthy, Progressing, Degraded, Missing, Suspended. Independent of sync: Synced + Degraded means Git faithfully declares something broken.
  • Refresh vs sync: refresh re-reads Git and compares (automatic every few minutes; --refresh only skips the wait). Sync applies the difference. Without a sync policy, syncing is manual.
  • Reading it: OutOfSync + Missing = in Git, not deployed yet. Synced + Progressing = pods starting. Unknown + ComparisonError = read the conditions.
  • History: one row per sync (a deploy), not per commit. REVISION shows the branch or tag and the exact commit.

Pitfalls

  • Pushing an Application file to Git does nothing until something applies it: kubectl apply here, the root app from module 04.
  • A typo in path gives Unknown and a ComparisonError. The running workload is untouched: Argo CD changes nothing it cannot render.
  • Synced does not mean working. Always read health as well.

Commands

kubectl apply -n argocd -f apps/checkout-dev.yamlCreate or update the Application object. Pushing the file is not enough yet
argocd app get APPSync, health, conditions and every resource
argocd app get APP --refreshLook at Git now instead of waiting
argocd app diff APPWhat a sync would change
argocd app sync APPApply the difference
argocd app wait APP --health --timeout 180Wait until it is healthy
argocd app history APPOne row per deploy, with the commit

An Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-dev
  namespace: argocd           # Applications live here
spec:
  project: default            # the fence
  source:                     # where in Git
    repoURL: https://github.com/YOUR_GITHUB_USER/checkout-config.git
    targetRevision: main
    path: manifests
  destination:                # where in the cluster
    server: https://kubernetes.default.svc
    namespace: checkout-dev
  syncPolicy:
    syncOptions: [CreateNamespace=true]

Test yourself

Answer in your head first, then open the answer.

An Application is Synced and Degraded. What does that mean?

The cluster matches Git, but the workload is not working.

Without a sync policy, what happens when you push a change?

Argo CD detects it and shows OutOfSync, then waits for someone to sync.

Where does an Application object live?

In the argocd namespace, on the cluster where Argo CD runs. Its destination namespace holds what it deploys.

What does --refresh do, and what does it not do?

It makes Argo CD read Git right now. It deploys nothing.

Where module 02 leaves you: The checkout-config repo holds the manifests, the Application checkout-dev exists in Argo CD, and the demo app (podinfo) runs in the checkout-dev namespace with 2 replicas, deployed from Git.

03 Reconciliation: auto-sync, prune and self-heal

This module only · Full module

The idea

Drift is the cluster no longer matching Git. Automated sync applies new commits, prune deletes what Git removed, and self-heal undoes changes made by hand. With all three on, Git is the only way to change what runs.

Remember

  • Automated sync (syncPolicy.automated): new commits are applied without waiting for a person.
  • Prune (prune: true): objects deleted from Git are deleted from the cluster. Off by default even with auto-sync, so a bad commit cannot silently delete half an app.
  • Self-heal (selfHeal: true): live changes made outside Git are put back within seconds. Without it, auto-sync only reacts to new commits.
  • The loop: Git is re-read every timeout.reconciliation (60 s here, plus jitter). Live changes are noticed sooner, through watches. A Git webhook makes detection instant.
  • Sync options worth knowing: Prune=false on one resource (volumes, namespaces), PruneLast, ServerSideApply, ApplyOutOfSyncOnly, RespectIgnoreDifferences, and Replace (rare; causes downtime).
  • Every sync is recorded: what triggered it (automated or a person), the revision, the result per resource.

Pitfalls

  • "Argo CD keeps reverting my fix" is not a bug. With self-heal on, the only fix that stays is a commit; take away write access and manual fixes become impossible too.
  • Auto-sync without prune leaves deleted objects running, shown as OutOfSync. Pruning is a separate decision on purpose.
  • Data you must not lose to a deleted line needs argocd.argoproj.io/sync-options: Prune=false on the resource.

Commands

argocd app get APP | grep 'Sync Policy'Expect Automated (Prune)
kubectl -n NS get deployment NAME -wWatch self-heal undo a hand change
git revert --no-edit HEAD && git pushUndo the last change the GitOps way
argocd app get APP --show-operationThe last sync: trigger, revision, result per resource
argocd app sync APP --prune --dry-runPreview a sync, deletions included

The sync policy

syncPolicy:
  automated:
    prune: true       # delete what was deleted from Git
    selfHeal: true    # undo changes made outside Git
  syncOptions:
    - CreateNamespace=true

Test yourself

Answer in your head first, then open the answer.

Auto-sync is on, prune is off. You delete a manifest from Git. What happens?

The object stays, and the app shows OutOfSync.

What does self-heal react to?

Live changes made outside Git. Auto-sync alone only reacts to new commits.

Why set Prune=false on a single resource?

So a deleted line in Git can never delete it, for example a volume claim that holds data.

How do you make a 2 a.m. hotfix stick?

Commit it to the environment repo. Undo it the same way, with a revert, afterwards.

Where module 03 leaves you: checkout-dev syncs by itself: new commits are applied, objects deleted from Git are deleted from the cluster, and hand changes are undone within seconds. Git is now the only way to change what runs.

04 App of apps: everything declarative, Argo CD included

This module only · Full module

The idea

One root Application, applied by hand once, turns every file in apps/ into an Application, including one for Argo CD itself. After that, everything, upgrading Argo CD included, is a reviewed file in Git.

Remember

  • App of apps: root's source is the apps/ folder. Add a file, get an Application; delete it, the Application goes.
  • The one manual step, ever: kubectl apply -f bootstrap/root.yaml. Root lives in bootstrap/, outside apps/, so a bad commit cannot delete it.
  • Argo CD manages itself: the argocd Application points at the same chart version and values you installed with, and adopts what Helm created. Never helm upgrade that release again; helm list still shows the old record, harmlessly.
  • Guardrails first: a validate CI check plus branch protection (require a pull request and the check) make pull requests the only way into main.
  • The resources finalizer resources-finalizer.argocd.argoproj.io: deleting the Application deletes what it deployed. Without it the workload keeps running, unmanaged.
  • Adopt, then upgrade: keep the chart at the installed version when Argo CD takes over. Upgrading is a separate pull request.

Pitfalls

  • Deleting a file in apps/ is an uninstall. Without the finalizer, the workload keeps running with nothing managing it: no self-heal, and the next real change never arrives.
  • Running helm upgrade on the original release after Argo CD adopted it: two owners for one install.
  • Requiring approvals on a repo you work on alone blocks you, because you cannot approve your own pull request. Require the check instead.

Commands

kubectl apply -n argocd -f bootstrap/root.yamlThe last kubectl apply for Applications
argocd app listRoot and its children
argocd app resources rootEverything root owns
scripts/validate.shRun the CI check locally before you push
gh pr create --fillPropose the change
gh pr checks --watchWait for validate
gh pr merge --squash --delete-branchMerge it

Test yourself

Answer in your head first, then open the answer.

How do you add a new Application after this module?

A pull request that adds its manifest to apps/.

Why does root.yaml live in bootstrap/ and not in apps/?

If root managed itself, a bad commit to its own file could delete it and everything below it.

How do you upgrade Argo CD from now on?

A pull request that changes targetRevision in apps/argocd.yaml.

What does the resources finalizer change about deleting an Application?

With it, deleting the Application also deletes its workload. Without it, the workload stays, unmanaged.

Where module 04 leaves you: One root Application owns argocd and checkout-dev. Adding or removing a file in apps/ adds or removes an Application. main accepts changes only through a pull request that passes validate. kubectl apply for Applications is over.

05 Helm and Kustomize in practice

This module only · Full module

The idea

Argo CD renders Helm and Kustomize itself and applies plain manifests, so there is no Helm release. Helm packages apps other people install; Kustomize holds the small differences between your own environments. Render locally before you push.

Remember

  • How Argo CD picks the tool: only YAML files = a plain directory; a kustomization.yaml = Kustomize; a Chart.yaml = a Helm chart in Git; chart: instead of path: = a chart from a chart repository.
  • Helm in Argo CD: the repo-server runs helm template and applies the output. helm list shows nothing, and Helm hooks become Argo CD hooks. Values stay in Git (helm.valuesObject or value files).
  • Kustomize: the base is ordinary YAML every environment shares; an overlay points at the base and lists only what differs (images, replicas, patches). The base is never edited.
  • Helm or Kustomize? Helm for reusable packages (templates plus values.yaml: vendor add-ons, a chart your team publishes). Kustomize for your own environments (no templates; the base is valid YAML). Many teams use both.
  • Parameter overrides (argocd app set -p, the UI's Parameters tab) change the Application, not Git. A debugging tool only: root self-heals them away.

Pitfalls

  • An images entry whose name does not match the base (stefanprodan/podinfo instead of ghcr.io/stefanprodan/podinfo) silently changes nothing. Argo CD is Synced and Healthy on the old version. Render and grep before you push.
  • Deploying with argocd app set: the change is not in Git and gets undone.
  • Expecting helm history or helm rollback for charts Argo CD deployed. History and rollback live in Argo CD and Git.

Commands

kubectl kustomize OVERLAY_DIRRender an overlay exactly as Argo CD will
helm template NAME CHART_DIR -f values.yamlRender a chart locally
argocd app manifests APPWhat Argo CD rendered from Git
helm list -AShows nothing for charts Argo CD deployed
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/Traefik with no routes answers 404: alive

A Kustomize overlay

# overlays/prod/kustomization.yaml
resources:
  - ../../base
replicas:
  - name: hello
    count: 3
images:
  - name: nginx        # must match the image name in the base exactly
    newTag: "1.28"

Test yourself

Answer in your head first, then open the answer.

You deploy a chart with Argo CD. What does helm list show?

Nothing. Argo CD renders with helm template and applies the output itself.

Where should the differences between dev and prod live?

In overlays (or value files) in the environment repo, changed by pull request.

Why can't an override set with argocd app set survive here?

Root self-heals the Application back to its manifest in Git.

A promotion merged, Argo CD is Synced, prod still runs the old tag. What do you check first?

Render the overlay: the images name probably does not match the base.

Where module 05 leaves you: Three things are now deployed from Git: the traefik ingress controller, checkout-dev (version 6.7.1, 1 replica) and checkout-prod (version 6.7.0, 2 replicas). helm list shows nothing for Traefik, because Argo CD rendered it.

06 Expose Argo CD safely, and upgrade it

This module only · Full module

The idea

A team needs one front door (an ingress with TLS), personal accounts instead of a shared admin, and upgrades that are routine pull requests. All three are lines in Argo CD's values file or its Application.

Remember

  • Ingress with TLS: TLS ends at Traefik and Argo CD serves plain HTTP behind it (server.insecure: true). Safe only because nothing else can reach the server Service.
  • The CLI through an ingress speaks gRPC-web: argocd login HOST --grpc-web.
  • Accounts in Git, passwords not: accounts.alice: login and g, alice, role:admin in policy.csv are config. The password is set once with argocd account update-password and stored in argocd-secret.
  • policy.default: "": logged in but not listed means you see nothing.
  • Switch off admin (admin.enabled: false) once someone else can administer, and delete argocd-initial-admin-secret.
  • An upgrade is one line: read the release notes, render old and new and diff them, change targetRevision, merge, verify. Patch: routine. Minor: read the notes. Major: test first. Downgrades across minor versions are not supported.

Pitfalls

  • Removing the only admin from policy.csv locks everyone out of the UI. The application controller needs no login, so reverting the commit still fixes it. Keep a break-glass account and a CI check that one admin remains.
  • connection refused on localhost:8080 after the switch: the CLI is still logged in to the old port-forward. Log in again through the ingress.
  • --insecure is only for the self-signed laptop certificate. Production uses real certificates (cert-manager) and SSO.

Commands

argocd login HOST --grpc-web --insecure --username alice --password ...Log in through the ingress
argocd account update-password --account NAME --current-password ... --new-password ...Set a local account's password
argocd account get-user-infoWho am I logged in as?
diff <(helm template ... --version OLD) <(helm template ... --version NEW)See what an upgrade changes before you make it
argocd version --shortClient and server versions after the upgrade

The values that matter

configs:
  params:
    server.insecure: true      # TLS ends at the ingress
  cm:
    admin.enabled: false
    accounts.alice: login
  rbac:
    policy.default: ""
    policy.csv: |
      g, alice, role:admin
server:
  ingress:
    enabled: true
    ingressClassName: traefik

Test yourself

Answer in your head first, then open the answer.

Why is server.insecure: true acceptable here?

TLS terminates at the ingress, and only the ingress can reach the Argo CD server.

Where does alice's password live?

In the argocd-secret Secret, set with the CLI. Git only declares that the account exists.

What is an Argo CD upgrade, technically?

A pull request changing targetRevision of the argocd Application.

You locked yourself out with an RBAC commit. How do you get back?

Revert the commit. The controller applies Git without needing a login.

Where module 06 leaves you: Argo CD answers at https://argocd.localtest.me:8443 with TLS (no more port-forward), alice is the administrator, admin is disabled, and Argo CD runs the newer chart version, upgraded by a pull request.

07 Three repositories and multiple sources

This module only · Full module

The idea

Code, environment settings and platform config change at different speeds with different risks, so they live in three repos. One Application reads the chart from one repo and the values from another with sources and a ref. A tag is a release; a change to the environment repo is a deploy.

Remember

  • Application repo (checkout-app): code, CI and the chart. A release is a Git tag. Merging here deploys nothing.
  • Environment repo (checkout-config): values per environment. Changing a tag or a value here is a deploy.
  • Platform repo (argocd-platform): Argo CD, Applications, projects. Few changes, widest blast radius.
  • Multiple sources: sources: is a list. A source with ref: config only contributes files; another source reads one as $config/helm/prod.yaml.
  • One revision per source: argocd app history shows only the first source. argocd app get APP --show-operation lists every source's commit under Sync Revision:, in source order.
  • Immutable fields: a Deployment's spec.selector cannot change. Switching from Kustomize to the chart needed a delete and recreate; in prod, plan renames (new name, then delete the old).
  • Not a grouping tool: one Application per thing, grouped by an app of apps. Bundling unrelated things into sources loses separate sync, health and history.

Pitfalls

  • Moving a release tag deploys to every environment that pins it, with no change to the environment repo. Protect release tags on GitHub (tag rulesets).
  • Reading history and concluding nothing changed: it only shows the first source.
  • field is immutable on spec.selector comes from Kubernetes, not Argo CD.

Commands

git tag v1.0.0 && git push origin v1.0.0Release the chart
helm lint chartCheck the chart
helm template checkout CHART_DIR -f helm/prod.yamlRender with one environment's values
argocd app get APP --show-operation | grep 'Sync Revision'The deployed commit of every source
argocd app get APP --hard-refreshRe-read Git and drop cached manifests (a tag moved)

Chart from one repo, values from another

sources:
  - repoURL: https://github.com/YOUR_GITHUB_USER/checkout-app.git           # source 1: the chart
    path: chart
    targetRevision: v1.0.0      # a release
    helm:
      valueFiles:
        - $config/helm/prod.yaml  # read from the ref source
  - repoURL: https://github.com/YOUR_GITHUB_USER/checkout-config.git         # source 2: the values
    targetRevision: main
    ref: config                 # the nickname behind $config

Test yourself

Answer in your head first, then open the answer.

Which change is a deploy to prod?

A change to helm/prod.yaml in checkout-config.

What does ref: config do?

Names a source so other sources can read files from it as $config/.... It deploys nothing itself.

One Application for checkout, catalog and Traefik: good idea?

No. Unrelated things lose separate sync, health and history. One Application each, grouped by an app of apps.

Why does history keep saying v1.0.0 after a values change?

It shows only the first source, the chart, which did not change.

Where module 07 leaves you: Three repos exist: argocd-platform (Applications), checkout-app (the chart, tagged v1.0.0) and checkout-config (values per environment). checkout-dev and checkout-prod deploy from chart plus values, and prod now runs 6.7.1.

08 Private repositories and secrets

This module only · Full module

The idea

Credentials are the one thing that must not sit in Git. Argo CD reads a private repo with a read-only deploy key stored as a labelled Secret. Values live in a secret store; Git holds only ExternalSecrets, references that say which key to fetch from which store, and External Secrets builds the Secrets in the cluster.

Remember

  • Repository credential: a Secret in argocd labelled argocd.argoproj.io/secret-type: repository, with type, url and a key or token. A repo-creds Secret does the same for every repo under a URL prefix.
  • Deploy key: an SSH key that can read one repository and belongs to no person. The narrowest credential GitHub offers, and no outage when someone leaves.
  • ExternalSecret: names a store (secretStoreRef), the key in it (remoteRef) and the Secret to build (target). target.template adds labels, such as the repository label Argo CD looks for. Safe to commit: it holds no value.
  • ClusterSecretStore: how External Secrets reaches a store and logs in. conditions lists the only namespaces that may read from it. One store per team.
  • On kind: each store is a namespace (store-platform, store-checkout) you fill with kubectl create secret. In production it is a vault reached with workload identity; the ExternalSecrets do not change.
  • The URL must match: the Application's repoURL must be the SSH URL the credential was made for.
  • The options: External Secrets with a store (this course), SOPS (you have a cloud KMS), encrypt-for-one-cluster controllers (avoid: a rebuild means re-encrypting everything). Avoid the Vault plugin: secrets end up in the repo-server cache.

Pitfalls

  • Pointing an ExternalSecret at another team's store: denied by spec.condition, Degraded, and no Secret. Use your own store.
  • Forgetting the value: an ExternalSecret whose key is not in the store stays SecretSyncedError. Git says what; someone still has to put the value in the store.
  • A changed value reaches the Secret at the next refresh, but a pod reads environment variables only at start. Restart the Deployment after a rotation.
  • A repo made private, or a deleted deploy key, shows up as Unknown plus an authentication error at the next fetch, not at once. --hard-refresh shows it now.

Commands

kubectl -n store-platform create secret generic NAME --from-file=sshPrivateKey=KEYPut a value in a store, never in Git
kubectl get clustersecretstoreIs each store Valid and READY True?
kubectl -n NS get externalsecret,secret NAMEThe ExternalSecret from Git (SecretSynced) and the Secret built from it
kubectl -n NS describe externalsecret NAMEWhy a Secret was not built: Conditions and Events
kubectl -n NS annotate externalsecret NAME force-sync=$(date +%s) --overwriteFetch again now, instead of waiting for refreshInterval
ssh-keygen -q -t ed25519 -N '' -f KEYA key pair for Argo CD (no passphrase: it runs unattended)
gh repo deploy-key add KEY.pub -R OWNER/REPO --title argocdRegister the public half, read-only
argocd repo listIs each repository connection Successful?

Test yourself

Answer in your head first, then open the answer.

What makes a Secret a repository credential for Argo CD?

The label argocd.argoproj.io/secret-type: repository, plus url and credentials. Here the ExternalSecret's template sets the label.

Why a deploy key instead of a personal access token?

It reads one repository only and belongs to no person.

Someone reads your ExternalSecret on GitHub. What can they do with it?

Nothing useful. It names a store and a key; only identities you granted can read the store.

What keeps team-checkout from reading the platform's deploy keys?

The conditions on the platform ClusterSecretStore: only argocd may use it.

Where module 08 leaves you: checkout-config is private and Argo CD reads it over SSH with a deploy key from the platform store. Each environment has a checkout-db Secret built by an ExternalSecret, and the checkout pod gets DB_PASSWORD. Git holds only references.

09 AppProjects and RBAC: team boundaries

This module only · Full module

The idea

Two fences per team: an AppProject decides what Argo CD may deploy for them (from which repos, into which namespaces, which kinds), and Argo CD RBAC decides what a person may do in Argo CD. Namespaces belong to the platform.

Remember

  • AppProject: sourceRepos, destinations (cluster plus namespace), clusterResourceWhitelist (cluster-scoped kinds allowed) and namespaceResourceBlacklist (namespaced kinds denied). Anything outside fails with not permitted in project.
  • Argo CD RBAC: policy.csv lines p, role, resource, action, project/app, allow and g, account-or-group, role. Failure: permission denied.
  • Kubernetes RBAC is the third fence, for kubectl. Failure: forbidden.
  • Platform-owned namespaces: teams deploy into namespaces they cannot create, label or delete, so CreateNamespace goes away for team apps.
  • The default project allows everything. In production restrict it (sourceRepos: []) and map SSO groups instead of local accounts.
  • Also in projects: project roles issue tokens for automation in one project; sync windows allow or deny syncs on a schedule.

Pitfalls

  • Changing bootstrap/root.yaml and expecting it to apply: root does not manage itself. Apply it by hand.
  • A team adding a ClusterRole or a NetworkPolicy in its own repo: the sync fails as not permitted and nothing is applied. Access requests go to the platform repo, under review; grant the narrowest thing (view in their namespaces).
  • "Just admin in dev": the habits a team forms in dev come with it to prod. Give dev a looser project if needed, but keep the fence.

Commands

argocd proj listEvery AppProject
argocd app list -o wideApplications with their project
argocd account can-i sync applications 'PROJECT/*'Would Argo CD RBAC allow this?
kubectl auth can-i list pods -n NS --as USER --as-group GROUPTest Kubernetes RBAC as someone else
kubectl apply -n argocd -f bootstrap/root.yamlRoot's own file changed: apply it by hand

A team role in policy.csv

p, role:checkout-team, applications, get, checkout/*, allow
p, role:checkout-team, applications, sync, checkout/*, allow
p, role:checkout-team, applications, action/*, checkout/*, allow
p, role:checkout-team, logs, get, checkout/*, allow
p, role:checkout-team, projects, get, checkout, allow
g, checkout-dev, role:checkout-team

Test yourself

Answer in your head first, then open the answer.

checkout-dev runs argocd app delete checkout-prod. Which fence stops it?

Argo CD RBAC: its role has get, sync, actions and logs only.

team-checkout adds a NetworkPolicy to its namespace through Git. What happens?

Argo CD refuses to sync it: NetworkPolicy is on the project's namespace blacklist.

Why do namespaces belong to the platform team?

Labels, quotas and policies on the namespace are the platform's guarantees. A team must not be able to remove them.

not permitted in project versus permission denied: which fence said each?

The AppProject (what may be deployed) versus Argo CD RBAC (what a person may do).

Where module 09 leaves you: Platform apps sit in the platform project, checkout apps in the checkout project (its two repos and two namespaces only). The platform team creates the namespaces. The account checkout-dev can see, sync and read logs for checkout, and nothing else.

10 Deployment ordering: hooks, phases and waves

This module only · Full module

The idea

Argo CD orders a sync by phase (PreSync, Sync, PostSync), then by wave (lowest first, each healthy before the next), then by kind and name. A PreSync migration that fails stops the release before anything changes.

Remember

  • Phases: PreSync, Sync, PostSync, and SyncFail when something fails. (PreDelete and PostDelete run around deleting an app.)
  • Hooks: resources annotated argocd.argoproj.io/hook: <phase>, usually Jobs. They run when a sync runs and are not part of the compared state, so adding one leaves the app Synced. A delete policy says when to clean them up.
  • Waves: argocd.argoproj.io/sync-wave: "-1". Lower waves go first, and Argo CD waits for each wave to be healthy before the next.
  • Within a wave: by kind (namespaces, CRDs, config, workloads, ...), then alphabetically by name.
  • Waves between Applications (app of apps) only wait if Applications report health: add the Lua health check resource.customizations.health.argoproj.io_Application to argocd-cm.
  • Migrations must be safe to run twice and backwards compatible: the old pods keep serving while they run.

Pitfalls

  • A failed PostSync smoke test marks the sync Failed while the app stays Healthy: the deploy happened, the check after it failed. Read the hook Job's logs.
  • Using hooks so an app can wait for its database. Make the app retry; keep hooks for things that run exactly once per release.
  • Expecting waves between child Applications to wait without the Application health check.

Commands

argocd app sync APPRun a sync, and with it the hooks
argocd app get APP --show-operation | sed -n '/^GROUP/,$p'The order the last sync applied things in
kubectl -n NS get jobsHook Jobs left behind
kubectl -n NS logs job/NAMEWhat a hook did
grep -H 'sync-wave' apps/*.yamlWhich Applications carry a wave

The two annotations

# A migration Job: runs before the deploy
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
---
# A ConfigMap: applied before the Deployment that reads it
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Test yourself

Answer in your head first, then open the answer.

The PreSync migration fails. What happens to the running Deployment?

Nothing. The sync stops before the Sync phase, and the SyncFail hook runs.

Two resources have sync-wave -1 and 0. When does the second start?

Once wave -1 is healthy.

Why does the app of apps need a health check for Applications?

Without it a child Application counts as healthy at once, so waves between Applications do not wait.

You add a hook to the chart. Why does the app stay Synced?

Hooks are not part of the state Argo CD compares. They run on the next sync.

Where module 10 leaves you: Every sync of checkout-dev runs a migration before the deploy and a smoke test after it. You have seen a broken migration leave the old version running, and checkout-dev is back to healthy on 6.7.1.

11 Debugging labs: seven ways Argo CD goes wrong

This module only · Full module

The idea

Start at the Application and go one layer down at a time: status and conditions, the resource tree, the object's events, pod logs, then Argo CD's own logs. Render before you guess, and fix through Git.

Remember

  • Read the whole argocd app get before typing anything else. Sync, health, conditions and the last operation answer most questions.
  • Render or apply? Broken YAML fails at rendering (Unknown, ComparisonError, nothing applied). A wrong value such as memory: lots fails at applying (the sync fails).
  • The seven failures: a deny sync window holds a merge back; a bad image tag goes Synced, then Degraded; invalid manifests of both kinds; AppProject and RBAC refusals; a deleted deploy key; an Application stuck deleting; an autoscaler and Argo CD fighting over replicas.
  • Two owners, one field: pick one owner and make it permanent in Git. Here: declare the autoscaler in the values instead of creating it with kubectl.
  • Stuck deletion: find what the deletion waits for (another resource's finalizer) and fix that. Never strip Argo CD's own finalizer first.
  • Revert is the default fix. kubectl only restores service while the real fix goes through Git. Write one runbook entry per incident: symptom, where to look, cause, fix, prevention, owner.

Pitfalls

  • Removing resources-finalizer.argocd.argoproj.io to unstick a deletion leaves everything the app deployed behind, unmanaged.
  • Switching off self-heal to stop a fight over a field: the flapping stops, but Git no longer describes what runs.
  • Guessing before rendering. Nine times out of ten argocd app manifests or a local render shows the problem.

Commands

argocd app get APP --refreshStatus, conditions and the last operation
argocd app manifests APPExactly what Argo CD will apply
argocd app diff APPWhat differs, field by field
argocd proj windows list PROJECTSync windows that can block syncs
argocd repo listRepository connections and their status
argocd app terminate-op APPCancel a stuck sync

Test yourself

Answer in your head first, then open the answer.

An Application is Unknown with a ComparisonError. What is true?

Argo CD could not render or read the source. Nothing was applied and the running workload is untouched.

An app goes OutOfSync again right after every sync, always on the same field. Likely cause?

Another controller changes that field. Two owners for one field: pick one.

A change merged, but prod stays OutOfSync and never syncs. Where do you look?

Sync windows on the project: argocd proj windows list, or the Window line in argocd app get.

An Application is stuck deleting. What should you not do first?

Remove Argo CD's resources finalizer from it. Fix the stuck resource instead.

Where module 11 leaves you: All seven labs are cleaned up: the platform is healthy again and every Application is Synced and Healthy. Your platform repo has docs/runbook.md with one real entry.

12 How product teams work with Argo CD

This module only · Full module

The idea

Tools do not make an operating model; agreements do. Teams own their chart, values, secrets, releases and rollbacks inside their project; the platform owns Argo CD, projects, namespaces, RBAC and credentials. Write it down, enforce it with CODEOWNERS and CI, and roll back with a revert.

Remember

  • Teams own: their chart (checkout-app), values and secrets (checkout-config), releases (tags), and syncs and rollbacks in their project. Inside it, they need nobody.
  • The platform owns: the Argo CD install, upgrades, exposure, SSO and RBAC; AppProjects, namespaces and quotas; repository credentials; add-ons. Teams propose apps/checkout-*.yaml; the platform reviews.
  • How teams deploy: a pull request to checkout-config (versions, values) or to apps/checkout-*.yaml (a chart release), reviewed through CODEOWNERS.
  • How teams inspect: read-only in Argo CD for their apps: status, diff, history, resources, logs. Write access is Git.
  • When to escalate: the fix is outside the team's repos (permissions, credentials, add-ons, Argo CD itself), or every app is Unknown and the UI is down.
  • Contracts that matter are CI checks, not trust: for example, chart sources must be release tags like v1.2.3.

Pitfalls

  • Using the rollback button: it refuses while auto-sync is on, and turning auto-sync off leaves Git saying something other than the cluster. Revert in Git.
  • Pointing prod at a branch "for an hour": from then on every push to that branch deploys to prod, unreviewed. A CI check that chart sources are release tags stops it.
  • Escalating what the team owns (its own failed syncs, Degraded pods), or fixing what the platform owns.

Commands

argocd app get APPIs it OK?
argocd app history APPWhat was deployed, and when?
argocd app diff APP; echo $?Anything waiting to be applied? Exit code 0 means nothing
argocd app resources APPWhat objects does it own?
argocd app logs APP --tail 5What is it saying? No kubectl needed
git revert --no-edit HEAD && git pushRoll back

Test yourself

Answer in your head first, then open the answer.

team-checkout wants a new chart release in dev. Which change?

A pull request changing targetRevision in apps/checkout-dev.yaml, which they can approve through CODEOWNERS.

Why is the Argo CD rollback button the wrong rollback here?

It refuses while auto-sync is on, and turning auto-sync off leaves Git disagreeing with the cluster.

team-checkout sees authentication required on its apps. Who acts first?

team-checkout checks its deploy key is still on the repo, and escalates if the credential in Argo CD must change.

What is worth measuring about this agreement?

How often a product team has to ask the platform team for something. Each escalation is a candidate for self-service.

Where module 12 leaves you: The platform repo documents the rules and enforces who reviews what. You have run a full release and rollback as team-checkout, and dev is back on the release it started with.

13 Capstone: onboard and operate catalog

This module only · Full module

The idea

Onboarding a team is a set of reviewed files: an AppProject, platform namespaces, an account and role, a secret store, a repo credential, ExternalSecrets and two Applications. Handling an incident is: health first, then the pods, then a revert, then the five answers.

Remember

  • Platform pieces: projects/catalog.yaml (only its repos, only its namespaces, no cluster-scoped kinds, quotas, limits, network policies and RBAC denied), namespaces/catalog.yaml, secret-store/catalog.yaml (store-catalog, readable only from catalog-dev and catalog-prod), and accounts.catalog-dev with role:catalog-team in policy.csv.
  • Team pieces: a private catalog-config with helm/dev.yaml and helm/prod.yaml (nameOverride: catalog) and an ExternalSecret catalog-db per environment, reading from the catalog store. Its deploy key sits in store-platform, with an ExternalSecret in the platform repo.
  • The Applications: three sources each (the shared chart at v1.0.0, the values repo as ref: config, and secrets/<env>), project catalog, the resources finalizer, and automated sync with prune and self-heal.
  • Done means: both apps Synced and Healthy, the team account limited to its project, a secret from its own store in use, a PreSync migration in dev, and an incident written up.
  • The five answers: what failed, how it was detected, who responded, how it was corrected, how it is prevented.
  • Rebuild: your repos recreate everything on a fresh cluster. On kind the store values live in the cluster, so put them back; with a real vault there is nothing to redo.

Pitfalls

  • Leaving the shared chart repo out of the project's sourceRepos: the Applications are not permitted.
  • Copying checkout's ExternalSecrets for catalog without changing the store: the checkout store does not serve catalog's namespaces. Point them at catalog.
  • Pushing straight to main, as the course did from module 07 to stay on Argo CD. A team protects main in every repo and works through pull requests with the validate check.

Commands

argocd app get APP --refresh | grep -E 'Health|Deployment'Health first
kubectl -n NS get podsThe reason: OOMKilled, CrashLoopBackOff
git revert --no-edit HEAD && git pushRecover
gh issue create -R OWNER/REPO --title "Incident: ..." --body "What failed: ..."Write the five answers
kind delete cluster --name argocdTear down. Git keeps the setup

Test yourself

Answer in your head first, then open the answer.

What did onboarding team-catalog require from the platform team?

A project, namespaces, a secret store, an account and role, a repo credential and two Application files, all as reviewed pull requests.

Where do you look first when an app is Degraded?

The resource tree, then the failing object's events and logs. Degraded is about the workload, not Argo CD.

What would you automate when a third team arrives?

Generating the project, namespaces and Applications from one file per team with ApplicationSets: the Fleet GitOps course.

Where module 13 leaves you: team-catalog runs catalog in dev and prod, with the same boundaries as team-checkout, entirely from Git. You handled an incident and wrote the five answers in an issue. You can run Argo CD for a team.