Module 12 · Operate with teams · 35 min · You are: Platform and product

How product teams work with Argo CD

Tools do not make an operating model; agreements do. Everything is in place: projects, accounts, three repos. This module makes the rules explicit, adds review rules that enforce them, and runs a release and a rollback as team-checkout from start to end.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

What teams own

Their chart, their values, their secrets, their releases and rollbacks. Inside their project, they need nobody.

What the platform owns

Argo CD, projects, namespaces, RBAC, repo credentials, add-ons. Teams propose; the platform reviews.

How teams deploy

A pull request to checkout-config (versions, values) or to apps/checkout-*.yaml (chart release), reviewed through CODEOWNERS.

How teams inspect

Read-only in Argo CD for their apps: status, diff, history, logs, events. Write access is Git.

When to escalate

When the fix is outside their repos: permissions, credentials, add-ons, Argo CD itself.

Ownership

When team-checkout escalates

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• docs/operating-model.md (reference): The contract, written down. Copy it into your platform repo.
  1. Release, run in dev, promote, roll back L15–20
  2. One command per question L22–30
• platform/.github/CODEOWNERS (reference): The review rules that enforce it.
  1. The team approves its own chart bumps; the platform owns the rest L4

Steps

1. Write the contract into the platform repo

Copy the operating model and the review rules. In an organisation, replace @YOUR_GITHUB_USER with team handles (@acme/platform, @acme/team-checkout) and require code-owner review in branch protection.

Copy the two documents: who owns what, and how to debug:

cd ~/work/argocd-platform && git switch -q main && git pull -q
mkdir -p docs && cp ~/work/argocd-gitops/docs/operating-model.md ~/work/argocd-gitops/docs/debugging.md docs/

Copy the review rules. CODEOWNERS tells GitHub who must approve a change to which file. Your GitHub user replaces the placeholder.

cp ~/work/argocd-gitops/platform/.github/CODEOWNERS .github/ && perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' .github/CODEOWNERS

git add -A && git commit -qm "Operating model and review rules" && git push -q

2. team-checkout ships a release, end to end

As the team: a new chart release, then a pull request bumping dev to it. Watch it as checkout-dev, not as admin.

cd ~/work/checkout-app && git switch -q main && git pull -q

Change in ~/work/checkout-app/chart/Chart.yaml

- version: 1.0.0
+ version: 1.1.0

Release it. Commit and push the new chart version:

git commit -qam "checkout chart 1.1.0" && git push -q

Then tag that commit as the release v1.1.0 and push the tag:

git tag v1.1.0 && git push -q origin v1.1.0

Now the platform repo: point dev at the new release.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Change in ~/work/argocd-platform/apps/checkout-dev.yaml

-       targetRevision: v1.0.0          # a release of the chart
+       targetRevision: v1.1.0          # a release of the chart

git add -A && git commit -qm "checkout-dev: chart v1.1.0" && git push -q

Watch it as the team, not as admin:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username checkout-dev --password 'another-long-one'

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 240

3. Inspect like a product team

Everything team-checkout needs to answer "what is running and is it OK", without kubectl.

Is it OK? Expect Synced and Healthy, and the chart source at v1.1.0.

argocd app get checkout-dev

What was deployed, and when? The newest row is the release you just shipped.

argocd app history checkout-dev

Is anything waiting to be applied? Expect no diff output, and exit code 0 (the number means: nothing to apply).

argocd app diff checkout-dev; echo "diff exit: $?"

What objects does it own? Expect the Deployment, Service, ConfigMap and the rest, each Synced.

argocd app resources checkout-dev

What is it saying? The last five log lines, without kubectl:

argocd app logs checkout-dev --tail 5

4. Roll back with a revert, not the button

The release is unwanted. Revert the pull request. The UI rollback refuses to run while auto-sync is on, and turning auto-sync off to use it would leave Git saying something else than the cluster.

Try the rollback command first. Expect a refusal that mentions auto-sync:

argocd app rollback checkout-dev 2>&1 | tail -1

Log back in as alice, then undo the release the Git way:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'

cd ~/work/argocd-platform && git switch -q main && git pull -q

git revert --no-edit HEAD && git push -q

Wait until dev is back on the earlier release and healthy:

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health

Break it on purpose: Test in prod from a branch

team-checkout wants to try a chart change in prod "just for an hour". The team pushes a branch and, through a pull request it may approve itself, points checkout-prod at the branch instead of a release.

You practise: spotting an Application that follows a branch, why that turns every push into a deploy, and enforcing the contract with a check instead of trust.

1. Break it

cd ~/work/checkout-app && git switch -q main && git pull -q
git switch -c try-it

Change in ~/work/checkout-app/chart/values.yaml

- progressDeadlineSeconds: 120   # report a failed rollout after 2 minutes
+ progressDeadlineSeconds: 90    # report a failed rollout after 2 minutes

git commit -qam "try a shorter deadline" && git push -qu origin try-it

cd ~/work/argocd-platform && git switch -q main && git pull -q

Change in ~/work/argocd-platform/apps/checkout-prod.yaml

-       targetRevision: v1.0.0          # a release of the chart
+       targetRevision: try-it          # a release of the chart

git add -A && git commit -qm "checkout-prod: try a branch" && git push -q

argocd app get root --refresh >/dev/null; sleep 30

2. What you see

The check passed and the pull request merged. argocd app get checkout-prod shows the chart source at try-it, and prod runs the branch: kubectl -n checkout-prod get deploy checkout -o jsonpath='{.spec.progressDeadlineSeconds}' prints 90. Expected: prod only ever runs a release. From now on, every push to try-it deploys to prod with no review.

3. Fix it yourself

Put prod back on release v1.0.0 and delete the branch. Then change the platform repo's check so that a pull request like this one fails: the chart source of every Application must use a release tag like v1.2.3.

cd ~/work/argocd-platform && git switch -q main && git pull -q

git revert --no-edit HEAD && git push -q

git -C ~/work/checkout-app push -q origin --delete try-it

cd ~/work/argocd-platform && git switch -q main && git pull -q

- import glob, sys, yaml
+ import glob, re, sys, yaml

            for src in spec.get("sources", []):
                rev = str(src.get("targetRevision", ""))
                if src.get("path") == "chart" and not re.fullmatch(r"v\d+\.\d+\.\d+", rev):
                    print(f"FAIL {f}: chart source must be a release tag like v1.2.3, not {rev}"); bad += 1

-       targetRevision: v1.0.0          # a release of the chart
+       targetRevision: try-it          # a release of the chart

scripts/validate.sh

-       targetRevision: try-it          # a release of the chart
+       targetRevision: v1.0.0          # a release of the chart

scripts/validate.sh

git add -A && git commit -qm "CI: chart sources must use release tags" && git push -q

Quiz

1. team-checkout wants to run a new chart release in dev. Which change?
   • A pull request changing targetRevision in apps/checkout-dev.yaml, which they can approve through CODEOWNERS
   • Edit the Application in the UI and click sync, which leaves an entry in the audit log and needs no review from anyone else on the platform
   • Ask the platform team to create a new AppProject for dev, then point the team's Application at it so that the release can be rolled out
2. Why is the Argo CD rollback button the wrong rollback here?
   • It is slower than a revert and takes longer to bring the old version back, since it has to rebuild every object of the application first
   • It refuses while auto-sync is on, and turning auto-sync off leaves Git disagreeing with the cluster
   • It deletes the Application and all of its resources, so everything has to be created again from Git before the old version is back
3. team-checkout sees authentication required on its apps. Who acts first?
   • The platform team acts first, rotating every deploy key and sealed credential on the cluster, because they own all of the repository access
   • Nobody yet: wait until the error clears by itself when Argo CD retries the repository, which it normally does within a few minutes
   • team-checkout checks its deploy key is still on the repo, and escalates if the credential in Argo CD must change

Field note

The best platform teams I have worked with measure one thing here: how often a product team has to ask them for something. Every escalation in the table above is a candidate for self-service later.

Report a problem with this module