Module 07 · Product self-service · 50 min · You are: Product team

Ship, promote and roll back

A precise approval model: non-prod deploys automatically after merge, prod needs an explicit approval, and Git is the audit trail. You will see all three, and the changelog they leave behind.

Open this module in the interactive course to track progress and switch tracks.

Change journey practised: Requested → Proposed → Validated → Previewed → Approved → Merged → Reconciled → Healthy

Concepts

Promotion is a pull request

dev and prod are folders. Promoting means changing newTag in prod/kustomization.yaml. The diff is the release note.

The production gate

production-gate detects changes that reach prod and waits on the production GitHub environment. Required reviewers approve; non-prod changes skip it.

Deployment records

After merge, deployment-record creates a GitHub deployment per product environment, linked to the pull request and the issue: what, who, when.

Reconciled, not merged

Merged is not running. scripts/trace.sh <commit> shows which Applications have synced a revision that contains the commit.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• product-deployments/checkout/prod/kustomization.yaml (reference, your repo): Prod config. Promotion changes one line.
  1. The version in prod L10
  2. Two replicas in prod L11–13
• .github/workflows/production-gate.yaml (reference, your repo): Approval only when a change reaches prod.
  1. What counts as "reaches prod" L36–41
  2. Only then: wait for the production environment approval L51–55
• .github/workflows/deployment-record.yaml (reference, your repo): The changelog, written after every merge.
  1. Which Applications this commit feeds L32
  2. One GitHub deployment per product environment, with PR and issue L51–54
• scripts/trace.sh (reference, your repo): From commit to cluster.
  1. "Reconciled" means the synced revision contains your commit L36–43

Steps

1. Change dev: no approval

Change the dev banner. The pull request skips production-gate.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c checkout-banner

Change in ~/work/fleet-gitops/product-deployments/checkout/dev/kustomization.yaml

-         value: "checkout (dev)"
+         value: "checkout (dev) - new banner"

Review your change. Expect one changed line: the banner text.

git diff

git add -A && git commit -m "checkout: new banner in dev"
git push -u origin HEAD

Open the pull request, then merge it ↗

2. Follow it to dev

Merged is not the same as running. The trace script asks Argo CD whether it has applied your commit yet.

cd ~/work/fleet-gitops && git switch main && git pull -q

Run this right after merging. Find the checkout-dev row: pending means Argo CD has not applied your commit yet, reconciled means it has. It turns into reconciled within about three minutes (or click Refresh in Argo CD).

scripts/trace.sh --context kind-platform --selector fleet.gitops/product=checkout

3. Promote 6.7.1 to prod: approval required

Same journey, plus the gate: the pull request shows production-gate / approve waiting. Approve it under Review deployments, then merge.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c checkout-promote-6.7.1

Change in ~/work/fleet-gitops/product-deployments/checkout/prod/kustomization.yaml

-     newTag: 6.7.0
+     newTag: 6.7.1

git add -A && git commit -m "checkout: promote 6.7.1 to prod"
git push -u origin HEAD

Open the pull request ↗

4. Read the changelog

Open Deployments: checkout-prod with the commit, pull request and issue. The deployment-record run summary lists the affected Applications and why.

cd ~/work/fleet-gitops && git switch main && git pull -q

Trace again. Expect checkout-prod to be reconciled once the sync is done:

scripts/trace.sh --context kind-platform --selector fleet.gitops/product=checkout

The prod history, in Git. Expect your promotion commit at the top; every prod change is one commit here.

git log --oneline -- product-deployments/checkout/prod

5. Roll back with a revert

A revert is a change like any other: pull request, checks, the same production approval, merge. Then trace it.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c revert-6.7.1

Undo the promotion. This creates a new commit that reverses the last one. It assumes the promotion is the latest commit; if not, use its commit id in place of HEAD.

git revert --no-edit HEAD

git push -u origin HEAD

Open the revert pull request ↗

Simplified for the workshop

Deployment records say "queued" (merged) rather than "success" (running). In production, Argo CD Notifications updates the GitHub deployment status when the sync finishes, and posts to the team's channel.

For larger fleets add progressive delivery on top: ApplicationSet progressive sync (dev clusters, then prod) or Argo Rollouts canaries inside a cluster. Argo Rollouts is already installed as an add-on.

Break it on purpose: Skip the gate

Prod needs a third replica, now. Someone commits on main and pushes straight to GitHub.

You practise: what branch protection does with a direct push, and taking the fast path that still leaves a record.

1. Break it

cd ~/work/fleet-gitops && git switch main && git pull -q

Change in ~/work/fleet-gitops/product-deployments/checkout/prod/kustomization.yaml

-     count: 2
+     count: 3

git commit -am "Hotfix: 3 replicas in prod" && git push origin main

2. What you see

The push is rejected: the branch is protected and changes must go through a pull request. The commit exists only on your laptop, and your local main is now ahead of GitHub. Expected (by someone in a hurry): prod at 3 replicas in a minute.

3. Fix it yourself

It really is urgent. Get prod to 3 replicas the fastest way the rules allow, keeping the commit you already made. Then put prod back to 2 the same way.

git switch -c hotfix-prod-replicas
git branch -f main origin/main

git push -u origin HEAD

kubectl --context kind-prod -n checkout get deployment checkout

git switch main && git pull -q
git switch -c prod-replicas-back && git revert --no-edit HEAD && git push -u origin HEAD

Quiz

1. Does a dev-only change need the production approval?
   • No: the gate job is skipped, which counts as passing
   • Yes, always: every merge to main needs the production approval, whatever it touches
   • Only on Fridays, when the freeze rules apply to every change in the repo
2. How do you roll back a bad prod release?
   • Click the Argo CD rollback button, then leave auto-sync off until things calm down
   • git revert in a pull request, through the same checks and approval
   • Run kubectl rollout undo on the Deployment, which restores the previous ReplicaSet at once
3. What does scripts/trace.sh tell you that the merge does not?
   • Who approved the pull request, and when each reviewer left their comments on it
   • The result of the CI checks, which the merge button already hides from you
   • Whether Argo CD has synced a revision containing the commit on each cluster

Field note

Teams always ask for "approvals in Argo CD". Put them where the change is decided, on the pull request, and keep Argo CD fully automatic. Then the history is in one place.

Report a problem with this module