Module 08 · Operate · 30 min · You are: Platform and product
Operate releases with teams
In this module
- You learn: Argo CD resource actions on a Rollout, least-privilege RBAC for them, Argo Rollouts notifications, and
progressDeadlineAbort. - You do: Narrow team-checkout's actions to three Rollout actions, use them as the team's account, add a release log and subscribe checkout to it, turn on
progressDeadlineAbortin prod, and copy the runbook. - You end with: The
checkout-devaccount can resume, abort and retry checkout's Rollouts and nothing else. Every completed or aborted release writes a line to the release log. Prod runs 6.13.0.docs/releases.mdis in your platform repo.
A release that needs a person needs the right person: team-checkout, at 3 p.m., without asking the platform team for kubectl. Argo CD already has their accounts and projects, so the release actions go there, scoped to their apps. And a release that ends at night should leave a line somewhere a person will read in the morning.
Open this module in the interactive course to track progress and switch tracks.
Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back
Concepts
- Resource actions
- Argo CD has built-in actions for a Rollout (
resume,abort,retry,restart,promote-fulland more). They run in the UI or withargocd app actions run. - Action RBAC
action/argoproj.io/Rollout/resumeallows one action on one kind.action/*allows every action on every kind.- Notifications
- Argo Rollouts sends events (completed, aborted, paused) to services the platform configures. A Rollout subscribes with an annotation.
- progressDeadlineAbort
- When a release has not progressed for
progressDeadlineSeconds, abort it instead of only reporting it.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
platform/argocd/values.yaml(reference): team-checkout's permissions, narrowed.- Three actions on Rollouts, in project checkout only L46–48
platform/argo-rollouts/values.yaml(reference): Where release events go.checkout-app/chart/templates/rollout.yaml(reference): The subscription, from values.- checkout subscribes to both events L13–14
platform/docs/releases.md(reference): The release runbook.
Steps
1. Narrow team-checkout's actions
Since the Argo CD course the team may run every action on every kind in its project. Keep only what a release needs.
cd ~/work/argocd-platform && git switch -q main && git pull -q
Change in ~/work/argocd-platform/argocd/values.yaml
- p, role:checkout-team, applications, action/*, checkout/*, allow
+ # Progressive delivery, module 08: only the release actions on a Rollout,
+ # instead of every action on every kind.
+ p, role:checkout-team, applications, action/argoproj.io/Rollout/resume, checkout/*, allow
+ p, role:checkout-team, applications, action/argoproj.io/Rollout/abort, checkout/*, allow
+ p, role:checkout-team, applications, action/argoproj.io/Rollout/retry, checkout/*, allow
git add -A && git commit -qm "Argo CD: team-checkout may resume, abort and retry Rollouts only" && git push -q
argocd app get argocd --refresh >/dev/null
argocd app wait argocd --sync --health --timeout 300
2. A release log, and checkout subscribed to it
The platform adds the log and the notification settings; team-checkout subscribes and asks prod to abort stuck releases.
mkdir -p release-log docs
cp ~/work/progressive-delivery/platform/apps/release-log.yaml apps/
cp ~/work/progressive-delivery/platform/release-log/release-log.yaml release-log/
cp ~/work/progressive-delivery/platform/docs/releases.md docs/
perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/release-log.yaml
Add to ~/work/argocd-platform/namespaces/platform-tools.yaml at the end of the file
---
apiVersion: v1
kind: Namespace
metadata:
name: release-log
Add to ~/work/argocd-platform/argo-rollouts/values.yaml at the end of the file
# Module 08: notifications to the platform's release log. Product teams
# subscribe a Rollout with an annotation; the platform owns where it goes.
notifications:
notifiers:
service.webhook.release-log: |
url: http://release-log.release-log.svc.cluster.local
headers:
- name: Content-Type
value: application/json
templates:
template.rollout-completed: |
webhook:
release-log:
method: POST
body: |
{"event": "completed", "rollout": "{{.rollout.metadata.name}}", "namespace": "{{.rollout.metadata.namespace}}", "image": "{{(index .rollout.spec.template.spec.containers 0).image}}"}
template.rollout-aborted: |
webhook:
release-log:
method: POST
body: |
{"event": "aborted", "rollout": "{{.rollout.metadata.name}}", "namespace": "{{.rollout.metadata.namespace}}", "image": "{{(index .rollout.spec.template.spec.containers 0).image}}", "message": "{{.rollout.status.message}}"}
triggers:
trigger.on-rollout-completed: |
- send: [rollout-completed]
trigger.on-rollout-aborted: |
- send: [rollout-aborted]
scripts/validate.sh
git add -A && git commit -qm "Release log and Argo Rollouts notifications" && git push -q
argocd app get root --refresh >/dev/null
argocd app wait release-log --sync --health --timeout 300
argocd app wait argo-rollouts --sync --health --timeout 300
cd ~/work/checkout-config && git switch -q main && git pull -q
Add to ~/work/checkout-config/helm/dev.yaml at the end of the file
notify:
releaseLog: true # module 08
Add to ~/work/checkout-config/helm/prod.yaml at the end of the file
notify:
releaseLog: true # module 08
progressDeadlineAbort: true # module 08
git add -A && git commit -qm "checkout: release log; prod aborts stuck releases" && git push -q
argocd app get checkout-dev --refresh >/dev/null; argocd app get checkout-prod --refresh >/dev/null
3. Release as team-checkout
Log in as the team's account, ship 6.13.0 to prod, and drive it with Argo CD actions.
In a second terminal, follow the release log:
kubectl -n release-log logs deploy/release-log -f
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username checkout-dev --password 'another-long-one'
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- tag: "6.12.0"
+ tag: "6.13.0"
git add -A && git commit -qm "checkout prod: 6.13.0" && git push -q
argocd app get checkout-prod --refresh >/dev/null
At the header-route pause, list the actions on the Rollout. Expect resume, abort and retry among them, resume enabled:
argocd app actions list checkout-prod --kind Rollout
Resume it as the team. In the UI this is the Rollout's three-dot menu, Resume.
argocd app actions run checkout-prod resume --kind Rollout
And a permission the team does not have any more. Expect permission denied:
argocd app actions run checkout-prod restart --kind Rollout
About eight minutes later, the release log prints a line with "event": "completed" and image podinfo:6.13.0. Then log in as alice again:
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'
Simplified for the course
The release log prints to its own pod log. In production, point the webhook at your chat (Argo Rollouts has Slack, Teams and many other services built in) and subscribe the team's channel, not a shared one.
Break it on purpose: Trimmed too far
In review, someone trims the policy to "only resume, that is all a release needs". It merges on a Friday.
You practise: finding which permission is missing from an Argo CD denial, and granting exactly that.
1. Break it
cd ~/work/argocd-platform && git switch -q main && git pull -q
Delete from ~/work/argocd-platform/argocd/values.yaml
- p, role:checkout-team, applications, action/argoproj.io/Rollout/abort, checkout/*, allow
- p, role:checkout-team, applications, action/argoproj.io/Rollout/retry, checkout/*, allow
git add -A && git commit -qm "Argo CD: team-checkout may resume only" && git push -q
argocd app get argocd --refresh >/dev/null
argocd app wait argocd --sync --health --timeout 300
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username checkout-dev --password 'another-long-one'
cd ~/work/checkout-config && git switch -q main && git pull -q
Change in ~/work/checkout-config/helm/prod.yaml
- tag: "6.13.0"
+ tag: "6.13.1"
git add -A && git commit -qm "checkout prod: 6.13.1" && git push -q
argocd app get checkout-prod --refresh >/dev/null
2. What you see
Expected: at the header-route pause the tester finds a problem, and team-checkout aborts the release. You get: argocd app actions run checkout-prod abort --kind Rollout fails with permission denied. The release waits at the pause, but the next promotion would send real users to it.
3. Fix it yourself
Give team-checkout back exactly what it needs to stop its own releases (not action/*), then abort 6.13.1 as the team and revert it. Done when the abort works as checkout-dev, prod runs 6.13.0 Healthy, and the policy lists resume, abort and retry only.
Hint 1
The error names the action and the object it was refused on. Which line in policy.csv would allow it?
Hint 2
Add action/argoproj.io/Rollout/abort and action/argoproj.io/Rollout/retry for role:checkout-team in project checkout/*, as alice.
The fix
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'
cd ~/work/argocd-platform && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get argocd --refresh >/dev/null
argocd app wait argocd --sync --health --timeout 300
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username checkout-dev --password 'another-long-one'
argocd app actions run checkout-prod abort --kind Rollout
cd ~/work/checkout-config && git switch -q main && git pull -q
git revert --no-edit HEAD && git push -q
argocd app get checkout-prod --refresh >/dev/null
argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'
After you fixed it: the five questions
- What failed?
- The team could not stop its own release: the policy allowed resume only.
- How was it detected?
- Argo CD:
permission deniednaming the action. - Who responds?
- The platform team owns RBAC; team-checkout noticed when it mattered.
- How is it corrected?
- The two missing lines back in the policy.
- How is it prevented?
- Write the release runbook first (who may do what) and derive the policy from it, then test each line with
argocd account can-iin CI.
Quiz
- Why not keep
action/*for team-checkout?- It allows every action on every kind in the project, such as restarting or deleting other objects
- It does not work for Rollouts
- Argo CD warns about it on every login
- Where is it decided which chat or webhook a release notification goes to?
- In the Rollout's annotation
- In the platform's Argo Rollouts values
- In the AnalysisTemplate
- What does
progressDeadlineAbort: truechange?- Releases that stop progressing are aborted instead of only reported
- Every release is aborted after two minutes
- Analysis runs abort when they take too long
Answers
- It allows every action on every kind in the project, such as restarting or deleting other objects Least privilege: the release needs three actions on one kind.
- In the platform's Argo Rollouts values The platform owns the services; teams only subscribe to triggers.
- Releases that stop progressing are aborted instead of only reported A stuck release goes back to stable on its own.
Field note
The release log is the cheapest observability you will ever add. A week of it answers "how often do we abort, and who reverts afterwards?" better than any dashboard.