PlatformEngr® Argo CD course

Module 13 · Operate with teams · 75 min · You are: Platform and product

Capstone: onboard and operate catalog

In this module

  • You learn: Nothing new. You use everything from the course on a task with no step-by-step commands: onboarding a second team and handling an incident.
  • You do: Onboard team-catalog from a brief: its project, namespaces, account, private repo with a sealed deploy key, sealed secrets, and Applications for dev and prod. Then break prod, find it with Argo CD, and fix it through Git.
  • You end with: team-catalog runs catalog in dev and prod, with the same boundaries as team-checkout, entirely from Git. You handled an incident and wrote the five answers in an issue. You can run Argo CD for a team.

Everything so far came with commands. Real onboarding does not. You have the reference repo, your own three repos, the runbook and the operating model. Use them. Each part lists what "done" means; hints and one possible solution are folded away for when you are stuck.

Open this module in the interactive course to track progress and switch tracks.

Concepts

The brief
team-catalog runs catalog in dev and prod. They reuse checkout's chart (nameOverride: catalog) at v1.0.0, keep values and secrets in a private catalog-config repo, and get the same boundaries as team-checkout.
The rules
Every change through Git. No kubectl apply except reading. Platform changes by pull request to argocd-platform.
Done means
Both apps Synced and Healthy, the team account limited to its project, a sealed secret in use, a PreSync migration in dev, and an incident handled with the five answers.
When stuck
Compare with checkout, which is the worked example. Then the hints. Then the solution.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. Platform: the catalog platform pieces (Challenge)

Done when:

Hint 1

Copy the checkout files and replace checkout with catalog; read each line you change.

Hint 2

The role lines in policy.csv follow the checkout ones; remember the g, line that binds the account.

One solution

One pull request to the platform repo with four files, then a password for the new account.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Create ~/work/argocd-platform/projects/catalog.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: catalog
  namespace: argocd
spec:
  description: team-catalog. Only its own repos, only its own namespaces.
  sourceRepos:
    - https://github.com/YOUR_GITHUB_USER/checkout-app.git           # catalog reuses checkout's chart
    - [email protected]:YOUR_GITHUB_USER/catalog-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: catalog-dev
    - server: https://kubernetes.default.svc
      namespace: catalog-prod
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: ''
      kind: LimitRange
    - group: networking.k8s.io
      kind: NetworkPolicy
    - group: rbac.authorization.k8s.io
      kind: Role
    - group: rbac.authorization.k8s.io
      kind: RoleBinding
  orphanedResources:
    warn: true

Create ~/work/argocd-platform/namespaces/catalog.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: catalog-dev
  labels:
    team: catalog
    env: dev
---
apiVersion: v1
kind: Namespace
metadata:
  name: catalog-prod
  labels:
    team: catalog
    env: prod

Add to ~/work/argocd-platform/argocd/values.yaml under cm:, after accounts.checkout-dev

    accounts.catalog-dev: login      # team-catalog

Add to ~/work/argocd-platform/argocd/values.yaml at the end of policy.csv

      # team-catalog: its own project only.
      p, role:catalog-team, applications, get, catalog/*, allow
      p, role:catalog-team, applications, sync, catalog/*, allow
      p, role:catalog-team, logs, get, catalog/*, allow
      p, role:catalog-team, projects, get, catalog, allow
      g, catalog-dev, role:catalog-team

git add -A && git commit -qm "Onboard team-catalog: project, namespaces, account" && git push -q

argocd app get root --refresh >/dev/null
argocd app wait argocd projects namespaces --sync --health

Set the new account's password:

argocd account update-password --account catalog-dev --current-password 'choose-a-long-one' --new-password 'catalog-long-one'

2. team-catalog: a private environment repo (Challenge)

Done when:

Hint 1

Module 08 has every command; only the names change.

Hint 2

Seal each catalog-db Secret for its own namespace (-n catalog-dev), or it will not decrypt (module 08's break-it).

One solution 
cd ~/work && gh repo create YOUR_GITHUB_USER/catalog-config --private --clone

cd catalog-config && git switch -C main
mkdir -p helm secrets/dev secrets/prod

Create ~/work/catalog-config/helm/dev.yaml

nameOverride: catalog
message: catalog (dev)
database:
  secretName: catalog-db
migration:
  enabled: true

Create ~/work/catalog-config/helm/prod.yaml

nameOverride: catalog
message: catalog (prod)
replicas: 2
database:
  secretName: catalog-db

Seal a database password for each environment, and commit:

for env in dev prod; do kubectl create secret generic catalog-db -n catalog-$env --dry-run=client -o yaml --from-literal=password="$(openssl rand -hex 16)" | kubeseal --cert ~/work/sealing-cert.pem -o yaml > secrets/$env/catalog-db.yaml; done

git add -A && git commit -qm "catalog: values and sealed secrets" && git push -qu origin main

Create a deploy key and register its public half:

ssh-keygen -q -t ed25519 -N '' -C argocd-catalog-config -f ~/work/keys/catalog-config

gh repo deploy-key add ~/work/keys/catalog-config.pub -R YOUR_GITHUB_USER/catalog-config --title argocd

In the platform repo, seal the private half as a repository credential:

cd ~/work/argocd-platform && git switch -q main && git pull -q

kubectl create secret generic repo-catalog-config -n argocd --dry-run=client -o yaml --from-literal=type=git \
    [email protected]:YOUR_GITHUB_USER/catalog-config.git --from-file=sshPrivateKey=~/work/keys/catalog-config \
  | kubectl label --local -f - argocd.argoproj.io/secret-type=repository -o yaml \
  | kubeseal --cert ~/work/sealing-cert.pem -o yaml > secrets/repo-catalog-config.yaml

git add -A && git commit -qm "Sealed deploy key for catalog-config" && git push -q

3. Platform: the catalog Applications, then ship to prod (Challenge)

Done when:

Hint 1

Copy the checkout Application files and replace the names; the chart source stays checkout-app.

Hint 2

If the apps are Unknown, run argocd repo list: is the catalog repo Successful?

One solution

Two Application files, then one pull request. catalog-prod is the same with prod instead of dev.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Create ~/work/argocd-platform/apps/catalog-dev.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: catalog-dev
  namespace: argocd
  labels:
    team: catalog
    env: dev
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: catalog
  sources:
    - repoURL: https://github.com/YOUR_GITHUB_USER/checkout-app.git
      path: chart
      targetRevision: v1.0.0
      helm:
        valueFiles:
          - $config/helm/dev.yaml
    - repoURL: [email protected]:YOUR_GITHUB_USER/catalog-config.git
      targetRevision: main
      ref: config
    - repoURL: [email protected]:YOUR_GITHUB_USER/catalog-config.git
      targetRevision: main
      path: secrets/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: catalog-dev
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

git add -A && git commit -qm "catalog: dev and prod" && git push -q

argocd app get root --refresh >/dev/null
argocd app wait catalog-dev catalog-prod --sync --health --timeout 300

Log in as the team:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username catalog-dev --password 'catalog-long-one'

List the apps: expect only the two catalog apps.

argocd app list

Log back in as yourself:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'

4. The incident (Challenge)

An eager team member "saves memory" in prod. Apply their change, then handle it as team-catalog would: find it, fix it through Git, and write the five answers in an issue on catalog-config.

Done when:

cd ~/work/catalog-config && git pull -q

Add to ~/work/catalog-config/helm/prod.yaml at the end
Their change, as they made it.

resources:
  requests: { cpu: 10m, memory: 4Mi }
  limits: { memory: 4Mi }

git commit -qam "catalog: save memory in prod" && git push -q
Hint 1

Health first: which pods, which state? argocd app resources catalog-prod, then the pod events.

Hint 2

A container that is killed as soon as it starts, with a reason that mentions memory.

One solution

Health first:

argocd app get catalog-prod --refresh | grep -E 'Health|Deployment'

The pods show the reason (OOMKilled or CrashLoopBackOff):

kubectl -n catalog-prod get pods

Revert the bad commit, and wait for recovery:

cd ~/work/catalog-config && git revert --no-edit HEAD && git push -q

argocd app wait catalog-prod --health --timeout 240

Write it up:

gh issue create -R YOUR_GITHUB_USER/catalog-config --title "Incident: catalog-prod OOMKilled" --body "What failed: ... How detected: ... Who responded: ... Corrected: revert ... Prevented: ..."

5. Tear down

Your three (four) repos keep everything. Recreating the cluster and running the bootstrap gives you this setup back, except that SealedSecrets must be resealed for the new cluster's key.

kind delete cluster --name argocd
On an existing cluster 
kubectl delete namespace argocd checkout-dev checkout-prod catalog-dev catalog-prod traefik

Simplified for the course

Pull requests. From module 07 on you pushed straight to main so the lessons stayed on Argo CD. That is not how a team works. The right way is what modules 04 to 06 showed: every change goes on a branch, opens a pull request, passes the validate check, gets a review, and is merged. Protect main in all three repos, and let nobody (including admins) bypass it. Go back and do your last few changes that way once, to make it a habit.

Quiz

  1. What did onboarding team-catalog require from the platform team?
    • A project, namespaces, an account and role, a sealed repo credential, and two Application files: all reviewed pull requests
    • A separate Argo CD installation for that team, plus its own cluster, ingress and monitoring stack, all set up by hand by the platform engineers
    • Nothing at all: the team could create the project, namespaces and Applications itself through the Argo CD UI, with no pull request involved
  2. Where do you look first when an app is Degraded?
    • Argo CD's own logs first, since every sync error is written there before anywhere else
    • The resource tree, then the failing object's events and logs
    • Git history for the last commit, then revert it before looking at the cluster at all
  3. What would you automate next if a third team arrived?
    • Nothing, since two teams onboarded by hand is how it is meant to be done, and the same process scales to any number of teams
    • More kubectl scripts that apply each team's files straight to the cluster in the right order, run by whoever is on call that day
    • Generating the project, namespaces and Applications from one file per team: ApplicationSets, the next course
Answers
  1. A project, namespaces, an account and role, a sealed repo credential, and two Application files: all reviewed pull requests The platform repo is the onboarding interface. Fleet GitOps turns it into one file per product.
  2. The resource tree, then the failing object's events and logs Degraded is about the workload, not about Argo CD.
  3. Generating the project, namespaces and Applications from one file per team: ApplicationSets, the next course Copy-and-rename onboarding does not scale. That is where Fleet GitOps starts.

Field note

If you finished this without the solutions, you can run Argo CD for a team. If you needed them, that is normal: run the capstone again in a week, from a fresh cluster, and see how much you remember.

Report a problem with this module