Module 04 · Platform foundation · 40 min · You are: Platform team

Platform add-ons on every cluster

Platform capabilities only count if every cluster has them, at the same version, configured the same way. A matrix of "every cluster times every add-on" makes that a property of the design rather than of discipline.

Open this module in the interactive course to track progress and switch tracks.

Change journey practised: Requested → Proposed → Validated → Previewed → Approved → Merged → Reconciled → Healthy

Concepts

Matrix generator

Combines two generators: clusters from the inventory times a list of add-ons. 2 clusters times 7 add-ons = 14 Applications from one file.

Pinned versions

Every chart has an exact targetRevision. Renovate opens a pull request to bump it, which goes through the same journey.

Order is eventual

Sync waves express intent (CRDs before users), but generated Applications are independent. Something that needs a CRD fails and retries until it exists.

Add a cluster, get everything

A new inventory file produces all of its add-on Applications. The plan comment lists them before you merge.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• applicationsets/platform-helm-addons-appset.yaml (reference, your repo): Every add-on on every cluster, from one file.
  1. Child 1: every cluster in the inventory L9–15
  2. Child 2: one entry per add-on, version pinned (first of seven) L18–25
  3. Every combination is an Application: dev-cert-manager, prod-kyverno, ... L75
• renovate.json (reference, your repo): Version bumps arrive as pull requests.
  1. Teaches Renovate to read the add-on list L18–25
  2. All add-on bumps in one reviewable PR L49

Steps

1. Propose the add-ons

The plan comment should list 14 new Applications: seven per cluster.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c addons

Copy the add-on ApplicationSet and point it at your repo:

git checkout reference/main -- applicationsets/platform-helm-addons-appset.yaml
scripts/set-repo-url.sh

List what the fleet would contain. head -40 keeps the first 40 lines. Expect each cluster with its add-ons: cert-manager, ingress, kyverno and so on, 14 in total.

python3 scripts/fleet_check.py | head -40

git add -A && git commit -m "Platform add-ons on every cluster"
git push -u origin HEAD

Open the pull request ↗

2. Merge and watch them roll out

Give it a few minutes. On kind, Docker needs about 8 GB of memory for everything.

Watch the add-on Applications appear on the hub. Wait until they are Synced and Healthy, then press Ctrl-C.

kubectl --context kind-platform -n argocd get applications -l platform-addon -w

See the add-ons running on dev. Expect pods for cert-manager, ingress, kyverno, rollouts and external-secrets, all Running:

kubectl --context kind-dev get pods -A | grep -E 'cert-manager|ingress|kyverno|rollouts|external-secrets'

3. Enable Renovate (optional)

Install the Renovate GitHub app on your repo. Its pull requests bump chart versions and go through validate and plan like yours.

Break it on purpose: Add an add-on twice

Someone wants Kyverno with three admission controller replicas, and copies the kyverno entry with new values instead of changing the existing one.

You practise: reading what an ApplicationSet will generate before it does, and why every generated Application needs a unique name.

1. Break it

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c duplicate-addon

Add to ~/work/fleet-gitops/applicationsets/platform-helm-addons-appset.yaml in the add-on list, right after the existing kyverno entry (same indentation)
Kyverno a second time, with different values.

                - addon: kyverno
                  namespace: kyverno
                  chart: kyverno
                  repoURL: https://kyverno.github.io/kyverno
                  targetRevision: 3.2.6
                  syncWave: "6"
                  values: "admissionController: {replicas: 3}"

git add -A && git commit -m "Kyverno with 3 admission replicas"
git push -u origin HEAD

Open the pull request.

2. What you see

The validate check fails: duplicate Application name dev-kyverno. Expected (by the author): a second Kyverno with new values. Without the check, the ApplicationSet controller would refuse to generate two Applications with one name and report an error on the ApplicationSet.

3. Fix it yourself

Keep the intent (three admission controller replicas) with a single kyverno entry, and get the check green. Then close the pull request without merging: on a laptop one replica is plenty.

-                 - addon: kyverno
-                   namespace: kyverno
-                   chart: kyverno
-                   repoURL: https://kyverno.github.io/kyverno
-                   targetRevision: 3.2.6
-                   syncWave: "6"
-                   values: "admissionController: {replicas: 3}"

-                   syncWave: "6"
-                   values: ""
+                   syncWave: "6"
+                   values: "admissionController: {replicas: 3}"

python3 scripts/fleet_check.py --quiet

git commit -am "Kyverno: 3 admission replicas, one entry" && git push

git switch main && git branch -D duplicate-addon && git push origin --delete duplicate-addon

Quiz

1. How many Applications does a matrix of 3 clusters and 7 add-ons generate?
   • 21
   • 10 Applications, because 3 clusters plus 7 add-ons is simply added up
   • 7 Applications, because each add-on is deployed once and shared by all of the clusters
2. How do chart versions get updated?
   • Argo CD picks the latest chart version on its own whenever a new one is published
   • Renovate opens a pull request bumping the pinned version
   • Someone logs in to each cluster and bumps the version by hand during a maintenance window
3. An Application needs a CRD another Application installs. What happens if it syncs first?
   • Argo CD orders all Applications globally and always syncs the one that owns the CRD first
   • The whole cluster breaks, and the CRD has to be applied by hand before any sync can work
   • It fails and retries until the CRD exists

Field note

Pin everything and let a bot propose the bumps. The worst outage I have cleaned up started with a chart floating to a new major on a Friday.

Report a problem with this module