Module 05 · Real-world delivery · 55 min · You are: Platform and product

Helm and Kustomize in practice

Almost nothing in real clusters is plain YAML. Platform add-ons come as Helm charts, and teams describe environments as overlays. Argo CD renders both itself, so you need to know what it does differently from helm install, and how to see what it will apply before it applies it.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

Argo CD renders Helm

The repo-server runs helm template and applies the output. There is no Helm release: helm list shows nothing, and Helm hooks become Argo CD hooks.

Values: in Git, two ways

helm.valuesObject inside the Application (fine for add-ons with a few settings) or value files in a repo (module 07). Both are in Git.

What Kustomize is, and why

A tool, built into kubectl, that changes plain Kubernetes YAML without editing the original files and without a template language. Dev and prod need almost the same YAML: copies drift apart, and templates are hard to read. Kustomize keeps one copy plus a small change per environment.

Base + overlays

The base is ordinary YAML every environment shares. An overlay is a folder that points at the base and lists only what differs: image tag, replica count, a patch. Each overlay is one environment. Argo CD needs no plugin: a folder with a kustomization.yaml is built with kustomize build automatically.

Helm or Kustomize?

Helm is a package manager for reusable, configurable applications. Kustomize customises manifests you already have. Many teams use both. The table below helps you choose.

Parameter overrides

argocd app set -p ... and the UI Parameters tab change the Application, not Git. Treat them as a debugging tool, never a deploy method.

How Argo CD picks the tool

Helm or Kustomize?

A common pattern is both together: install a vendor chart with Helm, and let Kustomize patch what the chart does not expose (Kustomize can inflate a chart with helmCharts:). This course keeps them separate: Helm for add-ons here, then for checkout's own chart in module 07.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• platform/apps/traefik.yaml (reference): A platform add-on from a public chart repository.
  1. chart: and a pinned version instead of path: L14–16
  2. The few values that differ from the chart defaults, in Git L18–29
  3. Add-on namespaces are created by Argo CD L37–38
• checkout-config/kustomize/dev/kustomization.yaml (reference): dev: the base plus what differs.
  1. Start from the base L4–5
  2. The newest version runs in dev L6–8
• checkout-config/kustomize/prod/kustomization.yaml (reference): prod: an older, proven version and two replicas.
  1. Promoting means changing this line L8
  2. Two replicas in prod L9–11

Steps

1. Platform: an ingress controller from a Helm chart

An ingress controller is the front door of a cluster: it receives web traffic from outside and routes it to the right service. Traefik will be the front door for Argo CD's UI. You install it the way you install any add-on: an Application that points at the public Helm chart, added by pull request. On kind it listens on fixed ports that the cluster maps to your laptop.

Existing cluster: if you already run an ingress controller, skip Traefik and use your ingress class in module 06.

cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/argocd-gitops/platform/apps/traefik.yaml apps/

Change in ~/work/argocd-platform/apps/traefik.yaml

-   project: platform
+   project: default

Propose it as a pull request. Put it on its own branch and push it:

git switch -c traefik && git add -A && git commit -qm "Add-on: Traefik ingress controller"
git push -qu origin HEAD

Open the pull request. It prints a link:

gh pr create --fill

Open that link in your browser. Files changed shows exactly your diff. In the Checks tab, wait for validate to turn green (or run gh pr checks --watch).

Merge it in the portal (Squash and merge, then Delete branch), or from the terminal:

gh pr merge --squash --delete-branch

Either way, bring your copy up to date:

git switch -q main && git pull -q

Root creates the traefik Application from the new file. Wait until it is healthy:

argocd app get root --refresh >/dev/null
argocd app wait traefik --health --timeout 300

Knock on the front door. Expect 404: Traefik answers, but has no routes yet, so it has nothing to show. A 404 here is the good result; no answer at all would mean it is not running.

curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/

kubectl get ingressclass      # note your class name; skip Traefik

2. See what Argo CD rendered, and that there is no Helm release

Argo CD did not run helm install. It turned the chart into plain manifests itself and applied those. Two commands show it.

What Argo CD rendered from the chart, counted by kind of object. Expect a list like Deployment, Service, ServiceAccount, ClusterRole: the pieces of Traefik.

argocd app manifests traefik | grep -E '^kind:' | sort | uniq -c

What Helm knows about. Expect no traefik row: Helm never installed it.

helm list -A

3. Kustomize in five minutes, on a scratch folder

Kustomize is new to many people, so try it before using it on checkout. You build a tiny base and one overlay in a scratch folder (not in a Git repo, nothing touches the cluster), and kubectl kustomize shows what Argo CD would build. The base is a plain Deployment that runs 1 replica of nginx 1.27. The overlay says only: "3 replicas, nginx 1.28".

mkdir -p ~/work/kustomize-demo/base ~/work/kustomize-demo/overlays/prod

The base: a normal Deployment. It is valid on its own; you could kubectl apply it.

Create ~/work/kustomize-demo/base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: nginx:1.27

The base's kustomization.yaml. The file that makes the folder a Kustomize folder. It lists the files to build:

Create ~/work/kustomize-demo/base/kustomization.yaml

resources:
  - deployment.yaml

Render the base. Expect your Deployment back unchanged: replicas: 1 and image: nginx:1.27.

kubectl kustomize ~/work/kustomize-demo/base | grep -E 'replicas|image:'

The prod overlay. resources points at the base. replicas and images are the only differences. The base file is never edited.

Create ~/work/kustomize-demo/overlays/prod/kustomization.yaml

resources:
  - ../../base
replicas:
  - name: hello
    count: 3
images:
  - name: nginx
    newTag: "1.28"

Render the overlay. Same command, other folder. Expect replicas: 3 and image: nginx:1.28: the base with the two differences applied.

kubectl kustomize ~/work/kustomize-demo/overlays/prod | grep -E 'replicas|image:'

You now know the whole model. The next step uses it on checkout: a base, a dev overlay and a prod overlay. Argo CD runs this same render whenever an Application points at a folder with a kustomization.yaml.

4. team-checkout: Kustomize overlays

Kustomize builds each environment from one shared base plus a small overlay that holds only the differences. Copy them in, then render each overlay on your laptop the way Argo CD will, before you push.

cd ~/work/checkout-config && git switch -q main && git pull -q
cp -r ~/work/argocd-gitops/checkout-config/kustomize .

Render dev. kubectl kustomize builds the final YAML; grep keeps the three lines that differ between environments. Expect image 6.7.1, one replica, banner checkout (dev).

kubectl kustomize kustomize/dev | grep -E 'image:|replicas|value: checkout'

Render prod. Expect image 6.7.0, replicas: 2 and banner checkout (prod): same base, different overlay.

kubectl kustomize kustomize/prod | grep -E 'image:|replicas|value: checkout'

Propose it as a pull request. Put it on its own branch and push it:

git switch -c kustomize && git add -A && git commit -qm "checkout: Kustomize base and overlays"
git push -qu origin HEAD

Open the pull request. It prints a link:

gh pr create --fill

Open that link in your browser. Files changed shows exactly your diff.

Merge it in the portal (Squash and merge, then Delete branch), or from the terminal:

gh pr merge --squash --delete-branch

Either way, bring your copy up to date:

git switch -q main && git pull -q

5. Platform: checkout-dev on the overlay, and a new checkout-prod

Two Applications, same repo, different paths. In checkout-dev, point path at the overlay and add a finalizer (deleting the Application then deletes what it deployed). Then create checkout-prod: the same file with prod instead of dev.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Change in ~/work/argocd-platform/apps/checkout-dev.yaml

-     path: manifests
+     path: kustomize/dev

Add to ~/work/argocd-platform/apps/checkout-dev.yaml under metadata:, after namespace: argocd

  finalizers:
    - resources-finalizer.argocd.argoproj.io   # deleting the app deletes what it deployed

Create ~/work/argocd-platform/apps/checkout-prod.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # deleting the app deletes what it deployed
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_GITHUB_USER/checkout-config.git
    targetRevision: main
    path: kustomize/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Propose it as a pull request. Put it on its own branch and push it:

git switch -c checkout-kustomize && git add -A && git commit -qm "checkout: Kustomize overlays, add prod"
git push -qu origin HEAD

Open the pull request. It prints a link:

gh pr create --fill

Open that link in your browser. Files changed shows exactly your diff. In the Checks tab, wait for validate to turn green (or run gh pr checks --watch).

Merge it in the portal (Squash and merge, then Delete branch), or from the terminal:

gh pr merge --squash --delete-branch

Either way, bring your copy up to date:

git switch -q main && git pull -q

Root applies the changed and the new Application. Wait until both are Synced and Healthy:

argocd app get root --refresh >/dev/null; sleep 10
argocd app wait checkout-dev checkout-prod --sync --health --timeout 180

6. Compare the environments

Same app, same repo, two environments. Check that prod got its own settings.

What Argo CD says about prod. Expect Synced and Healthy, and the Deployment line Synced:

argocd app get checkout-prod | grep -E 'Deployment|Sync Status|Health'

What really runs in prod. Expect 2 replicas and the image ending in 6.7.0:

kubectl -n checkout-prod get deploy checkout -o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].image}'; echo

See it. Run this in a new terminal and leave it running, then open http://localhost:9899. Expect the banner checkout (prod) and version 6.7.0 (dev shows 6.7.1).

kubectl -n checkout-prod port-forward svc/checkout 9899:80

Break it on purpose: A promotion that does nothing

team-checkout promotes 6.7.1 to prod. The engineer copies the image lines from a blog post, which leaves out the registry: stefanprodan/podinfo instead of ghcr.io/stefanprodan/podinfo.

You practise: checking what Argo CD rendered instead of trusting Synced, and rendering an overlay yourself before you push it.

1. Break it

cd ~/work/checkout-config && git switch -q main && git pull -q

Change in ~/work/checkout-config/kustomize/prod/kustomization.yaml

-   - name: ghcr.io/stefanprodan/podinfo
-     newTag: 6.7.0
+   - name: stefanprodan/podinfo
+     newTag: 6.7.1

Propose it as a pull request. Put it on its own branch and push it:

git switch -c promote-prod && git add -A && git commit -qm "checkout: promote 6.7.1 to prod"
git push -qu origin HEAD

Open the pull request. It prints a link:

gh pr create --fill

Open that link in your browser. Files changed shows exactly your diff.

Merge it in the portal (Squash and merge, then Delete branch), or from the terminal:

gh pr merge --squash --delete-branch

Either way, bring your copy up to date:

git switch -q main && git pull -q

argocd app get checkout-prod --refresh >/dev/null; sleep 15

2. What you see

Argo CD is happy: checkout-prod is Synced and Healthy, and its history shows your commit. Expected: prod runs 6.7.1. You get: kubectl -n checkout-prod get deploy checkout -o jsonpath='{..image}' still prints ghcr.io/stefanprodan/podinfo:6.7.0. No error anywhere.

3. Fix it yourself

Find out why prod still runs 6.7.0 without guessing. Prove your fix on your laptop before you push it, then ship it. Finally, put prod back on 6.7.0: the promotion was practice.

cd ~/work/checkout-config && git switch -q main && git pull -q

-   - name: stefanprodan/podinfo
+   - name: ghcr.io/stefanprodan/podinfo

kubectl kustomize kustomize/prod | grep image:

git switch -c fix-promote-prod && git add -A && git commit -qm "checkout: fix the image name for prod"
git push -qu origin HEAD

gh pr create --fill

gh pr merge --squash --delete-branch

git switch -q main && git pull -q

argocd app get checkout-prod --refresh >/dev/null
argocd app wait checkout-prod --sync --health

kubectl -n checkout-prod get deploy checkout -o jsonpath='{..image}'; echo

-     newTag: 6.7.1
+     newTag: 6.7.0

git switch -c prod-back-to-6.7.0 && git add -A && git commit -qm "checkout: prod back on 6.7.0"
git push -qu origin HEAD

gh pr create --fill

gh pr merge --squash --delete-branch

git switch -q main && git pull -q

Quiz

1. You deploy a chart with Argo CD. What does helm list show?
   • Nothing: Argo CD renders with helm template and applies the output itself
   • The release, with its revision history, just as if you had run helm install yourself
   • An error, because Helm refuses to list releases that another tool manages
2. Where should the differences between dev and prod live?
   • In parameters typed into the Argo CD UI for each environment, saved on the Application
   • In overlays (or value files) in the environment repo, through pull requests
   • In the shared base, with if-statements that switch behaviour on the environment name
3. Why can't an override set with argocd app set survive here?
   • Overrides were removed in Argo CD 3, so the command now fails before it changes anything
   • A kubectl cron job in the cluster resets every Application parameter once an hour
   • root self-heals the Application back to its manifest in Git

Field note

Run kubectl kustomize or helm template before every push, as you just did. Nine out of ten "Argo CD is broken" questions in my workshops were answered by rendering locally.

Report a problem with this module