PlatformEngr® Fleet GitOps course

Cheat notes · Fleet GitOps

Fleet GitOps: cheat notes

Every module on one page, cut to what you need to remember: the idea, the commands, the pitfalls and a few questions to test yourself. Made for review after you have done the course, not instead of it. Back to the course.

00 Orientation: the platform you will run

This module only · Full module

The idea

A platform team and a product team share a fleet of clusters, and every change either makes travels the same journey: issue, pull request, checks, approval, merge, then Argo CD reconciles it until it is healthy.

Remember

  • Two roles: the platform team owns clusters, Argo CD, policies and the namespace baseline. Product teams (team-checkout) own their deployment config. CODEOWNERS turns that split into review rules.
  • The change journey: requested, proposed, validated, previewed, approved, merged, reconciled, healthy. Each module practises some of these steps.
  • Reference repo vs your repo: the reference is the finished fleet. Your repo starts empty and takes one folder at a time from it, by pull request. Argo CD watches your repo, never the reference.
  • Required vs flexible: GitHub (issues, pull requests, Actions) and Git as the source of truth are required. Where Kubernetes runs is flexible: kind by default, AKS, EKS, GKE or any cluster.
  • End result vs full course: the quick start builds the finished fleet in 30 minutes as a preview; the course builds it piece by piece and explains each piece.

Pitfalls

  • Changing things with kubectl because it is quicker. The fleet only changes reliably through the repo Argo CD watches; the habit is the lesson.
  • Pointing Argo CD at the reference repo: you cannot merge there, so your pushes do nothing.

Test yourself

Answer in your head first, then open the answer.

Why does Argo CD watch your repo and not the reference repo?

GitOps is pull-based. You change the fleet by merging into the repo it watches, so it must be one you can merge into.

Who owns a product's namespace baseline (quota, network policy, RBAC)?

The platform team. Product teams consume it; that is what makes it consistent.

How does the quick start differ from the full course?

Same fleet. The quick start creates it from the template; the course builds and explains each piece.

Where module 00 leaves you: Nothing is built yet. Your details are saved in the course, every tool is checked, and you know what you are about to build and in which order.

01 Your fleet repo and the change workflow

This module only · Full module

The idea

Protect the process before you build anything. One bootstrap commit brings CI, issue forms and review rules; from then on main only accepts pull requests that pass validate and plan, starting with the cluster inventory.

Remember

  • One bootstrap commit: an empty repo cannot receive a pull request, so .github/ and scripts/ go in with one direct push. It is the only direct push in the course.
  • Required checks: validate runs scripts/validate.sh (YAML, schemas, references, ownership, policies, Terraform). plan comments which Argo CD Applications the change adds, removes or re-targets.
  • Branch protection: require a pull request, require validate and plan, and do not allow bypassing, so it applies to the owner too. Allow squash merging only: one commit per change keeps reverts simple.
  • CODEOWNERS: maps folders to owners. The platform team owns everything by default; product teams own their deployment folders.
  • The production environment with required reviewers is how prod approvals work later (production-gate).
  • The inventory clusters/*.yaml is the single source of truth for which clusters exist. Terraform and every ApplicationSet read it.

Pitfalls

  • Copying an API address from your own kubeconfig into the inventory (https://127.0.0.1:6443). validate fails and merge stays blocked: the hub needs an address it can reach, https://dev-control-plane:6443.
  • Forgetting scripts/set-repo-url.sh after a copy: manifests still point at the reference repo and pushes "do nothing". fleet_check.py now fails on it.
  • The required checks only appear in the branch rule after they have run once: come back after the first pull request.

Commands

git remote add reference https://github.com/platformengr/fleet-gitops.git && git fetch referenceThe finished fleet as a second remote, to copy from
git checkout reference/main -- FOLDERBring one folder from the reference into your branch
scripts/set-repo-url.shPoint copied files at your repo (--codeowners @you the first time)
scripts/validate.shRun CI's checks locally
python3 scripts/fleet_check.pyInventory and fleet checks; lists the Applications the fleet would contain

Test yourself

Answer in your head first, then open the answer.

Why does the bootstrap commit bring .github/ and scripts/ before any manifest?

So every later change, starting with the inventory, is validated and reviewed.

What does the plan check add on top of validate?

A comment saying which Argo CD Applications the change adds, removes or re-targets: the effect on the fleet, not just a YAML diff.

Where do you add a third cluster?

A new file in clusters/, through a pull request.

Where module 01 leaves you: A repo with CI, issue forms and review rules; main only accepts pull requests that pass validate and plan; and clusters/dev.yaml and clusters/prod.yaml merged through that process. No cluster exists yet.

02 Clusters as code with Terraform

This module only · Full module

The idea

The inventory decides which clusters exist and Terraform makes it so: it creates the clusters, installs Argo CD on the hub and registers each spoke with labels copied from the inventory. Apply from main, never from a branch.

Remember

  • The bootstrap chain: inventory, clusters, Argo CD, cluster registration, then (module 03) the root app. Terraform owns the chain up to the point where Git takes over.
  • Inventory drives Terraform: infrastructure/kind reads every clusters/*.yaml with provider: kind. Add a file and terraform apply creates the cluster.
  • Declarative registration: a cluster is a Secret in argocd labelled argocd.argoproj.io/secret-type: cluster. Terraform writes it with fleet.gitops/environment, region and upgrade-wave labels from the inventory; ApplicationSets select clusters by them.
  • Stable addresses: kind nodes share a Docker network, so the hub reaches dev-control-plane:6443 by name. Rebuilds keep the address, and the API certificate is valid for it, so TLS verification stays on.
  • Read the plan before yes: here 3 clusters, Argo CD, 2 cluster Secrets and no root app.

Pitfalls

  • Applying from a branch: what runs is then something nobody reviewed. Merge first, apply from main.
  • A lost cluster is not a crisis: terraform plan shows it will be recreated, terraform apply brings it back with the same name and address and fresh credentials in the cluster Secret.
  • Using 127.0.0.1 for a spoke: the hub runs in its own container, so that address is the hub itself.

Commands

cd infrastructure/kind && terraform initDownload the providers (once)
terraform planPreview; changes nothing
terraform applyCreate or repair the fleet's infrastructure
kubectl --context kind-platform -n argocd get secrets -l argocd.argoproj.io/secret-type=cluster --show-labelsThe clusters Argo CD knows, with their labels
terraform output registered_clustersTerraform's view of the registrations
kubectl --context kind-platform -n argocd port-forward svc/argocd-server 8080:443Reach the Argo CD UI on the hub

Test yourself

Answer in your head first, then open the answer.

Where do the labels on an Argo CD cluster Secret come from?

Terraform copies them from clusters/*.yaml.

Why did this module's plan not create the root app?

root-app.yaml was not in the repo yet; the fleet-hub module only applies it when it exists.

Why https://dev-control-plane:6443 and not your kubeconfig address?

It is reachable from the hub on the kind network, stable across rebuilds, and valid for the API certificate.

Where module 02 leaves you: Three kind clusters: a hub running Argo CD, plus dev and prod. Argo CD knows dev and prod, with labels copied from your inventory. Argo CD deploys nothing yet.

03 Hand the fleet over to Git

This module only · Full module

The idea

Terraform built the clusters; a root Application hands everything that runs on them to Git. Root points at applicationsets/, and an ApplicationSet turns the cluster list into one Application per cluster.

Remember

  • Root Application: root-app.yaml points Argo CD at applicationsets/ in your repo (app of apps). Terraform creates it once the file exists.
  • ApplicationSet: a template plus a generator. The git files generator reads clusters/*.yaml and creates one Application per cluster: dev-platform-namespaces, prod-platform-namespaces.
  • AppProject fleet-platform: the project for platform-owned Applications.
  • After the handover Terraform only runs when clusters or the hub bootstrap change. Everything on the clusters goes through Git and Argo CD.
  • Self-heal: a manual change survives only until the next comparison. Per-environment differences go in Kustomize overlays, reviewed.

Pitfalls

  • Deleting a namespace Git declares with kubectl: it is back within seconds, with a new age. Make the change in the environment's overlay, by pull request.
  • Putting logic in the root app. Keep it boring; it is what you need on the day you rebuild a hub.

Commands

kubectl --context kind-platform -n argocd get applications -wWatch root create the fleet's Applications
kubectl --context kind-dev get namespaces -l environment=devWhat Argo CD created on a spoke
kubectl kustomize addons/namespaces/overlays/devRender an environment's overlay before you push
git diff root-app.yamlCheck repoURL is your repo after set-repo-url.sh

Remove one object in one environment

# addons/namespaces/overlays/dev/kustomization.yaml
patches:
  - patch: |-
      $patch: delete
      apiVersion: v1
      kind: Namespace
      metadata:
        name: platform-system

Test yourself

Answer in your head first, then open the answer.

What does root-app.yaml point at?

applicationsets/. The ApplicationSets generate everything else.

You delete a namespace that Git declares. What happens?

Argo CD recreates it (self-heal). Terraform is not involved after the handover.

When does Terraform run after this module?

Only when clusters or the hub bootstrap change.

Where module 03 leaves you: Git runs the fleet. Argo CD syncs dev-platform-namespaces and prod-platform-namespaces from your repo, and from now on Terraform only runs when clusters change.

04 Platform add-ons on every cluster

This module only · Full module

The idea

One matrix ApplicationSet, every cluster times every add-on, puts the same pinned add-ons on all clusters. Consistency becomes a property of the design, and a new cluster gets everything for free.

Remember

  • Matrix generator: combines two generators, clusters from the inventory times a list of add-ons. 2 clusters × 7 add-ons = 14 Applications from one file.
  • Pinned versions: every chart has an exact targetRevision. Renovate opens pull requests to bump them, through the same validate and plan.
  • Order is eventual: sync waves express intent (CRDs before their users), but generated Applications are independent. Something that needs a CRD fails and retries until it exists.
  • Add a cluster, get everything: a new inventory file produces all its add-on Applications; the plan comment lists them before you merge.
  • The add-ons: ingress, cert-manager, external-secrets, metrics, Kyverno, Argo Rollouts and more, labelled platform-addon.

Pitfalls

  • Adding the same add-on twice with new values: duplicate Application names (dev-kyverno). CI catches it; without the check the ApplicationSet controller would refuse and report an error. One entry per add-on, values in that entry.
  • Floating chart versions: a chart that moves to a new major on its own is a classic Friday outage. Pin and let a bot propose bumps.
  • On kind, Docker needs about 8 GB of memory for every add-on.

Commands

python3 scripts/fleet_check.py | head -40Every cluster with the add-ons it will get
kubectl --context kind-platform -n argocd get applications -l platform-addon -wWatch the add-on Applications
kubectl --context kind-dev get pods -AThe add-ons running on a spoke

Test yourself

Answer in your head first, then open the answer.

How many Applications does a matrix of 3 clusters and 7 add-ons generate?

21: every combination.

How do chart versions get updated?

Renovate opens a pull request that bumps the pinned version.

An Application needs a CRD another Application installs, and syncs first. What happens?

It fails and retries until the CRD exists. Ordering between generated Applications is eventual.

Where module 04 leaves you: Every cluster runs the same pinned platform add-ons (ingress, certificates, secrets, metrics, policy engine, rollouts): 14 add-on Applications from one file, kept up to date by version-bump pull requests.

05 Guardrails and the namespace baseline

This module only · Full module

The idea

Self-service needs a consistent floor. One chart renders every product's namespace baseline from a few fields, and Kyverno policies stop anyone from eroding it: Audit in dev, Enforce in prod, and the same rules in CI.

Remember

  • Baseline as a chart: charts/product-namespace renders the Namespace, a ResourceQuota, a LimitRange, three NetworkPolicies and a RoleBinding from name, owner, cost center, Pod Security level and size.
  • Size profiles: teams pick small, medium or large; the platform decides what each means. More quota is a one-word pull request.
  • Policies: protect-platform-baseline blocks changes to baseline objects by anyone but the platform. require-requests-limits and disallow-latest-tag are Audit in dev, Enforce in prod (Kustomize overlays).
  • Same rules in CI: validate.sh runs the prod policies with the Kyverno CLI against every product's manifests, so violations fail the pull request, not the sync.
  • Prod changes need approval: a change under addons/*/overlays/prod makes production-gate wait for you.
  • The namespace carries Prune=false: removing a product from Git must not silently delete its data.

Pitfalls

  • An unpinned image (nginx:latest) with no requests: dev lets it in with warnings naming the policies, prod refuses it. Pin the tag and set requests and a memory limit.
  • Raising quota with kubectl: the policy blocks it on purpose. Change size in product.yaml by pull request.

Commands

helm template checkout charts/product-namespace -f product.yaml --set environment=prod --set size=mediumRender the baseline a product gets
kubectl --context kind-prod get clusterpoliciesThe policies on a cluster
kubectl --context kind-dev get clusterpolicy disallow-latest-tag -o jsonpath='{.spec.validationFailureAction}'Audit in dev, Enforce in prod

Test yourself

Answer in your head first, then open the answer.

A team needs more CPU quota in prod. What is the right change?

A pull request changing size for prod in its product.yaml.

Why are some policies Audit in dev and Enforce in prod?

Teams can experiment in dev and still see warnings; prod gets the hard rule.

Why is the product namespace annotated Prune=false?

Removing a product from Git must not silently delete its data. Deleting a namespace is a deliberate platform step.

Where module 05 leaves you: Kyverno policies run on both clusters (Audit in dev, Enforce in prod) and the product-namespace chart is ready to use. No product uses it yet: that is the next module.

06 Onboard a product, declaratively

This module only · Full module

The idea

A product team writes one product.yaml; three ApplicationSets turn it into an Argo CD project, a namespace baseline and the workload on every matching cluster. Products name environments; the platform decides which clusters those are.

Remember

  • Product declaration products/<name>/product.yaml: owner, cost center, repository and path, Pod Security level, and environments with a size each. request: links the onboarding issue.
  • Environments, not clusters: a product asks for prod; clusters labelled fleet.gitops/environment=prod receive it. A second prod cluster needs no product change.
  • Three ApplicationSets: product-projects (an AppProject on the hub), product-namespaces (the baseline per environment and cluster), product-apps (the workload, e.g. checkout-dev).
  • Nested matrix: products × their environments (an elementsYaml list) × the clusters whose label matches. One file in, checkout-dev and checkout-prod out.
  • The fence: each product's AppProject allows only its own namespace and no cluster-scoped resources.
  • Ownership is required metadata: every product folder needs a CODEOWNERS line, or fleet_check.py fails.

Pitfalls

  • Asking for an environment that no cluster provides (staging): fleet_check.py fails and names it. Open an "Add or change a cluster" issue instead.
  • Forgetting the CODEOWNERS line for a new product: CI fails, because ownership is required.

Commands

python3 scripts/fleet_check.pyWhat the platform will generate from the products
kubectl --context kind-platform -n argocd get applications -l fleet.gitops/product=checkoutA product's Applications
kubectl --context kind-platform -n argocd get appproject checkout -o jsonpath='{.spec.destinations}'The product's fence
kubectl --context kind-prod -n checkout get resourcequota,limitrange,networkpolicy,rolebindingThe baseline nobody on the team wrote

product.yaml

apiVersion: platform.platformengr.io/v1alpha1
kind: Product
product:
  name: checkout
  owner: team-checkout
  costCenter: cc-4711
  request: https://github.com/YOUR_GITHUB_USER/fleet-gitops/issues/2          # the onboarding issue
  repository: https://github.com/YOUR_GITHUB_USER/fleet-gitops.git           # where the deployment config lives
  path: product-deployments/checkout
  podSecurity: restricted
  environments:
    - environment: dev
      size: small
    - environment: prod
      size: medium

Test yourself

Answer in your head first, then open the answer.

Which file does a product team write to onboard?

products/<name>/product.yaml, plus its own deployment config.

What decides which clusters receive checkout-prod?

Clusters labelled fleet.gitops/environment=prod, from the inventory.

Can checkout's Application create a ClusterRole?

No. Its AppProject allows no cluster-scoped resources and only the checkout namespace.

Why does production-gate wait on an onboarding pull request?

A product declaration can reach prod, so it needs the production approval.

Where module 06 leaves you: checkout runs in dev (6.7.1) and prod (6.7.0, two replicas), each in a namespace with quota, limits, network policies and access rules, inside its own Argo CD project. A pull request for a second product, catalog, is open.

07 Ship, promote and roll back

This module only · Full module

The idea

Approvals sit where the change is decided, on the pull request: dev deploys after merge, prod waits for an approval, and Git plus GitHub Deployments are the audit trail. Merged is not running; trace it until it is reconciled.

Remember

  • Promotion is a pull request: dev and prod are folders; promoting means changing newTag in prod/kustomization.yaml. The diff is the release note.
  • The production gate: production-gate detects changes that reach prod and waits on the production environment. Non-prod changes skip it, and a skipped job counts as passing.
  • Deployment records: after merge, deployment-record creates a GitHub deployment per product environment, linked to the pull request and issue.
  • Reconciled, not merged: scripts/trace.sh shows, per Application, pending or reconciled for a commit.
  • Rollback is a revert, through the same checks and approval. The UI rollback disables auto-sync and leaves Git wrong.

Pitfalls

  • Pushing straight to main in a hurry: the push is rejected and your local main is ahead of GitHub. Move the commit to a branch (git switch -c hotfix then git branch -f main origin/main) and open a pull request.
  • Saying "deployed" when it is only merged. Deployment records say queued; check the trace or Argo CD.
  • Asking for approvals inside Argo CD. Keep Argo CD fully automatic and the approval on the pull request, so the history is in one place.

Commands

scripts/trace.sh --context kind-platform --selector fleet.gitops/product=checkoutHas Argo CD applied my commit yet, per Application?
git log --oneline -- product-deployments/checkout/prodProd's history: one commit per change
git revert --no-edit HEADRoll back, on a branch, through a pull request

Test yourself

Answer in your head first, then open the answer.

Does a dev-only change need the production approval?

No. The gate job is skipped, which counts as passing.

How do you roll back a bad prod release?

git revert in a pull request, through the same checks and approval.

What does scripts/trace.sh tell you that the merge does not?

Whether Argo CD has synced a revision containing the commit, per Application and cluster.

Where module 07 leaves you: Dev shows its new banner. Prod was promoted to 6.7.1 and then rolled back to 6.7.0 by a revert pull request. Git history and the GitHub Deployments page show who changed what, when.

08 Operate the fleet: failure drills

This module only · Full module

The idea

You learn a platform by watching it fail safely. For each failure answer five questions: what failed, how it was detected, who responds, how it is corrected, how it is prevented. Most failures that restore desired state need no human.

Remember

  • Detection: CI for proposed changes, Argo CD status for drift and sync failures, Kyverno at admission, terraform plan for infrastructure drift, logs and health for the rest.
  • Ownership: product failures go to product teams, platform failures to the platform team. Mixing them up is the most expensive incident pattern.
  • Correction: a fix that changes desired state is a pull request. A fix that restores it (self-heal, retries, rebuilds) needs nobody.
  • The five drills: a deleted namespace heals itself (retry); hand-scaled prod goes back to Git; a team cannot change its baseline (Kyverno, RBAC); an unreachable cluster shows Unknown and recovers; a deleted hub is rebuilt by terraform apply.
  • The hub is stateless: desired state lives in Git, registrations in Terraform. Nothing about the fleet lives only on the hub.
  • Prevention: each drill ends with the control that stops it. If there is none, that is the follow-up issue.

Pitfalls

  • Treating Unknown on a whole cluster's Applications as a Git problem: Argo CD cannot reach the cluster, so it cannot compare.
  • Never rehearsing the hub rebuild. The team that has done it on a quiet day handles the real one calmly.
  • Skipping the write-up. The incident note, written while you remember what you saw first, is what makes the operating model real.

Commands

kubectl --context kind-prod -n checkout delete networkpolicy default-deny-ingress --as=dev-user --as-group=team-checkoutAct as the team: refused by protect-platform-baseline
docker stop dev-control-planeTake a spoke offline (then docker start)
kind delete cluster --name platformLose the hub
cd infrastructure/kind && terraform applyRebuild it from code; Argo CD re-adopts every Application

Test yourself

Answer in your head first, then open the answer.

Who responds when a product's pod uses :latest and is rejected in prod?

The product team. The platform provided the guardrail and the message.

What does Argo CD need to rebuild the fleet after losing the hub?

Git and the cluster registrations, which Terraform recreates.

Why is the namespace a separate Application from the product workload?

Different owners and lifecycles: the baseline must exist and stay protected whatever the product config says.

Where module 08 leaves you: Everything is back to normal: the fleet healed itself or was rebuilt from code. You have an incident note in your repo, and you know who responds to each kind of failure.

09 Observability and change visibility

This module only · Full module

The idea

An incident asks two questions: what is happening, and what changed. Logs from every cluster in one Grafana answer the first; Argo CD links, deployment records and the trace script answer the second.

Remember

  • Agents per cluster, store on the hub: Alloy runs on every cluster and pushes logs to Loki on the hub; one Grafana for the fleet.
  • Reachability is a fleet fact: spokes cannot resolve the hub's cluster-local DNS, so the hub exposes the Loki gateway (a fixed NodePort here) and each inventory file sets lokiEndpoint.
  • Same labels everywhere: logs carry cluster and environment; Applications carry fleet.gitops/product, owner, environment.
  • From running app to change: Argo CD shows the synced commit and links to product.yaml and the config history; the namespace's fleet.gitops/request annotation links the onboarding issue; GitHub Deployments list who changed what.
  • CI proves shape, not function: an address can be valid YAML and still lead nowhere. Alert when a cluster stops sending logs.

Pitfalls

  • A pull request "moves" a cluster's Loki endpoint to a port where nothing listens: every check passes, every Application is Healthy, and that cluster's logs stop. Read the agent's logs; revert.
  • Naming things differently in Grafana, Argo CD and Git. When all three say checkout, people find things in seconds.

Commands

kubectl --context kind-platform -n observability port-forward svc/grafana 3000:80Open Grafana at localhost:3000
{cluster="prod", namespace="checkout"}LogQL in Grafana Explore: one product in prod (drop cluster for the whole fleet)
kubectl --context kind-prod get namespace checkout -o jsonpath='{.metadata.annotations.fleet\.gitops/request}'The issue that asked for this namespace
kubectl --context kind-dev -n observability-agents logs -l app.kubernetes.io/name=alloy --tail 30Debug a log route from the agent's side

Test yourself

Answer in your head first, then open the answer.

Why is lokiEndpoint in the inventory and not in the Alloy template?

How a spoke reaches the hub depends on its network: a per-cluster fact. Templates stay identical.

How do you find the onboarding request for a running namespace?

The fleet.gitops/request annotation on the namespace.

CI passed but logs stopped. What does that teach?

Validation proves a change is well-formed, not that it works. Runtime signals and alerts are the other half.

Where module 09 leaves you: Alloy agents on every cluster ship logs to Loki on the hub, Grafana shows both clusters, and you can trace a running app back to its pull request. dev\'s log route is back after the exercise.

10 Secrets and identity

This module only · Full module

The idea

The External Secrets setup from the Argo CD course, at fleet scale: the baseline renders a store into every product namespace on every cluster, production stores log in with workload identity instead of a password, and SSO groups mapped to projects limit each team to its own apps.

Remember

  • SecretStore per namespace: the baseline chart renders one in every product namespace (how to reach the vault), per environment. Only that namespace can use it, so there is no conditions list to keep. The platform owns it.
  • Same ExternalSecret as in Argo CD: the product team commits it next to the app; only secretStoreRef changes, to the namespaced store product-secrets.
  • Workload identity: on AKS, EKS and GKE the operator authenticates to the vault as a cloud identity bound to its ServiceAccount. No static credentials anywhere.
  • SSO and RBAC in Argo CD: admin.enabled: "false", logins through oidc.config, and policy.csv mapping a group to a role limited to checkout/*.
  • The fake provider keeps values in the SecretStore itself, so they are in Git. It exists to show the flow on kind; never use it for real secrets.

Pitfalls

  • An ExternalSecret for a key the store does not have: the ExternalSecret is Degraded and no Secret appears. The team's file was right; the platform adds the key. Make "add a secret to the store" a request with an owner.
  • Choosing secrets encrypted into Git for a fleet because they look simpler: key rotation across twenty clusters is where they stop being simple.

Commands

kubectl --context kind-dev -n checkout get externalsecret,secret checkout-dbSecretSynced, and the Secret it created
kustomize build product-deployments/checkout/dev | grep -A3 'kind: ExternalSecret'Check the ExternalSecret is in the render
kubectl -n NS annotate externalsecret NAME force-sync=$(date +%s) --overwriteAsk External Secrets to sync now

An ExternalSecret

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: SecretStore, name: product-secrets }
  target: { name: checkout-db }       # the Kubernetes Secret to create
  data:
    - secretKey: password
      remoteRef: { key: db-password }  # the key in the store

Test yourself

Answer in your head first, then open the answer.

What does an ExternalSecret in Git contain?

A reference: which key in which store becomes which Kubernetes Secret. No value.

What replaces static cloud credentials for the External Secrets operator?

Workload identity bound to its ServiceAccount.

How does team-checkout get access to only its own Applications in Argo CD?

SSO groups mapped to a role scoped to project checkout.

Where module 10 leaves you: checkout-dev has a Secret, checkout-db, created from a store the baseline rendered. Git holds only references, never values. You know how production swaps the stand-in store for a vault per environment, reached with workload identity.

11 Going to cloud, and splitting the repo

This module only · Full module

The idea

Everything above the cluster line is cloud-agnostic. In a real cloud two things change: how clusters are created and how Argo CD authenticates to them (workload identity, not tokens). As teams grow, one repo becomes three along ownership lines.

Remember

  • fleet-hub is portable: it needs a hub provider configuration and, per spoke, a server, labels and an Argo CD config. Feed it from your cloud's Terraform.
  • Identity per spoke: AKS: workload identity and argocd-k8s-auth azure. EKS: IRSA or Pod Identity and awsAuthConfig. GKE: workload identity and argocd-k8s-auth gcp. No tokens to rotate.
  • Three repositories: infrastructure-live (infrastructure/, clusters/: Terraform, slow, high blast radius), platform-fleet (root app, ApplicationSets, add-ons, charts, products, .github/), <product>-deployment (each team's config, with the production gate).
  • product.yaml stays in the platform repo, pointing at the team's repo: the declaration is the contract with the platform.
  • The cloud scripts leave one long-lived credential: a ServiceAccount token per spoke, stored on the hub. Workload identity replaces it.

Pitfalls

  • Splitting repos on day one. Split when a second team owns a product or infrastructure cadence starts blocking product changes; the folders make it a mechanical move.
  • Forgetting down.sh after the cloud track: clusters keep costing money.

Commands

infrastructure/cloud/CLOUD/up.sh --dry-runPrint every command the script would run; changes nothing
infrastructure/cloud/CLOUD/up.shCreate the cloud fleet (costs money while it runs)
infrastructure/cloud/CLOUD/down.shDelete it all
grep -n "bearerToken\|service-account-token" infrastructure/cloud/lib/argocd.shFind the long-lived token

Test yourself

Answer in your head first, then open the answer.

What has to change in fleet-hub to support EKS?

Nothing. You pass EKS spokes with an awsAuthConfig-based config.

Why split infrastructure into its own repository?

Different owners, cadence, permissions and blast radius.

After the split, where does checkout's product.yaml live?

In platform-fleet, pointing at checkout-deployment.

Where module 11 leaves you: A written plan: which folders go to which repository, who owns each, and what replaces the token with workload identity. If you ran the cloud track, an inventory pull request with real cluster addresses.

12 Reference architecture and the road to production

This module only · Full module

The idea

Every layer on one page: Terraform from the inventory builds clusters, Argo CD and the root app; ApplicationSets add platform capabilities and translate products; every change by every team travels the same journey. That journey, not the tools, is the operating model.

Remember

  • Infrastructure: Terraform from clusters/*.yaml: clusters, Argo CD, registrations, the root app.
  • Platform: ApplicationSets for add-ons, guardrails, observability and the product translation.
  • Products: one declaration per product; a baseline and apps per environment and cluster; promotion by pull request.
  • Operations: required checks, plan comments, production approvals, deployment records, trace, logs, drills.
  • One inventory is consumed by every layer: the single source of truth for which clusters exist.
  • Your repo can rebuild the fleet: tear it down with terraform destroy; nothing is lost.

Pitfalls

  • Drawing the architecture only from the diagram. Sketch it from memory first (GitHub, repo, Terraform, hub, spokes, the three product ApplicationSets, the journey), then compare.
  • Leaving the production checklist as a list. Open one issue per item you will do first.

Commands

git diff --stat reference/main -- . ':!products/catalog' ':!product-deployments/catalog'How your repo differs from the reference
cd infrastructure/kind && terraform destroyTear the kind fleet down

Test yourself

Answer in your head first, then open the answer.

What is the single source of truth for which clusters exist?

clusters/*.yaml, read by Terraform and the ApplicationSets.

Which layer would you split out first as the organisation grows?

Whichever ownership boundary hurts first: products into team repos, or infrastructure into its own repo.

What makes this a GitOps operating model rather than a set of commands?

Every change by any team travels the same journey: issue, pull request, checks, approval, merge, reconcile, healthy.

Where module 12 leaves you: You have a prioritised list of production work as issues in your repo, and a clean machine. The fleet is destroyed; your repo can rebuild it.