Module 10 · Operate and grow · 45 min · You are: Platform and product

Secrets and identity

GitOps puts everything in Git, except the things that must never be there. The answer is to put references in Git (which secret, from which store) and let an operator fetch the values with an identity, not a password.

Open this module in the interactive course to track progress and switch tracks.

Change journey practised: Requested → Proposed → Validated → Previewed → Approved → Merged → Reconciled → Healthy

Concepts

SecretStore per namespace

The baseline chart can render a SecretStore (how to reach the vault) in every product namespace. Off by default.

ExternalSecret with the app

The product team commits an ExternalSecret (which key, into which Kubernetes Secret). Safe to commit: it holds no values.

Workload identity

On AKS, EKS and GKE, the operator authenticates to the vault as a cloud identity bound to its ServiceAccount. No static credentials anywhere.

SSO and RBAC

Argo CD logins through your identity provider; groups map to projects, so team-checkout sees and syncs only checkout.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• charts/product-namespace/templates/secretstore.yaml (reference, your repo): The optional per-namespace secret store.
  1. Rendered only when enabled; the provider block is passed through L1–10
• infrastructure/argocd/values-prod.yaml (reference, your repo): Where SSO and RBAC go in a real hub.
  1. Disable the admin user and add SSO here L28–32
  2. Read-only by default; map SSO groups to roles L33–37
• infrastructure/cloud/lib/argocd.sh (reference, your repo): How cloud spokes are registered today, and the identity-based replacement.
  1. Today: a long-lived token per spoke L121–124
  2. The workload identity replacements for AKS, EKS and GKE L126–139

Steps

1. Platform: a SecretStore in every product namespace

The External Secrets fake provider serves values from the store definition itself: a stand-in for a vault on kind. Enable it in the baseline values of product-namespaces. In production this block is your vault provider with workload identity, usually different per environment.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c secret-stores

Add to ~/work/fleet-gitops/applicationsets/product-namespaces.yaml in the Helm values: |- block, after the size: line (same indentation)

            secretStore:
              enabled: true
              provider: { fake: { data: [ { key: db-password, value: not-a-real-secret } ] } }

Review your change. Expect only the added secretStore lines:

git diff

Run the checks. Expect them to pass:

scripts/validate.sh

git add -A && git commit -m "Secrets: a SecretStore per product namespace (fake provider on kind)"
git push -u origin HEAD

Open the pull request ↗

2. team-checkout: fetch a secret

The ExternalSecret is safe to commit: it says which key becomes which Kubernetes Secret, never the value.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c checkout-db-secret

Create ~/work/fleet-gitops/product-deployments/checkout/dev/external-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: SecretStore, name: product-secrets }
  target: { name: checkout-db }
  data:
    - secretKey: password
      remoteRef: { key: db-password }

Add to ~/work/fleet-gitops/product-deployments/checkout/dev/kustomization.yaml under resources:, after - ../base

  - external-secret.yaml

Render dev the way Argo CD will, and check your file is included. Expect kind: ExternalSecret followed by name: checkout-db:

kustomize build product-deployments/checkout/dev | grep -A3 'kind: ExternalSecret'

git add -A && git commit -m "checkout: database password from the secret store"
git push -u origin HEAD

Open the pull request and merge it. After the sync, expect two rows: the ExternalSecret with status SecretSynced, and the ordinary Secret it created:

kubectl --context kind-dev -n checkout get externalsecret,secret checkout-db

Open the pull request ↗

3. Read: SSO and per-team RBAC in Argo CD

Argo CD maps identity-provider groups to roles per project. For checkout: members of team-checkout may view and sync applications in project checkout, nothing else. The admin user is disabled once SSO works.

This is not a command to run. It is an example of the settings you would add to the Argo CD values for production, to read and adapt. Find three things: the shared admin account is off, logins go through your identity provider (oidc.config), and a group is mapped to a role that is limited to one project (checkout/*).

# infrastructure/argocd/values-prod.yaml (excerpt to adapt)
configs:
  cm:
    admin.enabled: "false"
    oidc.config: |
      name: SSO
      issuer: https://login.example.com
      clientID: argocd
      clientSecret: $oidc.clientSecret
      requestedScopes: ["openid", "profile", "email", "groups"]
  rbac:
    policy.default: role:readonly
    policy.csv: |
      p, role:checkout, applications, *, checkout/*, allow
      g, team-checkout, role:checkout
      g, platform-team, role:admin

Simplified for the workshop

The fake provider keeps the value in the SecretStore itself, so it is in Git. It exists to show the flow on kind. Never use it for real secrets.

Break it on purpose: Ask for a secret the store does not have

team-checkout needs an API token and adds an ExternalSecret for it. Nobody asked the platform team to put the token in the store.

You practise: reading an ExternalSecret's status, and who changes what when a product needs a new secret.

1. Break it

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c checkout-api-token

Create ~/work/fleet-gitops/product-deployments/checkout/dev/external-secret-api.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-api
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: SecretStore, name: product-secrets }
  target: { name: checkout-api }
  data:
    - secretKey: token
      remoteRef: { key: api-token }

Add to ~/work/fleet-gitops/product-deployments/checkout/dev/kustomization.yaml under resources:, after - external-secret.yaml

  - external-secret-api.yaml

git add -A && git commit -m "checkout: API token from the store"
git push -u origin HEAD

Open the pull request, merge it, and wait for the sync.

2. What you see

In Argo CD, checkout-dev shows the checkout-api ExternalSecret as Degraded. Expected: a Secret called checkout-api. You get: kubectl --context kind-dev -n checkout get secret checkout-api says NotFound, while checkout-db works.

3. Fix it yourself

Find the exact error. Decide which team has to make the fix, then make it as that team, so the Secret gets created.

cd ~/work/fleet-gitops && git switch main && git pull -q
git switch -c store-api-token

-               provider: { fake: { data: [ { key: db-password, value: not-a-real-secret } ] } }
+               provider: { fake: { data: [ { key: db-password, value: not-a-real-secret }, { key: api-token, value: also-not-real } ] } }

scripts/validate.sh

git add -A && git commit -m "Secrets: api-token in the store" && git push -u origin HEAD

kubectl --context kind-dev -n checkout annotate externalsecret checkout-api force-sync=$(date +%s) --overwrite

kubectl --context kind-dev -n checkout get secret checkout-api

Quiz

1. What does an ExternalSecret in Git contain?
   • A reference: which key in which store becomes which Kubernetes Secret
   • The secret value, encrypted with a key the cluster holds, so Git can store it safely
   • Nothing useful, since the real object is created by hand and never described in the repository
2. What replaces static cloud credentials for the External Secrets operator?
   • A longer, randomly generated password, stored in a ConfigMap in the operator's namespace
   • Workload identity bound to its ServiceAccount
   • A ConfigMap holding the cloud access keys, rotated by a cron job once a month
3. How does team-checkout get access to only its own Applications in Argo CD?
   • A separate Argo CD installation for the team, with its own UI and its own repositories
   • Kubernetes RBAC on the argocd namespace, giving the team access to its Application objects
   • SSO groups mapped to a role scoped to project checkout

Field note

Teams ask for sealed secrets because they look simpler. They are, until the first key rotation across twenty clusters. A vault plus External Secrets plus workload identity is the setup I have never had to apologise for.

Report a problem with this module