Module 08 · Real-world delivery · 55 min · You are: Platform and product

Private repositories and secrets

In this module

You learn: How Argo CD gets into a private repository (a read-only deploy key, stored as a Secret), and how to keep passwords out of Git: you commit an encrypted SealedSecret that only the controller in your cluster can decrypt.
You do: Install Sealed Secrets. Make checkout-config private and see Argo CD lose access. Create a deploy key, seal it, and switch the Applications to SSH. Then seal a database password and watch the pod receive it.
You end with: checkout-config is private and Argo CD reads it over SSH with a sealed deploy key. Each environment has a checkout-db Secret created from a SealedSecret, and the checkout pod gets DB_PASSWORD. Git holds only ciphertext.

Real environment repos are private, and real apps need passwords. Both are credentials, and GitOps wants everything in Git, except credentials. The pattern: commit something that is useless outside the cluster (an encrypted secret), and let a controller in the cluster turn it back into a Secret.

Open this module in the interactive course to track progress and switch tracks.

GitOps loop practised: Committed → Reviewed → Detected → Rendered → Compared → Synced → Healthy → Observed

Concepts

Repository credentials: A Secret in argocd labelled argocd.argoproj.io/secret-type: repository, with url, type and a key or token. A repo-creds Secret is a template for every repo under a URL prefix.
Deploy keys: An SSH key that can read one repository, nothing else, and belongs to no person. The narrowest credential GitHub offers.
Sealed Secrets: kubeseal encrypts a Secret with the cluster's public key. The SealedSecret is safe to commit; only the controller in this cluster can decrypt it.
Scope: A SealedSecret is bound to its name and namespace by default: copying it elsewhere does not work. That is a feature.

Secrets with GitOps: the options

Approach	Git holds	Fits when
Sealed Secrets (this course)	Ciphertext for one cluster	One or few clusters; back up the sealing key
SOPS (with a plugin)	Ciphertext, keys in a KMS	You already use a cloud KMS
External Secrets + a vault	A reference: which key, from which store	Many clusters, rotation, audits: what Fleet GitOps uses
Argo CD Vault Plugin	Placeholders filled at render time	Avoid: secrets end up in the repo-server cache

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

platform/apps/sealed-secrets.yaml (reference): The controller, as a Helm add-on.
1. The name kubeseal expects L18
2. The controller holds the only decryption key: never auto-delete it L24
platform/apps/repo-credentials.yaml (reference): Syncs the sealed repository credentials into argocd.
1. One SealedSecret per private repo L13
platform/secrets/README.md (reference): What goes in secrets/, and what never does.
1. The rule L9
platform/apps/checkout-dev.yaml (reference): The third source: sealed secrets for dev.
1. Same repo, a plain directory of SealedSecrets L29
2. SSH URL: matched by the deploy key credential L24

Steps

1. Platform: install Sealed Secrets and fetch the public key

The certificate is public: anyone may seal, only the cluster can unseal.

cd ~/work/argocd-platform && git switch -q main && git pull -q
cp ~/work/argocd-gitops/platform/apps/sealed-secrets.yaml apps/

Change in ~/work/argocd-platform/apps/sealed-secrets.yaml

-   project: platform
+   project: default

git add -A && git commit -qm "Add-on: Sealed Secrets" && git push -q

Wait until the controller is running:

argocd app get root --refresh >/dev/null
argocd app wait sealed-secrets --health --timeout 300

Fetch the controller's public certificate. kubeseal uses it to encrypt. Anyone may have it: it can only lock, not unlock.

kubeseal --controller-name sealed-secrets-controller --controller-namespace kube-system --fetch-cert > ~/work/sealing-cert.pem

Check that you got a certificate. Expect the first line to be -----BEGIN CERTIFICATE-----:

head -2 ~/work/sealing-cert.pem

2. team-checkout: make the environment repo private, and watch Argo CD lose it

Until now Argo CD read the repo without a password because it was public. Make it private, and Argo CD has no way in: this is what an expired or deleted credential looks like in real life.

gh repo edit YOUR_GITHUB_USER/checkout-config --visibility private --accept-visibility-change-consequences

Now make Argo CD read the repo again, ignoring its cache. Expect an authentication error line and Sync Status: Unknown: Argo CD cannot read the source, so it cannot compare.

argocd app get checkout-dev --hard-refresh | grep -iE 'ComparisonError|authentication|Sync Status'

3. team-checkout: add a read-only deploy key

Generate a key for Argo CD, register the public half on the repo (read-only is the default). Hand the private half to the platform team; here, keep it out of every repo.

Create a key pair: a private half (stays with you) and a public half (goes to GitHub). -N '' means no passphrase, because Argo CD has to use it unattended.

mkdir -p ~/work/keys && ssh-keygen -q -t ed25519 -N '' -C argocd-checkout-config -f ~/work/keys/checkout-config

Register the public half on the repo. Now anyone holding the private half can read this one repo, and nothing else.

gh repo deploy-key add ~/work/keys/checkout-config.pub -R YOUR_GITHUB_USER/checkout-config --title argocd

Check that GitHub has it. Expect one row titled argocd:

gh repo deploy-key list -R YOUR_GITHUB_USER/checkout-config

4. Platform: seal the repository credential and switch to SSH

Build the Secret locally (never applied, never saved in plain form) and let kubeseal write the sealed file: a generated file, not one you edit. Then point the checkout Applications at the SSH URL the credential matches.

cd ~/work/argocd-platform && git switch -q main && git pull -q
mkdir -p secrets && cp ~/work/argocd-gitops/platform/secrets/README.md secrets/

Seal the credential. Read this as three steps joined by pipes: build a Secret holding the repo URL and the private key (--dry-run means nothing is created in the cluster), label it so Argo CD recognises it as a repository credential, then encrypt it with kubeseal into a file you can commit.

kubectl create secret generic repo-checkout-config -n argocd --dry-run=client -o yaml \
    --from-literal=type=git --from-literal=url=[email protected]:YOUR_GITHUB_USER/checkout-config.git \
    --from-file=sshPrivateKey=~/work/keys/checkout-config \
  | kubectl label --local -f - argocd.argoproj.io/secret-type=repository -o yaml \
  | kubeseal --cert ~/work/sealing-cert.pem -o yaml > secrets/repo-checkout-config.yaml

Check that the file is safe to commit. Expect kind: SealedSecret and a long unreadable sshPrivateKey value, not the key itself:

grep -E 'kind|sshPrivateKey' secrets/repo-checkout-config.yaml

Add the Application that syncs the secrets/ folder into the cluster:

cp ~/work/argocd-gitops/platform/apps/repo-credentials.yaml apps/ && perl -pi -e 's/YOUR_GITHUB_USER/YOUR_GITHUB_USER/g' apps/repo-credentials.yaml

Change in ~/work/argocd-platform/apps/repo-credentials.yaml

-   project: platform
+   project: default

Change in ~/work/argocd-platform/apps/checkout-dev.yaml, ~/work/argocd-platform/apps/checkout-prod.yaml
In both files: the HTTPS URL of checkout-config becomes the SSH URL.

-     - repoURL: https://github.com/YOUR_GITHUB_USER/checkout-config.git
+     - repoURL: [email protected]:YOUR_GITHUB_USER/checkout-config.git

Run the check. It also confirms no plain Secret is about to be committed:

scripts/validate.sh

git add -A && git commit -qm "Sealed deploy key for checkout-config" && git push -q

argocd app get root --refresh >/dev/null; sleep 30

Argo CD should now know the private repo. Expect a row for your checkout-config SSH URL with status Successful:

argocd repo list

And the apps work again:

argocd app wait checkout-dev checkout-prod --sync --health --timeout 180

5. Platform: give the checkout apps a secrets source

A third source per Application: the environment repo's secrets/<env> folder. Empty for now; that is fine.

cd ~/work/argocd-platform && git switch -q main && git pull -q

Add to ~/work/argocd-platform/apps/checkout-dev.yaml under sources:, after the ref: config line

    - repoURL: [email protected]:YOUR_GITHUB_USER/checkout-config.git
      targetRevision: main
      path: secrets/dev

Add to ~/work/argocd-platform/apps/checkout-prod.yaml the same place

    - repoURL: [email protected]:YOUR_GITHUB_USER/checkout-config.git
      targetRevision: main
      path: secrets/prod

Expect only added lines, one new source in each file:

git diff

git add -A && git commit -qm "checkout: sealed secrets source" && git push -q

6. team-checkout: seal a database password and use it

Seal one Secret per environment, reference it from the values, ship it. The pod gets DB_PASSWORD; Git only ever saw ciphertext.

cd ~/work/checkout-config && git switch -q main && git pull -q

Seal one password per environment. For dev and prod: generate a random password, put it in a Secret named checkout-db for that namespace, and encrypt it with kubeseal. You never see or store the plain password.

for env in dev prod; do
  mkdir -p secrets/$env
  kubectl create secret generic checkout-db -n checkout-$env --dry-run=client -o yaml \
      --from-literal=password="$(openssl rand -hex 16)" \
    | kubeseal --cert ~/work/sealing-cert.pem -o yaml > secrets/$env/checkout-db.yaml
done

Add to ~/work/checkout-config/helm/dev.yaml, ~/work/checkout-config/helm/prod.yaml at the end of both files

database:
  secretName: checkout-db      # created by secrets/<env>/checkout-db.yaml

git add -A && git commit -qm "checkout: database password, sealed" && git push -q

argocd app get checkout-dev --refresh >/dev/null
argocd app wait checkout-dev --sync --health --timeout 180

Expect two rows: the SealedSecret from Git, and the ordinary Secret the controller made from it:

kubectl -n checkout-dev get sealedsecret,secret checkout-db

Expect six characters of the password. The pod has it; cut hides the rest:

kubectl -n checkout-dev exec deploy/checkout -- printenv DB_PASSWORD | cut -c1-6

Simplified for the course

You are both teams, so the private key passes from one folder to another. In a real handover, team-checkout never sees the sealed credential and the platform team never sees the app password: each seals its own.
If the cluster is rebuilt, its sealing key is new and every SealedSecret must be resealed. Back up the key (kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml) into your vault, or move to External Secrets, which Fleet GitOps uses.

Break it on purpose: Copy a sealed secret

prod needs a second password, for a reporting job. Instead of sealing a new one, someone copies dev's sealed file into prod and renames it.

You practise: reading a SealedSecret's status, and why a sealed secret only works under the name and namespace it was sealed for.

1. Break it

cd ~/work/checkout-config && git switch -q main && git pull -q
cp secrets/dev/checkout-db.yaml secrets/prod/checkout-reporting.yaml

Change in ~/work/checkout-config/secrets/prod/checkout-reporting.yaml
Everywhere it appears (under metadata: and under template:).

-   name: checkout-db
-   namespace: checkout-dev
+   name: checkout-reporting
+   namespace: checkout-prod

git add -A && git commit -qm "checkout: reporting password for prod" && git push -q

argocd app get checkout-prod --refresh >/dev/null; sleep 20

2. What you see

In the UI, checkout-prod shows the checkout-reporting SealedSecret as Degraded. Expected: a Secret called checkout-reporting, like checkout-db. You get: kubectl -n checkout-prod get secret checkout-reporting says NotFound.

3. Fix it yourself

Find the exact error, then give prod a working checkout-reporting Secret. Its value does not matter.

Hint 1

The SealedSecret's status says why: click it in the UI, or kubectl -n checkout-prod describe sealedsecret checkout-reporting and read Conditions and Events.

Hint 2

The ciphertext is bound to the name and namespace it was sealed for. Seal a new Secret for checkout-reporting in checkout-prod, the way you sealed checkout-db.

The fix

cd ~/work/checkout-config
kubectl create secret generic checkout-reporting -n checkout-prod --dry-run=client -o yaml \
    --from-literal=password="$(openssl rand -hex 16)" \
  | kubeseal --cert ~/work/sealing-cert.pem -o yaml > secrets/prod/checkout-reporting.yaml

git add -A && git commit -qm "checkout: seal the reporting password for prod" && git push -q

argocd app get checkout-prod --refresh >/dev/null; sleep 20

Expect the Secret to exist now:

kubectl -n checkout-prod get secret checkout-reporting

After you fixed it: the five questions

What failed?: A SealedSecret could not be decrypted: it was sealed for a different name and namespace.
How was it detected?: The SealedSecret's status ("no key could decrypt secret"), and Argo CD showed it Degraded.
Who responds?: team-checkout.
How is it corrected?: Seal a new Secret for the target name and namespace.
How is it prevented?: Strict scope, the default, makes copies fail loudly instead of quietly reusing a secret somewhere it was not meant for.

Quiz

What makes a Secret a repository credential for Argo CD?
- The label argocd.argoproj.io/secret-type: repository, plus url and credentials
- Its name starts with repo-, which Argo CD scans for in every namespace on the cluster
- It lives in kube-system, where Argo CD looks for credentials before any other namespace
Why a deploy key instead of a personal access token?
- Deploy keys are faster than tokens because SSH skips the HTTPS handshake on every fetch
- It reads one repository only and belongs to no person
- Argo CD cannot use personal access tokens at all, so a key is the only option it supports
Someone gets your SealedSecret from GitHub. What can they do?
- Run kubeseal against it and read the password, since the file holds the secret in encoded form
- Apply it to their own cluster and read it there, as any controller can decrypt a SealedSecret
- Nothing useful: only the controller in your cluster holds the private key

Answers

The label argocd.argoproj.io/secret-type: repository, plus url and credentials Argo CD discovers credentials by label; repo-creds does the same for a URL prefix.
It reads one repository only and belongs to no person Least privilege, and no outage when someone leaves the company.
Nothing useful: only the controller in your cluster holds the private key The ciphertext is useless without the cluster's private key, and is bound to one name and namespace.

Field note

Sealed Secrets is fine for one cluster and a few teams. The day you rebuild a cluster and discover nobody backed up the sealing key, you will reseal everything by hand. If you keep it, put that backup in the runbook today.

Report a problem with this module