Module 02 · Run a workload · 40 min · You are: Platform and product

Configuration and secrets, and where the line is

The same image should run in every environment; what differs is configuration. Kubernetes has two objects for it that look alike and are treated very differently by people: everyone commits ConfigMaps, and sooner or later someone commits a Secret because "it is encoded". It is not protected in any way. This module shows what a Secret really gives you, and stops honestly at the point where plain Kubernetes has no good answer.

Open this module in the interactive course to track progress.

Concepts

ConfigMap

Key-value data in the cluster. A pod takes it as environment variables (envFrom) or as files in a volume.

Env is read once, files follow

Environment variables are set when the container starts and never change. Mounted files are updated by the kubelet within about a minute. Whether the app re-reads them is the app's business.

Versioned, immutable config

Name the ConfigMap ...-v2, set immutable: true, and change the name in the Deployment. A config change is then a rollout, and a rollback gets the old config back.

Secret

Like a ConfigMap, with values base64-encoded, its own resource name for permissions, and optional encryption at rest. base64 is an encoding, not encryption.

The gap

A Secret in Git is readable by everyone who can read the repo, for ever. A Secret created by hand is in no repo, so a rebuilt cluster does not have it. Closing that gap takes a secret store and an operator: Argo CD, module 08.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• apps/checkout/configmap.yaml (reference, your repo): checkout's settings, in their final form.
  1. The version is part of the name L6
  2. Nobody can change it in place: a change is a new name L8
  3. podinfo reads these two variables L9–11
• apps/checkout/deployment.yaml (reference, your repo): How the pod takes the ConfigMap and the Secret.
  1. A reference to a Secret by name; the value is not here L41–45
  2. Every key of the ConfigMap becomes an environment variable L46–48
  3. The same ConfigMap as files in /etc/checkout L84–87
• docs/secrets.md (reference, your repo): What must exist in the cluster that Git does not hold.
  1. One row per Secret: who owns the value and where it is kept L6–8

Steps

1. Move the banner into a ConfigMap

The banner is configuration, not part of the app. Put it in a ConfigMap, then take it into the pod twice: as environment variables and as files.

Create ~/platformengr/k8s-platform/apps/checkout/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-settings
  namespace: checkout-dev
data:
  PODINFO_UI_MESSAGE: checkout (dev)
  PODINFO_UI_COLOR: "#34577c"

In the Deployment, replace the env: block with envFrom (all keys as variables), a volume mount, and the volume. Mind the indentation: volumes: belongs to the pod, at the same depth as containers:.

Change in ~/platformengr/k8s-platform/apps/checkout/deployment.yaml

-           env:
-             - name: PODINFO_UI_MESSAGE
-               value: checkout (dev)
+           envFrom:
+             - configMapRef:
+                 name: checkout-settings
+           volumeMounts:
+             - name: settings
+               mountPath: /etc/checkout
+               readOnly: true
+       volumes:
+         - name: settings
+           configMap:
+             name: checkout-settings

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: settings in a ConfigMap" && git push -q

Apply the folder. Both files at once: the ConfigMap is created, the Deployment rolls out.

kubectl apply -f apps/checkout/

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

As an environment variable. Expect checkout (dev):

kubectl -n checkout-dev exec deployment/checkout -- printenv PODINFO_UI_MESSAGE

And as a file: every key of the ConfigMap is a file in the mounted folder. Expect the same text:

kubectl -n checkout-dev exec deployment/checkout -- cat /etc/checkout/PODINFO_UI_MESSAGE; echo

2. Create a Secret by hand

checkout needs a database password. podinfo ignores it; it stands in for a real one. This one change is not a commit, on purpose: the value must not be in your repo.

kubectl -n checkout-dev create secret generic checkout-db --from-literal=password=dev-only-s3cret

Now the Deployment refers to the Secret by name and key. Add this under the container, above envFrom::

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, above envFrom:

          env:
            - name: DB_PASSWORD         # a reference to a Secret by name; the value is not here
              valueFrom:
                secretKeyRef:
                  name: checkout-db
                  key: password

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: DB_PASSWORD from the Secret checkout-db" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

The container has the value. Expect dev-only-s3cret:

kubectl -n checkout-dev exec deployment/checkout -- printenv DB_PASSWORD

3. Read the Secret back

What does a Secret protect? Look at what the API server returns.

The stored value. Expect password: ZGV2LW9ubHktczNjcmV0:

kubectl -n checkout-dev get secret checkout-db -o yaml | grep 'password:'

That is not encrypted. Anyone who may get this Secret reads the value. Expect dev-only-s3cret:

kubectl -n checkout-dev get secret checkout-db -o jsonpath='{.data.password}' | base64 -d; echo

4. Commit a Secret on purpose, then try to take it back

The mistake every team makes once, in a safe place: nothing in this step is pushed.

Save the Secret as a file in your repo and commit it, as someone would to make a rebuild reproducible:

cd ~/platformengr/k8s-platform && kubectl -n checkout-dev get secret checkout-db -o yaml > apps/checkout/secret.yaml
git add -A && git commit -qm "checkout: db secret"

Someone notices in review. Delete it:

git rm -q apps/checkout/secret.yaml && git commit -qm "Remove secret"

The file is gone from the folder. Now ask Git for the version one commit back. Expect the base64 value, in full:

git show HEAD~1:apps/checkout/secret.yaml | grep 'password:'

Deleting a file does not delete it from history. Had this been pushed, the only fix would be a new password. Throw both commits away:

git reset -q --hard origin/main

5. See what happens when the cluster does not have it

A rebuilt cluster has everything in Git and nothing else. Simulate that for the Secret.

kubectl -n checkout-dev delete secret checkout-db

kubectl -n checkout-dev rollout restart deployment/checkout

Wait ten seconds. The old pods are still Running: a rollout does not remove a working pod until its replacement is ready. The new pod says CreateContainerConfigError:

kubectl -n checkout-dev get pods

The reason is in the pod's events: Error: secret "checkout-db" not found.

kubectl -n checkout-dev describe pod -l app.kubernetes.io/name=checkout | grep -A1 'Failed'

Put it back. The kubelet retries on its own:

kubectl -n checkout-dev create secret generic checkout-db --from-literal=password=dev-only-s3cret

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

6. Write down what Git does not hold

Whoever rebuilds this cluster needs a list. Write it now, while you know. Fill in the last paragraph in your own words.

Create ~/platformengr/k8s-platform/docs/secrets.md

# Secrets the cluster needs that Git does not hold

Every Secret below must exist in the cluster before the workload that uses it can start.
None of them is in this repo, on purpose.

| Secret | Namespace | Keys | Used by | Who owns the value | Where the value is kept |
| --- | --- | --- | --- | --- | --- |
| `checkout-db` | `checkout-dev` | `password` | Deployment `checkout`, env `DB_PASSWORD` | team-checkout | Typed in by hand with `kubectl create secret` |

(Two sentences in your own words: what has to happen before checkout can start on a rebuilt cluster,
and who does it.)

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "docs: secrets the cluster needs" && git push -q

Simplified for the course

The password is typed into a command, which also lands in your shell history. A platform team installs the External Secrets Operator: the value lives in a secret store (Vault, a cloud secret manager), Git holds an ExternalSecret that names it, and the operator builds the Secret. You build exactly that in Argo CD, module 08.

kind stores Secrets in etcd unencrypted. Production clusters turn on encryption at rest (a KMS provider), and limit who may get and list Secrets with RBAC (module 05).

Versioning ConfigMap names by hand is what Kustomize's configMapGenerator and Helm's checksum annotation automate (Argo CD, module 05).

Break it on purpose: A config change that never arrives

Black Friday: team-checkout changes the banner in the ConfigMap, commits and applies. It is a one-line change to a file that is obviously configuration.

You practise: telling a running container's view of its config from what the ConfigMap says now, and making a config change roll out from Git.

1. Break it

Change in ~/platformengr/k8s-platform/apps/checkout/configmap.yaml

-   PODINFO_UI_MESSAGE: checkout (dev)
+   PODINFO_UI_MESSAGE: Black Friday

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: Black Friday banner" && git push -q

Apply it. Expect configmap/checkout-settings configured:

kubectl apply -f apps/checkout/configmap.yaml

2. What you see

Expected: curl -s http://checkout.localtest.me:8090/api/info | grep message prints Black Friday. You get: checkout (dev), also five minutes later.

kubectl -n checkout-dev get configmap checkout-settings -o jsonpath='{.data.PODINFO_UI_MESSAGE}' says Black Friday. No pod restarted, and nothing is in an error state. Compare what the container sees in its two views of the same ConfigMap: its environment and its mounted folder.

3. Fix it yourself

Make the new banner show, with a change in Git that would also work next time, and that a git revert would undo cleanly. Running kubectl rollout restart by hand does not count: nobody will remember it next time. Done when the curl prints Black Friday and the ConfigMap in use is immutable.

-   name: checkout-settings
-   namespace: checkout-dev
- data:
+   name: checkout-settings-v2
+   namespace: checkout-dev
+ immutable: true
+ data:

-             - configMapRef:
-                 name: checkout-settings
+             - configMapRef:
+                 name: checkout-settings-v2

-           configMap:
-             name: checkout-settings
+           configMap:
+             name: checkout-settings-v2

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: settings v2, immutable" && git push -q

kubectl apply -f apps/checkout/

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

curl -s http://checkout.localtest.me:8090/api/info | grep message

kubectl -n checkout-dev delete configmap checkout-settings

Quiz

1. You change a value in a ConfigMap that a Deployment uses through envFrom. What happens to the running pods?
   • The kubelet updates the variable inside the running containers within about a minute
   • The Deployment notices the new ConfigMap and starts a rolling update by itself
   • Nothing: environment variables are set at container start, so the pods keep the old value until they are replaced
2. What protects a Secret's value?
   • Access control on the secrets resource and, if the platform turns it on, encryption at rest; the value itself is only base64-encoded
   • Encryption with a key that belongs to the namespace, so only pods in that namespace can decode it
   • A one-way hash, so the value can be mounted into a pod but never read back by a person
3. A Secret was committed and pushed, then deleted in the next commit. What has to happen?
   • Nothing more, since the file is no longer on the default branch and nobody can find it
   • The credential is changed, because the old value stays readable in the repo's history
   • The Secret is recreated in the cluster so that the committed value stops matching

Field note

I have never seen a leaked credential that came from someone being careless about security. It was always someone being careful about something else: getting a rebuild to work, making the deploy reproducible. Give people a reproducible way that is also safe, or they will pick reproducible.

Report a problem with this module