Module 03 · State, modules and environments · 45 min · You are: Platform team

State: a shared backend, a lock, and the secrets inside

The state file is the only link between your code and the real objects. On one laptop, a colleague cannot run a plan, and a lost disk means Terraform no longer knows what it owns. In Git, two people apply from two copies and the second overwrites the first, and every secret Terraform ever handled is in the history for good. A backend with a lock solves the first two. The third needs a decision per secret: who creates it, and does Terraform have to remember its value?

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

State

A JSON document that maps each address to a real object and stores all of its attributes, including ones marked sensitive.

Backend

Where state is kept. The default is a local file. A remote backend keeps it in a shared place; the code stays in Git, the state does not.

State lock

While a plan or apply runs, Terraform holds a lock so nobody else can write. With use_lockfile = true the S3 backend writes an object next to the state, ending in .tflock.

sensitive hides, it does not protect

A sensitive value is shown as (sensitive value) in the plan and in outputs. It is still written to state in plain text.

Ephemeral and write-only

An ephemeral value exists during one run and is never stored. A write-only argument (data_wo) passes it to the API and forgets it.

Where state can live

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• state/compose.yaml (reference, your repo): The state store: an S3-compatible server in Docker.
  1. The bucket is a folder, created before the server starts L8
  2. Credentials come from state/.env, never from this file L10–11
  3. Reachable from your own machine only L12–13
• envs/dev/backend.tf (reference, your repo): Where dev's state lives.
  1. One object per environment L4
  2. The lock: an object next to the state L6
  3. Only because this store is not AWS L9–14
• envs/dev/checkout.tf (reference, your repo): team-checkout's extras in dev, as they end up in this module.
  1. Made during the run, never stored L12–15
  2. Sent to the cluster and forgotten. Raise the revision to rotate L22–25
• docs/secrets.md (reference, your repo): What state holds, and what this repo keeps out of it.
  1. Read access to state is read access to every secret in it L5–9

Steps

1. Let Terraform make a secret

Give checkout a database password, made by Terraform and stored in a Kubernetes Secret. First a second provider, random, then a file for checkout's extras.

Change in ~/platformengr/tf-platform/envs/dev/versions.tf

-       version = "3.4.0"
-     }
+       version = "3.4.0"
+     }
+     random = {
+       source  = "hashicorp/random"
+       version = "3.9.1"
+     }

Expect - Installing hashicorp/random v3.9.1.... The lock file gains a second provider.

cd ~/platformengr/tf-platform/envs/dev

terraform init

Move the ConfigMap out of main.tf into a new file. An address does not depend on the file a block is in, so this changes nothing.

Delete from ~/platformengr/tf-platform/envs/dev/main.tf

- 
- resource "kubernetes_config_map_v1" "checkout_contact" {
-   metadata {
-     name      = "team-contact"
-     namespace = kubernetes_namespace_v1.checkout.metadata[0].name
-   }
-   data = {
-     oncall = "#checkout-oncall"
-   }
- }

Create ~/platformengr/tf-platform/envs/dev/checkout.tf

resource "kubernetes_config_map_v1" "checkout_contact" {
  metadata {
    name      = "team-contact"
    namespace = kubernetes_namespace_v1.checkout.metadata[0].name
  }
  data = {
    oncall = "#checkout-oncall"
  }
}

resource "random_password" "checkout_db" {
  length  = 24
  special = false
}

resource "kubernetes_secret_v1" "checkout_db" {
  metadata {
    name      = "checkout-db"
    namespace = kubernetes_namespace_v1.checkout.metadata[0].name
  }
  data = {
    password = random_password.checkout_db.result
  }
}

Apply. Expect Plan: 2 to add, 0 to change, 0 to destroy., and in the plan data = (sensitive value) on the Secret. Type yes.

terraform apply

2. Find it in the state

The plan hid the value. The state does not.

state show hides it too. Expect result = (sensitive value):

terraform state show random_password.checkout_db

Now read the raw state. Expect a line "result": "..." with the password, in plain text:

terraform state pull | grep '"result"'

Anyone who can read this file can read every secret Terraform has ever created or been given.

3. Start the state store

An S3-compatible server in Docker, on your laptop. It stands in for a cloud bucket: the backend cannot tell the difference.

mkdir -p ~/platformengr/tf-platform/state && cp ~/platformengr/terraform-platform/state/compose.yaml ~/platformengr/tf-platform/state/

The store's credentials. Replace change-me-please with a secret key of your own, 12 characters or more, no spaces. This file is in .gitignore.

Create ~/platformengr/tf-platform/state/.env

AWS_ACCESS_KEY_ID=tfstate
AWS_SECRET_ACCESS_KEY=change-me-please

Load the credentials into this shell, first. The backend reads AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment, and Docker Compose prefers what is already in the shell over the file, so a real AWS key in this terminal would otherwise become the store's key. Every new terminal needs this line before terraform commands in this course; keep real AWS work in another terminal.

set -a && . ~/platformengr/tf-platform/state/.env && set +a

Start it. Expect Container tfstate-s3-1 Started.

docker compose --env-file ~/platformengr/tf-platform/state/.env -f ~/platformengr/tf-platform/state/compose.yaml up -d

Expect one row with STATUS Up:

docker compose -f ~/platformengr/tf-platform/state/compose.yaml ps

4. Declare the backend

Tell Terraform where state lives from now on.

Create ~/platformengr/tf-platform/envs/dev/backend.tf

terraform {
  backend "s3" {
    bucket       = "tfstate"
    key          = "dev/terraform.tfstate"
    region       = "us-east-1" # required by the backend; the local store ignores it
    use_lockfile = true        # a lock object next to the state, no DynamoDB

    # Only because the store is not AWS:
    endpoints                   = { s3 = "http://127.0.0.1:9000" }
    use_path_style              = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
  }
}

Changing where state lives is an init job. -migrate-state copies the existing state across. Run it, and answer yes to Do you want to copy existing state to the new backend? Expect Successfully configured the backend "s3"!

terraform init -migrate-state

5. Check that it moved, then remove the local copy

Terraform now reads the state from the bucket.

Expect the same six addresses as before, now read from the bucket:

terraform state list

Look inside the store. Expect terraform.tfstate:

docker compose -f ~/platformengr/tf-platform/state/compose.yaml exec s3 ls /data/tfstate/dev

The local files are now only a stale copy. Remove them:

rm terraform.tfstate terraform.tfstate.backup

Expect No changes.:

terraform plan

6. Two people at once

Watch the lock work. You need two terminals.

1. In this terminal, start an apply and do not answer the question. Nothing is waiting to change, so a plain apply would finish at once; -replace makes it propose recreating the limit range, which gives it a question to wait at. While it waits, it holds the lock. You will answer no: nothing is replaced.

terraform apply -replace=kubernetes_limit_range_v1.checkout

2. Open a second terminal. Go to the same directory, load the credentials, and plan:

cd ~/platformengr/tf-platform/envs/dev && set -a && . ~/platformengr/tf-platform/state/.env && set +a

terraform plan

3. Expect Error: Error acquiring the state lock, a line with StatusCode: 412 and PreconditionFailed, and a Lock Info: block. Find ID, Operation: OperationTypeApply, Who (you, on this machine) and Created.

4. Still in the second terminal: the lock is an object next to the state. Expect terraform.tfstate and terraform.tfstate.tflock:

docker compose -f ~/platformengr/tf-platform/state/compose.yaml exec s3 ls /data/tfstate/dev

5. Back in the first terminal, answer no. Run the ls again: the lock object is gone.

7. Take the password out of the state

Terraform only has to hand the password to the cluster. It does not have to remember it. An ephemeral resource makes the value during the run; the Secret's write-only argument sends it and forgets it.

Change in ~/platformengr/tf-platform/envs/dev/checkout.tf

- resource "random_password" "checkout_db" {
+ ephemeral "random_password" "checkout_db" {

Change in ~/platformengr/tf-platform/envs/dev/checkout.tf

-   data = {
-     password = random_password.checkout_db.result
-   }
+   data_wo = {
+     password = ephemeral.random_password.checkout_db.result
+   }
+   data_wo_revision = 1

Apply. Expect random_password.checkout_db will be destroyed (it leaves the state; nothing in the cluster is deleted), kubernetes_secret_v1.checkout_db will be updated in-place, and Plan: 0 to add, 1 to change, 1 to destroy. Type yes.

terraform apply

Expect 0: no password in the state any more. (grep -c exits with an error code when it counts 0; here that is the good result.)

terraform state pull | grep -c '"result"'

The cluster has it. Expect 24 characters, and a different password from the one you found in the state: the ephemeral resource made a new one and data_wo wrote it. In a real system this first apply is a rotation, so whatever reads the password must pick up the new value.

kubectl get secret checkout-db -n checkout-dev -o jsonpath='{.data.password}' | base64 -d; echo

data_wo_revision is how you rotate: raise it, and the next apply writes a new password.

8. Commit

The backend and the store's compose file go to Git. The credentials and the state do not.

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Remote state with a lock; checkout-db without its value in state" && git push -q)

Expect 0: no state file and no .env is tracked.

git ls-files | grep -c -E 'tfstate|\.env$'

Simplified for the course

The state store runs on your laptop with one user that can do everything. A real bucket has versioning (so a bad write can be rolled back), encryption, and access for the pipeline and a few people only. docs/cloud.md in the reference repo shows it.

OpenTofu can also encrypt state and plan files itself (the encryption block). Terraform leaves that to the backend.

Terraform is rarely the right owner of an application's password. The usual design is a secret store that the cluster reads from: Argo CD module 08. You did it here to see what state does with a secret.

The store is the Versity S3 Gateway, pinned by digest. Any S3-compatible server that supports conditional writes (what use_lockfile uses) does the same job.

Break it on purpose: Terraform has forgotten what it owns

A colleague tidies up. The state key dev/terraform.tfstate does not match the naming they prefer, so they change it in backend.tf and run the init variant that does not ask questions.

You practise: telling "the infrastructure is gone" from "Terraform is looking in the wrong place", before anyone applies.

1. Break it

Change in ~/platformengr/tf-platform/envs/dev/backend.tf

- key          = "dev/terraform.tfstate"
+ key          = "checkout/dev.tfstate"

terraform init -reconfigure

terraform plan

2. What you see

Expected: No changes. You get: Plan: 5 to add, 0 to change, 0 to destroy. Every object you made is listed as new, although kubectl get namespace checkout-dev shows it is there. If you applied this, it would fail with namespaces "checkout-dev" already exists.

Look at what Terraform thinks it manages: terraform state list.

3. Fix it yourself

Get back to No changes. without applying, importing or deleting anything in the cluster. Done when terraform state list shows five addresses and the plan is empty. Then check what the detour left in the bucket.

- key          = "checkout/dev.tfstate"
+ key          = "dev/terraform.tfstate"

terraform init -reconfigure

terraform state list

terraform plan

docker compose -f ~/platformengr/tf-platform/state/compose.yaml exec s3 ls -R /data/tfstate

Quiz

1. An output is marked sensitive = true. Where can its value still be read?
   • Nowhere: Terraform encrypts sensitive values with a key from the provider
   • Only in a saved plan file, and only until that plan has been applied
   • In the state, in plain text, by anyone who can read the backend
2. Why not keep terraform.tfstate in Git, next to the code it belongs to?
   • Git has no lock, so two applies overwrite each other, and every secret in it stays in history
   • Git cannot store JSON files of the size that a real state file usually reaches
   • Terraform refuses to run when it finds its own state inside a Git repository
3. You changed the backend's key and ran init -reconfigure. The plan now wants to add everything. What happened?
   • The backend deleted the old objects in the cluster when the key changed
   • Terraform is reading a new, empty state; the real state is still under the old key
   • The lock file was removed, so Terraform lost track of its provider

Field note

I treat read access to a state bucket as read access to every secret behind it. When someone asks who can see the production database password, the honest answer starts with who can read the state.

Report a problem with this module