Module 05 · State, modules and environments · 35 min · You are: Platform team

Dev and prod without copy and paste

Prod must be the same code as dev with different numbers, and it must be hard to change by accident. Workspaces look like the answer: one directory, several states. But the selected workspace is invisible in the code and in a pull request, both states share one backend and one set of credentials, and both always run the same commit. Environments differ in exactly the things a workspace cannot express. A directory per environment makes each difference a line in a file somebody reviews.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

Environment directory

A root module that holds only wiring: which state (backend.tf), which target (providers.tf), which inputs (terraform.tfvars, the arguments of each module block). No logic.

Workspace

A second, named state for the same directory and the same backend. Made for short-lived copies of one configuration, such as a test of a branch. Not for dev and prod.

Blast radius

Everything one apply can change: one state. Two states mean a mistake in dev cannot touch prod, and a lock in dev does not block prod.

Promotion

A change reaches dev, is looked at, then reaches prod. Here that is one merge and two applies.

CODEOWNERS

A file that tells GitHub who must review changes under a path.

What differs between environments, and where it is written

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• envs/prod/backend.tf (reference, your repo): Prod's state.
  1. The only line that differs from dev's L4
• envs/prod/main.tf (reference, your repo): Same module, other numbers (as they end up after module 08).
  1. The source is the same module as dev's; only the inputs differ L1–6
• envs/prod/terraform.tfvars (reference, your repo): The only place that says this directory is prod.
  1. One line L1
• .github/CODEOWNERS (reference, your repo): Who must review what.
  1. envs/dev/ is left out on purpose L3–6

Steps

1. Try a workspace

See what a workspace is before deciding against it.

cd ~/platformengr/tf-platform/envs/dev

Expect * default, the only one:

terraform workspace list

Create prod and switch to it. Expect Created and switched to workspace "prod"!

terraform workspace new prod

Read the last line and one namespace name. Expect Plan: 16 to add, 0 to change, 0 to destroy. and name = "checkout-dev". A new workspace is an empty state for the same code and the same terraform.tfvars: it would try to create dev a second time.

terraform plan

Where its state would live: expect dev and a second folder, env:. After an apply, the state would be env:/prod/dev/terraform.tfstate, in the same bucket, readable with the same credentials.

docker compose -f ~/platformengr/tf-platform/state/compose.yaml exec s3 ls /data/tfstate

Go back and delete it:

terraform workspace select default && terraform workspace delete prod

2. Create the prod directory from dev's wiring

Four files of wiring, copied once: the versions, the provider, the variables and the lock file. Everything with logic stays in the module.

mkdir -p ~/platformengr/tf-platform/envs/prod && cd ~/platformengr/tf-platform/envs && cp dev/versions.tf dev/providers.tf dev/variables.tf dev/.terraform.lock.hcl prod/

Then three files of your own: which environment, which state, and which teams with which numbers.

Create ~/platformengr/tf-platform/envs/prod/terraform.tfvars

environment = "prod"

Create ~/platformengr/tf-platform/envs/prod/backend.tf

terraform {
  backend "s3" {
    bucket       = "tfstate"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1" # required by the backend; the local store ignores it
    use_lockfile = true        # a lock object next to the state, no DynamoDB

    # Only because the store is not AWS:
    endpoints                   = { s3 = "http://127.0.0.1:9000" }
    use_path_style              = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
  }
}

Create ~/platformengr/tf-platform/envs/prod/main.tf

module "checkout" {
  source = "../../modules/team-namespace"

  team        = "checkout"
  environment = var.environment
  quota       = { pods = 30, cpu = "8", memory = "16Gi" }
}

module "catalog" {
  source = "../../modules/team-namespace"

  team        = "catalog"
  environment = var.environment
  quota       = { pods = 20, cpu = "4", memory = "8Gi" }
}

3. Plan and apply prod

Its own directory, its own init, its own state.

cd ~/platformengr/tf-platform/envs/prod

terraform init

Expect Plan: 14 to add, 0 to change, 0 to destroy.: seven objects for each team.

terraform plan

Apply it (type yes):

terraform apply

Expect two rows, catalog-prod and checkout-prod:

kubectl get namespaces -l environment=prod

4. Two states, two locks

One object each in the bucket, each with its own lock.

Expect dev and prod:

docker compose -f ~/platformengr/tf-platform/state/compose.yaml exec s3 ls /data/tfstate

Now in two terminals, as in module 03. 1. In this terminal, start an apply in dev and leave it at the question. As in module 03, -replace gives it something to ask about; you will answer no.

cd ~/platformengr/tf-platform/envs/dev && terraform apply -replace=module.checkout.kubernetes_limit_range_v1.this

2. In the second terminal (load the credentials first: set -a && . ~/platformengr/tf-platform/state/.env && set +a), plan prod. Expect No changes.: dev's lock does not block prod.

cd ~/platformengr/tf-platform/envs/prod && terraform plan

3. Answer no in the first terminal.

5. One change, two applies

Add a label to the kit, in the module. Dev takes it first, then prod.

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

-     "app.kubernetes.io/managed-by" = "terraform"
+     "app.kubernetes.io/managed-by" = "terraform"
+     "platformengr.com/kit"         = "v1"

cd ~/platformengr/tf-platform/envs/dev

Expect Plan: 0 to add, 2 to change, 0 to destroy.: the two namespaces in dev. Apply it.

terraform plan

terraform apply

cd ~/platformengr/tf-platform/envs/prod

The same two changes, still waiting in prod. The code exists once; each environment takes it when its apply runs. Expect Plan: 0 to add, 2 to change, 0 to destroy., then apply.

terraform plan

terraform apply

6. Say who reviews what

GitHub asks the owners listed here to review any pull request that touches their paths.

mkdir -p ~/platformengr/tf-platform/.github

Create ~/platformengr/tf-platform/.github/CODEOWNERS

# Who must review a change under each path. In a company the owner is a team, such as @your-org/platform-team.
# envs/dev/ is left out on purpose: a product team may propose its own quota there.
/modules/    @YOUR_GITHUB_USER
/envs/prod/  @YOUR_GITHUB_USER
/state/      @YOUR_GITHUB_USER
/.github/    @YOUR_GITHUB_USER

7. Commit

Save it.

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "envs/prod, and CODEOWNERS" && git push -q)

Simplified for the course

Dev and prod are namespaces on one laptop cluster. In a company they are different clusters or accounts, and providers.tf or the tfvars of each directory points at its own. Nothing else in this layout changes.

Both directories read the module from the same path, so they can never run different module versions. With a tagged module source, prod's ref is bumped in its own pull request.

Two module blocks per environment is a little repetition, kept on purpose: each team's numbers are readable in one place. Tools such as Terragrunt or Terraform Stacks reduce it further, at the price of another layer.

Applies still run from your laptop. Where a pipeline applies, dev applies on merge and prod after an approval.

Break it on purpose: Prod wants to destroy a namespace

After a merge conflict in envs/prod/backend.tf, a colleague resolves it the quick way: take dev's file, it is nearly the same.

You practise: finding out which state a directory is really using, when every address looks right.

1. Break it

cp ~/platformengr/tf-platform/envs/dev/backend.tf ~/platformengr/tf-platform/envs/prod/backend.tf

cd ~/platformengr/tf-platform/envs/prod

terraform init -reconfigure

terraform plan

2. What you see

Expected: No changes. You get: Error: Instance cannot be destroyed twice, for module.checkout.kubernetes_namespace_v1.this and module.catalog.kubernetes_namespace_v1.this, under a plan that would also destroy kubernetes_config_map_v1.checkout_contact and kubernetes_secret_v1.checkout_db. Nobody asked to destroy anything.

terraform state list looks normal: the addresses are the ones prod should have. Look closer at one of them: terraform state show module.checkout.kubernetes_namespace_v1.this.

3. Fix it yourself

Make prod's plan say No changes. again without touching the cluster or either state. Done when terraform state show module.checkout.kubernetes_namespace_v1.this in envs/prod shows checkout-prod.

- key          = "dev/terraform.tfstate"
+ key          = "prod/terraform.tfstate"

terraform init -reconfigure

terraform plan

Quiz

1. What does terraform workspace new prod create?
   • A copy of the directory, with the prod values filled into its tfvars file
   • A new backend for prod, with its own bucket and its own credentials
   • A second, empty state for the same code, in the same backend
2. Why is a directory per environment safer than a workspace per environment?
   • Each difference (state, target, inputs, reviewers) is a line in a reviewed file
   • Directories are planned faster, because each one holds fewer resources
   • Workspaces cannot be used together with any of the remote backends
3. You change the module and apply in envs/dev only. What does terraform plan in envs/prod show?
   • No changes., because prod keeps the module at the previous commit
   • The same change, waiting: prod reads the same module and has not applied it
   • An error, because dev still holds the state lock for the module

Field note

When someone shows me a workspace per environment, I ask how a reviewer can tell from the pull request which environment a change will hit. If the answer is "from the pipeline settings", the environments are not in the code yet.

Report a problem with this module