Module 08 · Safe change · 45 min · You are: Platform team

Debugging labs: five plans that go wrong

In this module

You learn: One method for any Terraform failure: decide which of the three things is wrong (the code, the state or the real world), read the one message that names it, and never fix a state problem by changing infrastructure.
You do: Break your setup in five ways and repair each from what Terraform prints: a lock nobody holds, a provider upgrade whose plan destroys, a hand change in prod, a dependency cycle, and a for_each over a value that does not exist yet. Then write one runbook entry.
You end with: Every plan in envs/dev and envs/prod says No changes. labs/upgrade is on provider 3.5.0 with nothing recreated. Prod's checkout quota is 40 pods, in code. Every team namespace carries a platformengr.com/deployer annotation. Your repo has docs/runbook.md.

Terraform fails loudly and in few ways. Most bad outcomes come from the second step, not the first: someone forces a lock that was real, applies over drift that mattered, or deletes state to make an error go away. Each lab gives you the message and makes you decide what kind of problem it is before you touch anything. Keep docs/debugging.md open. The labs commit straight to main to stay short; in a team each fix is a pull request.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

Code, state or world: Every failure is a disagreement between two of the three. Find which two before choosing a fix.
A lock is information: The lock message says who, when and which operation. Forcing it is right only when you know that operation is dead.
The address is the identity: Type, name, module path and key. Change any part and Terraform sees a different object, unless a moved block, or removed plus import, says otherwise.
The graph has no loops: Terraform orders work by references. Two resources that refer to each other cannot be ordered.
Keys known at plan time: for_each needs to know how many instances there are, and their names, before anything is created. Values may be unknown; keys may not.

Message to move (the table in `docs/debugging.md`)

What you see	Which two disagree	Read this	Usual fix	Never
`Error acquiring the state lock`	Two operations, or a dead one	`Lock Info`: Who, Created, Operation	Wait, or `force-unlock <ID>` when it is dead	`-lock=false`
A plan with `to destroy` nobody asked for	Code and state (addresses)	The `#` line above each resource	`moved`, or `removed` + `import`	Apply and recreate
A plan with a change nobody wrote	State and world	`plan -refresh-only`	Code or apply, on purpose	Apply blind
`Error: Cycle`	Code with itself	The addresses in the message	Break one reference	`-target`
`Invalid for_each argument`	Code and what is knowable now	The expression in `for_each`	A map with static keys	`-target`
`already exists`	World and state	The address and the id	`import`	Delete the object

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

docs/debugging.md (reference, your repo): Message to move: keep it open during the labs.
1. Name the two that disagree, then pick the move L7–18
labs/upgrade/main.tf (reference, your repo): An old configuration for the platform team's own namespace (lab 2), as shipped.
1. No _v1: written for provider 2.x L1
labs/upgrade/versions.tf (reference, your repo): The provider version lab 2 starts from.
1. The last 2.x release L8

Steps

1. Lab 1: the lock nobody holds (Debug lab)

A colleague's laptop lost power in the middle of an apply. You play both: the colleague in this terminal, yourself in a second one.

cd ~/platformengr/tf-platform/envs/dev

1. In this terminal, start an apply and leave it at the question (as in module 03, -replace gives it something to ask about):

terraform apply -replace=module.checkout.kubernetes_limit_range_v1.this

2. In a second terminal (load the credentials first: set -a && . ~/platformengr/tf-platform/state/.env && set +a, then cd ~/platformengr/tf-platform/envs/dev), pull the plug on it:

pkill -9 -f "terraform apply"

3. Then, in the second terminal:

terraform plan

What you see.

Expected: No changes. You get: Error: Error acquiring the state lock with a Lock Info: block. Nobody is running anything.

Fix it yourself. Make terraform plan work again. Before you remove the lock, write down the two facts from the message that tell you it is safe.

Hint 1

Who (a user and a machine) and Created (a time) in Lock Info. Who would you ask, and is anything of theirs still running?

Hint 2

terraform force-unlock takes the ID from the message, as one argument.

One solution

The first terminal shows Killed. On a real team the first step is a message to the person named in Who.

The two facts. Who names you on this machine, and Created is the minute you started the apply that you killed. On a team, Who is the person you message first; here the first terminal shows Killed, so nothing of yours is still running.

Copy the ID from Lock Info in place of LOCK_ID, and answer yes. Expect Terraform state has been successfully unlocked!

terraform force-unlock LOCK_ID   # the ID from Lock Info, e.g. 39091cc2-978c-02b3-8536-666aba5f178c

Expect No changes.

terraform plan

The five questions, answered

What failed?: The lock outlived the process that took it.
How was it detected?: The next person's plan.
Who responds?: The platform team; the first step is to ask the person named in Who.
How is it corrected?: force-unlock with the ID, after confirming the operation is dead.
How is it prevented?: Applies from a pipeline, where a cancelled job is visible; never -lock=false as a habit.

2. Lab 2: an upgrade that wants to destroy (Debug lab)

The platform team has an old configuration for its own namespace, platform-tools, written for provider 2.x, with its state on disk (ignored by Git). Someone upgrades it and follows the deprecation warning.

mkdir -p ~/platformengr/tf-platform/labs && cp -r ~/platformengr/terraform-platform/labs/upgrade ~/platformengr/tf-platform/labs/ && cd ~/platformengr/tf-platform/labs/upgrade

terraform init

Expect Apply complete! Resources: 2 added: Namespace platform-tools and ConfigMap banner, made with provider 2.38.0.

terraform apply -auto-approve

Now the upgrade:

Change in ~/platformengr/tf-platform/labs/upgrade/versions.tf

- version = "2.38.0"
+ version = "3.5.0"

terraform init -upgrade

Expect No changes., and a Warning: Deprecated Resource that says to use kubernetes_namespace_v1.

terraform plan

Follow the warning:

Change in ~/platformengr/tf-platform/labs/upgrade/main.tf

- resource "kubernetes_namespace" "tools"
+ resource "kubernetes_namespace_v1" "tools"

Change in ~/platformengr/tf-platform/labs/upgrade/main.tf

- resource "kubernetes_config_map" "banner"
+ resource "kubernetes_config_map_v1" "banner"

Change in ~/platformengr/tf-platform/labs/upgrade/main.tf

- kubernetes_namespace.tools.metadata[0].name
+ kubernetes_namespace_v1.tools.metadata[0].name

terraform plan

What you see.

Expected: No changes.: the objects are the same. You get: Plan: 2 to add, 0 to change, 2 to destroy. Applying deletes the namespace and everything in it.

Fix it yourself. Finish the rename with 0 to destroy. Done when the plan in labs/upgrade has nothing to add, change or destroy apart from imports, you have applied it, and the next plan says No changes.

Hint 1

The type is part of the address. To Terraform, kubernetes_namespace.tools went away and kubernetes_namespace_v1.tools is new.

Hint 2

Try a moved block for each. Read the error for the one that fails: it names the resource type that cannot do this.

Hint 3

For that one: a removed block with lifecycle { destroy = false } makes Terraform forget the old address, and an import block adopts the same object under the new one. A ConfigMap's import id is namespace/name.

One solution

If your provider version refuses the moved for the namespace too (Move Resource State Not Supported), use removed plus import for both; the namespace's id is platform-tools.

Create ~/platformengr/tf-platform/labs/upgrade/refactor.tf

moved {
  from = kubernetes_namespace.tools
  to   = kubernetes_namespace_v1.tools
}

removed {
  from = kubernetes_config_map.banner

  lifecycle {
    destroy = false
  }
}

import {
  to = kubernetes_config_map_v1.banner
  id = "platform-tools/banner"
}

Find has moved to, will no longer be managed by Terraform, # kubernetes_config_map_v1.banner will be imported, and a last line with 1 to import and 0 to destroy. Apply it.

terraform plan

terraform apply

Expect No changes.

terraform plan

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "labs/upgrade on provider 3.5.0" && git push -q)

The five questions, answered

What failed?: Nothing yet; the plan would have deleted a namespace.
How was it detected?: The last line of the plan in the upgrade pull request.
Who responds?: The platform team.
How is it corrected?: Tell Terraform the objects are the same: moved where the provider supports it, removed plus import where it does not.
How is it prevented?: Upgrades are their own pull request with the plan pasted in; prevent_destroy on what must not be recreated; read the provider's upgrade guide before a major version.

3. Lab 3: someone fixed prod by hand (Debug lab)

During a sale, the on-call engineer raised checkout's quota in prod to 40 pods with kubectl. It was the right call and it was never written down. A week later you give catalog more memory.

kubectl -n checkout-prod patch resourcequota team-quota --type merge -p '{"spec":{"hard":{"pods":"40"}}}'

Change in ~/platformengr/tf-platform/envs/prod/main.tf

- memory = "8Gi"
+ memory = "12Gi"

cd ~/platformengr/tf-platform/envs/prod

terraform plan

What you see.

Expected: Plan: 0 to add, 1 to change, 0 to destroy. (catalog). You get: 0 to add, 2 to change. Your catalog change would also put checkout back to 30 pods, in the middle of whatever needed 40.

Fix it yourself. Ship the catalog change without changing checkout's real quota. Done when the plan changes only module.catalog.kubernetes_resource_quota_v1.this; apply it.

Hint 1

See the hand change on its own: terraform plan -refresh-only.

Hint 2

The value in the cluster is the right one. Which file should say 40?

One solution

Change in ~/platformengr/tf-platform/envs/prod/main.tf

- quota       = { pods = 30, cpu = "8", memory = "16Gi" }
+ quota       = { pods = 40, cpu = "8", memory = "16Gi" }

Expect Plan: 0 to add, 1 to change, 0 to destroy., for module.catalog.kubernetes_resource_quota_v1.this only. Apply it.

terraform plan

terraform apply

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "prod: checkout 40 pods (adopted from the incident), catalog 12Gi" && git push -q)

The five questions, answered

What failed?: The code no longer described prod.
How was it detected?: An unrelated plan, a week late.
Who responds?: The platform team owns envs/prod; the on-call engineer owns telling them.
How is it corrected?: Adopt the value in code.
How is it prevented?: An incident checklist line "what did we change by hand"; a scheduled refresh-only plan; fewer people with write access to prod outside the pipeline.

4. Lab 4: a cycle (Debug lab)

Someone wants each namespace to name its deployer account in an annotation, and writes it as a reference, as module 01 taught.

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf
In the namespace's metadata.

-     labels = local.labels
+     labels = local.labels
+     annotations = {
+       "platformengr.com/deployer" = kubernetes_service_account_v1.deployer.metadata[0].name
+     }

cd ~/platformengr/tf-platform/envs/dev

terraform plan

What you see.

You get: Error: Cycle: followed by addresses, among them module.checkout.kubernetes_service_account_v1.deployer and module.checkout.kubernetes_namespace_v1.this. No plan.

Fix it yourself. Keep the annotation and get a plan. Done when terraform plan in dev shows one in-place change per namespace and no error. Apply in dev, then prod.

Hint 1

The ServiceAccount needs the namespace's name. What does the namespace now need?

Hint 2

The account's name is not something the cluster decides. Put it in a local and let both use it.

One solution

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

-   namespace = coalesce(var.name_override, "${var.team}-${var.environment}")
+   namespace     = coalesce(var.name_override, "${var.team}-${var.environment}")
+   deployer_name = "deployer"

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

- "platformengr.com/deployer" = kubernetes_service_account_v1.deployer.metadata[0].name
+ "platformengr.com/deployer" = local.deployer_name

Change in ~/platformengr/tf-platform/modules/team-namespace/rbac.tf

- resource "kubernetes_service_account_v1" "deployer" {
-   metadata {
-     name      = "deployer"
+ resource "kubernetes_service_account_v1" "deployer" {
+   metadata {
+     name      = local.deployer_name

Expect Plan: 0 to add, 4 to change, 0 to destroy.: one annotation per namespace in dev. Apply it, then prod (2 to change).

terraform plan

terraform apply

cd ~/platformengr/tf-platform/envs/prod

terraform apply

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Annotate each namespace with its deployer" && git push -q)

The five questions, answered

What failed?: The configuration could not be ordered.
How was it detected?: Terraform, before planning. In a pull request, CI's validate job reports the cycle too.
Who responds?: The author of the module change.
How is it corrected?: Replace one reference with the value both sides already know.
How is it prevented?: Reference what a resource computes; share what you decide through locals.

5. Lab 5: for_each over something that does not exist yet (Debug lab)

team-catalog brings in contractors for a month. They get read access through a group whose name is generated, next to the team leads' group.

Create ~/platformengr/tf-platform/envs/dev/contractors.tf

resource "random_pet" "catalog_contractors" {
  length = 2
  prefix = "contractors"
}

resource "kubernetes_role_binding_v1" "catalog_extra_view" {
  for_each = toset(["team-catalog-leads", random_pet.catalog_contractors.id])

  metadata {
    name      = "view-${each.key}"
    namespace = module.catalog.namespace
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "view"
  }
  subject {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Group"
    name      = each.value
  }
}

cd ~/platformengr/tf-platform/envs/dev

terraform plan

What you see.

You get: Error: Invalid for_each argument. The message says the set includes values that cannot be determined until apply, and suggests -target. Do not use -target.

Fix it yourself. Get both bindings in one normal apply. Done when terraform plan shows 3 to add and kubectl get rolebinding -n catalog-dev lists two new bindings after apply.

Hint 1

Terraform must know the instance keys now. Which element of the set is unknown until the pet is created?

Hint 2

Use a map: keys you write yourself, values that may be unknown. { leads = "team-catalog-leads", contractors = random_pet.catalog_contractors.id }.

One solution

Change in ~/platformengr/tf-platform/envs/dev/contractors.tf

-   for_each = toset(["team-catalog-leads", random_pet.catalog_contractors.id])
+   for_each = {
+     leads       = "team-catalog-leads"
+     contractors = random_pet.catalog_contractors.id
+   }

The rest stays: each.key is now leads or contractors, each.value the group. Expect Plan: 3 to add, 0 to change, 0 to destroy. Apply it.

terraform plan

terraform apply

Expect view-contractors and view-leads next to team-view and deployer:

kubectl get rolebinding -n catalog-dev

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "catalog: contractors read catalog-dev" && git push -q)

The five questions, answered

What failed?: The plan; nothing was created.
How was it detected?: Terraform, at plan time.
Who responds?: The author.
How is it corrected?: Static keys, computed values.
How is it prevented?: Never derive for_each keys from attributes a provider computes; a two-step apply with -target hides the problem for the next person.

6. Write your first runbook entry

Pick one lab and write it up in your own words, for the colleague who meets the same message next month.

Done when:

At least one entry with the exact message
Which two of code, state and world disagreed
The fix you used
The move that would have made it worse

mkdir -p ~/platformengr/tf-platform/docs

Create ~/platformengr/tf-platform/docs/runbook.md
Fill in the four parts.

# Runbook

One entry per failure we have seen. Newest first.

## <the exact message>

- **Which two disagreed:** code and state / state and world / code and world
- **What it was:**
- **The fix:**
- **What would have made it worse:**

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Runbook: first entry" && git push -q)

Simplified for the course

Each lab has one cause. Real failures stack: a stale lock on top of drift on top of an upgrade. The method is the same; do one at a time and plan again between.
labs/upgrade uses a local state so the lab stays out of your environments. Its state file is ignored by Git like any other.

Quiz

Lock Info shows an apply started 30 seconds ago by a colleague. What do you do?
- Wait, or ask them: the lock is doing its job
- Run force-unlock with the ID so your plan can go ahead
- Add -lock=false to your command, just for this one plan
You rename a resource's type from kubernetes_config_map to kubernetes_config_map_v1 and the plan shows one to add and one to destroy. Why?
- The new provider version stores ConfigMaps differently and must recreate them
- The type is part of the address, so Terraform sees a removed object and a new one
- The ConfigMap drifted in the cluster while the provider was being upgraded
Why does for_each = toset([random_pet.x.id]) fail on the first plan?
- toset cannot hold strings that were produced by a provider
- random_pet cannot be used in the same configuration as for_each
- The keys of the instances would depend on a value that exists only after apply

Answers

Wait, or ask them: the lock is doing its job Forcing a live lock lets two writers corrupt the state.
The type is part of the address, so Terraform sees a removed object and a new one Identity is the address; tell Terraform the two are the same object.
The keys of the instances would depend on a value that exists only after apply Terraform must know every instance address while planning.

Field note

When a plan surprises me I stop changing things and name the disagreement out loud: code and state, state and world, or code and world. Most bad afternoons I have seen started with a fix chosen before that sentence was finished.

Report a problem with this module