Module 07 · Safe change · 45 min · You are: Platform team

Import, drift and provider upgrades

No platform starts empty. Teams made things by hand before you arrived, someone will patch a quota during an incident, and the provider you pinned gets a new release every few weeks. Each of these breaks the equation "code equals state equals reality" in a different place, and each has a wrong fix that looks fine: recreate it, apply over it, or stay on the old version for ever. You practise the right one for each, and you find the edge of what Terraform can see.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

import block

import { to = <address> id = "<the provider's id>" }. On the next apply Terraform records the existing object under that address, then changes it to match the code.

Drift

The real object no longer matches the state. Terraform finds it only when someone runs a plan.

Refresh-only plan

terraform plan -refresh-only reads every object and shows only what changed outside Terraform. It proposes updating the state, never the objects.

Unmanaged is invisible

Terraform compares only what is in its state. A RoleBinding someone adds by hand is in no plan.

Upgrade = constraint + lock + a plan you understand

Change the version, let init -upgrade rewrite the lock file, read the changelog, and read the plan. Every line must be one you can explain, with 0 to destroy; after the apply, the plan is No changes.

Out of step: which case is it?

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• envs/dev/imports.tf (reference, your repo): Namespaces that existed before Terraform managed them.
  1. The address in your code, and the name the cluster knows it by L2–5
• envs/dev/.terraform.lock.hcl (reference, your repo): The exact provider versions and their checksums. Written by init; committed.
  1. h1: lines are per platform; zh: lines are the release archives L4–7
• .github/dependabot.yml (reference, your repo): Proposes provider upgrades as pull requests.
  1. Both environment directories, once a week L5–8
• docs/debugging.md (reference, your repo): Message to move: the table for the debugging labs.
  1. Drift: find it with a refresh-only plan, then decide L11

Steps

1. Make what team-search made a year ago

Before there was a platform team, team-search made its namespace by hand and put something in it.

kubectl create namespace search-dev

kubectl label namespace search-dev owner=search

kubectl create configmap search-notes -n search-dev --from-literal=since=2025

2. Onboard search the normal way, and see the plan be wrong

A module "search" with the default quota, and search in the capacity check.

Add to ~/platformengr/tf-platform/envs/dev/main.tf at the end of the file

module "search" {
  source = "../../modules/team-namespace"

  team        = "search"
  environment = var.environment
}

Change in ~/platformengr/tf-platform/envs/dev/checks.tf

- [module.checkout, module.catalog]
+ [module.checkout, module.catalog, module.search]

cd ~/platformengr/tf-platform/envs/dev

terraform init

Expect Plan: 7 to add, 0 to change, 0 to destroy. The plan cannot know the namespace exists: it is not in state. Applying would fail with namespaces "search-dev" already exists. Do not apply.

terraform plan

3. Import it

Tell Terraform which existing object belongs at the module's address.

Create ~/platformengr/tf-platform/envs/dev/imports.tf

# Namespaces that teams made by hand before Terraform managed them.
import {
  to = module.search.kubernetes_namespace_v1.this
  id = "search-dev"
}

Find # module.search.kubernetes_namespace_v1.this will be updated in-place with # (imported from "search-dev"), the label lines (- "owner", + "team", + "environment"), and the last line Plan: 1 to import, 6 to add, 1 to change, 0 to destroy.

terraform plan

Apply it (type yes):

terraform apply

The AGE did not reset: the namespace was adopted, not recreated.

kubectl get namespace search-dev

Still there, and still not Terraform's: you imported the namespace, not what is in it.

kubectl get configmap search-notes -n search-dev

4. Drift

Someone raises catalog's quota by hand.

kubectl -n catalog-dev patch resourcequota team-quota --type merge -p '{"spec":{"hard":{"pods":"50"}}}'

See only what changed outside Terraform. Find Note: Objects have changed outside of Terraform, the line ~ "pods" = "20" -> "50", and the sentence that this plan will not change any objects. Do not apply it: it would write 50 into the state as if it were right.

terraform plan -refresh-only

Now the normal plan: the opposite arrow, "50" -> "20", and Plan: 0 to add, 1 to change, 0 to destroy.

terraform plan

Two honest outcomes: the hand change was right, so it goes into the code; or it was not, so you apply. Here nobody can say why it was made. Apply to undo it:

terraform apply

5. What no plan shows

Someone makes a user admin in checkout's namespace, by hand.

kubectl create rolebinding helper --clusterrole=admin --user=mallory -n checkout-dev

Expect No changes. Someone is admin in checkout's namespace and Terraform has nothing to report: the binding was never in its state.

terraform plan

Remove it yourself:

kubectl delete rolebinding helper -n checkout-dev

6. Commit

Save the import. The drift and the hand-made binding left nothing in the code.

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Import search-dev" && git push -q)

7. Upgrade the provider in dev

Kubernetes provider 3.5.0 is out. Upgrades get their own branch. Read the provider's changelog for 3.5.0 first: the BREAKING CHANGES and NOTES sections, before the plan, not after.

cd ~/platformengr/tf-platform && git switch -c provider-3.5.0

Change in ~/platformengr/tf-platform/envs/dev/versions.tf

- version = "3.4.0"
+ version = "3.5.0"

cd ~/platformengr/tf-platform/envs/dev

Expect an error: the locked provider 3.4.0 does not match the configured version constraint >= 3.0.0, 3.5.0, < 4.0.0 (the module's range and your pin, merged), and a hint to use init -upgrade. The lock file refused to let the version change quietly.

terraform init

terraform init -upgrade

Three things changed in the lock file: version, constraints and hashes.

git diff .terraform.lock.hcl

The proof. Read the plan before anything else. Expect in-place updates on kubernetes_role_binding_v1 only, each with labels = {} -> null and annotations = {} -> null under metadata: 3.5.0 rewrote this resource and stores an empty map as no map. Nothing in role_ref or subject changes, nothing is replaced, and the last line says 0 to destroy. That is an upgrade plan you understand. (If yours says No changes., even better.)

terraform plan

Apply it in dev, from the branch. This is the one exception to applying after the merge: dev is where an upgrade is tried, so the pull request can say it already ran once. Prod waits for the merge.

terraform apply

Expect No changes. now: the state is written by the new version.

terraform plan

init wrote checksums for your machine only. Add the other platforms, so the lock file works on every colleague's laptop and in CI:

terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64

8. Then prod, and merge

The same four commands in envs/prod, in the same pull request.

Change in ~/platformengr/tf-platform/envs/prod/versions.tf

- version = "3.4.0"
+ version = "3.5.0"

cd ~/platformengr/tf-platform/envs/prod

terraform init -upgrade

Expect the same kind of plan as in dev: role binding metadata only, 0 to destroy. Do not apply prod yet: prod waits for the review.

terraform plan

terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64

cd ~/platformengr/tf-platform && git add -A && git commit -qm "kubernetes provider 3.5.0" && git push -qu origin provider-3.5.0

gh pr create --fill

Expect four pass. static ran init -lockfile=readonly on Linux: it passes only because the lock file has Linux checksums.

gh pr checks --watch

gh pr merge --squash --delete-branch && git switch main && git pull -q

Merged and reviewed: now apply prod from main, after one more look at the plan.

cd ~/platformengr/tf-platform/envs/prod

terraform apply

Expect No changes.

terraform plan

9. Let a bot propose the next one

Dependabot reads the two directories once a week and opens a pull request when a provider has a new release, with the lock file already changed. Your part stays the same: the changelog and the plan.

Create ~/platformengr/tf-platform/.github/dependabot.yml

# Proposes provider upgrades as pull requests, with the lock file already updated.
version: 2
updates:
  - package-ecosystem: terraform
    directories: ["/envs/dev", "/envs/prod"]
    schedule:
      interval: weekly

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Dependabot for providers" && git push -q)

# Proposes provider upgrades as pull requests, with the lock file already updated.
version: 2
updates:
  - package-ecosystem: opentofu
    directories: ["/envs/dev", "/envs/prod"]
    schedule:
      interval: weekly

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Dependabot for providers" && git push -q)

Simplified for the course

Drift is found here when you run a plan. Teams schedule plan -refresh-only -detailed-exitcode every night and alert on exit code 2. It needs a runner that can reach the state, so it is in docs/cloud.md.

Terraform does not remove what it does not manage. Keeping a cluster equal to Git all the time, including deleting what should not be there, is what Argo CD does.

For many objects, terraform plan -generate-config-out=generated.tf writes the resource blocks for your import blocks. Treat the result as a draft.

An import block can be deleted after it has been applied everywhere. Keep yours: the lab checks read it.

Break it on purpose: The import wants to destroy what it imports

team-reporting's namespace is older than the naming rule: it is called reporting, with no environment. A colleague adds the module call and an import block, as you just did for search. Do this one, fix included: the next steps build on what the fix leaves in your repo.

You practise: adopting something that does not fit your module, without renaming it and without bending every other team.

1. Break it

kubectl create namespace reporting

Add to ~/platformengr/tf-platform/envs/dev/main.tf at the end of the file

module "reporting" {
  source = "../../modules/team-namespace"

  team        = "reporting"
  environment = var.environment
}

Add to ~/platformengr/tf-platform/envs/dev/imports.tf at the end of the file

import {
  to = module.reporting.kubernetes_namespace_v1.this
  id = "reporting"
}

cd ~/platformengr/tf-platform/envs/dev

terraform init

terraform plan

2. What you see

Expected: Plan: 1 to import, 6 to add, 1 to change, 0 to destroy. You get: Error: Instance cannot be destroyed for module.reporting.kubernetes_namespace_v1.this. The object you are importing is the one Terraform wants to destroy.

3. Fix it yourself

Adopt reporting under its existing name. Done when the plan says 1 to import, 6 to add, 1 to change, 0 to destroy and nothing for any other team. Apply it. A namespace cannot be renamed, and the team's workloads are in it.

variable "name_override" {
  description = "Only for namespaces that existed before this module. Leave unset for new teams."
  type        = string
  default     = null
}

- namespace = "${var.team}-${var.environment}"
+ namespace = coalesce(var.name_override, "${var.team}-${var.environment}")

-   team        = "reporting"
-   environment = var.environment
+   team          = "reporting"
+   environment   = var.environment
+   name_override = "reporting"

- module.search]
+ module.search, module.reporting]

terraform plan

terraform apply

cd ~/platformengr/tf-platform/modules/team-namespace && terraform test

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Adopt reporting under its legacy name" && git push -q)

Quiz

1. What does an import block do on the next apply?
   • It copies the object's settings into your .tf files, so the code matches
   • It records an existing object in state under the given address; nothing is created
   • It deletes the existing object and creates it again from the code
2. terraform plan says No changes. Can you conclude nobody changed anything in the namespace by hand?
   • No: Terraform only compares objects that are in its state
   • Yes: a plan reads every object in every namespace it manages
   • Yes, as long as the plan was run with -refresh-only
3. You raise a provider version in versions.tf and init fails. What is the right next step?
   • Delete .terraform.lock.hcl and run init again, so it is written fresh
   • Remove the version constraint so the newest provider is always used
   • Run init -upgrade, read the lock file diff and the changelog, then expect an empty plan

Field note

After an incident I ask what was changed by hand, and I run a refresh-only plan to check the answer. The list from the plan is usually longer than the list from memory.

Report a problem with this module