PlatformEngr® Terraform course

Module 09 · Safe change · 35 min · You are: Platform and product

Capstone: onboard team-payments on your own

In this module

  • You learn: Nothing new. You use everything from modules 01 to 08 for a team with one requirement the module cannot meet yet.
  • You do: Onboard team-payments in dev and prod without a script: adopt the namespace they made by hand, give an audit group read access to payments only, change the module without disturbing any other team, prove it with a test, and take it through the pipeline. Then retire a team.
  • You end with: payments-dev and payments-prod have the full kit plus a second read-only binding for team-finance-audit. No other team's objects were recreated. The module has an extra_viewers input, a moved block and five tests. docs/onboarding.md says how to do it again. team-reporting is out of Terraform's state and its namespace is gone.

The work of a platform team is this request, again and again: one more team, with one thing that is different. The module has to grow without breaking the teams already on it, the existing object has to be adopted and not recreated, and the change has to reach dev before prod. Nobody tells you the order this time.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

Extend, keep the default
A new optional input whose default reproduces today's behaviour. Existing callers change nothing and see an empty plan.
Static keys
When one resource becomes many with for_each, the keys are names you choose (team, audit), never values from elsewhere.
moved to an instance key
to = kubernetes_role_binding_v1.view["team"] keeps the existing binding when the resource gains for_each.
Forget, do not destroy
A removed block with destroy = false takes objects out of state and leaves them running. The decision to delete is then a separate, deliberate act.

The request

RequirementFrom
Namespaces payments-dev and payments-prod with the standard kitThe platform's rule
payments-dev already exists and has workloads in itteam-payments
Quota 10 pods in dev, 30 in prodteam-payments
The group team-finance-audit can read both payments namespaces and no otherThe auditors
No other team may see any change in its planThe platform team
Dev first, then prod, through one pull request with green checksThe platform team

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. The starting point

What team-payments did last month, and a check that you start clean.

kubectl create namespace payments-dev && kubectl label namespace payments-dev team=payments

Expect No changes. in both:

cd ~/platformengr/tf-platform/envs/dev && terraform plan

cd ~/platformengr/tf-platform/envs/prod && terraform plan

2. Onboard team-payments (Challenge)

Done when:

Hint 1

Order: module change and test first, with empty plans for existing teams; then the calls; then the import; then prod.

Hint 2

viewers = merge({ team = "team-${var.team}" }, var.extra_viewers) in the locals; for_each = local.viewers; the binding's name is each.key == "team" ? "team-view" : "view-${each.key}"; the subject is each.value.

Hint 3

moved { from = kubernetes_role_binding_v1.view to = kubernetes_role_binding_v1.view["team"] } goes in the module, so every caller gets it.

Hint 4

The test: variables { extra_viewers = { audit = "team-finance-audit" } } and length(kubernetes_role_binding_v1.view) == 2.

One solution

The reference repo's module, envs/*/main.tf, checks.tf, outputs.tf and envs/dev/imports.tf are this solution.

cd ~/platformengr/tf-platform && git switch -c payments

1. The module. A new optional input, a map of viewers, the binding over that map, an output from it, and the moved block.

Add to ~/platformengr/tf-platform/modules/team-namespace/variables.tf at the end of the file


variable "extra_viewers" {
  description = "More groups with read access to this namespace, keyed by a short name, for example { audit = \"team-finance-audit\" }."
  type        = map(string)
  default     = {}
}

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

-     "platformengr.com/kit"         = "v1"
-   }
- }
+     "platformengr.com/kit"         = "v1"
+   }
+   viewers = merge({ team = "team-${var.team}" }, var.extra_viewers)
+ }

Change in ~/platformengr/tf-platform/modules/team-namespace/rbac.tf

- resource "kubernetes_role_binding_v1" "view" {
-   metadata {
-     name      = "team-view"
+ resource "kubernetes_role_binding_v1" "view" {
+   for_each = local.viewers
+ 
+   metadata {
+     name      = each.key == "team" ? "team-view" : "view-${each.key}"

Change in ~/platformengr/tf-platform/modules/team-namespace/rbac.tf

-     name      = "team-${var.team}"
+     name      = each.value

Change in ~/platformengr/tf-platform/modules/team-namespace/outputs.tf

- value       = ["team-${var.team}"]
+ value       = values(local.viewers)

Create ~/platformengr/tf-platform/modules/team-namespace/moved.tf

# The team's read-only binding became one of several (capstone). Same object, new address.
moved {
  from = kubernetes_role_binding_v1.view
  to   = kubernetes_role_binding_v1.view["team"]
}

Add to ~/platformengr/tf-platform/modules/team-namespace/tests/unit.tftest.hcl at the end of the file


run "extra_viewer_gets_a_second_binding" {
  command = plan

  variables {
    extra_viewers = { audit = "team-finance-audit" }
  }

  assert {
    condition     = length(kubernetes_role_binding_v1.view) == 2
    error_message = "A team with one extra viewer gets two read-only bindings."
  }
}

Expect Success! 5 passed, 0 failed.

cd ~/platformengr/tf-platform/modules/team-namespace && terraform test

The bar for the module change: every existing team, empty. Expect four has moved to lines and Plan: 0 to add, 0 to change, 0 to destroy. in dev (two lines in prod). Apply both.

cd ~/platformengr/tf-platform/envs/dev

terraform plan

terraform apply

cd ~/platformengr/tf-platform/envs/prod

terraform apply

2. The calls and the import, dev first.

Add to ~/platformengr/tf-platform/envs/dev/main.tf at the end of the file


module "payments" {
  source = "../../modules/team-namespace"

  team          = "payments"
  environment   = var.environment
  quota         = { pods = 10 }
  extra_viewers = { audit = "team-finance-audit" }
}

Add to ~/platformengr/tf-platform/envs/dev/imports.tf at the end of the file


import {
  to = module.payments.kubernetes_namespace_v1.this
  id = "payments-dev"
}

Change in ~/platformengr/tf-platform/envs/dev/checks.tf

- module.search, module.reporting]
+ module.search, module.reporting, module.payments]

Add to ~/platformengr/tf-platform/envs/dev/outputs.tf at the end of the file


output "viewer_groups" {
  description = "Every group with read access to a namespace in this environment."
  value       = flatten([for m in [module.checkout, module.catalog, module.search, module.reporting, module.payments] : m.viewer_groups])
}

cd ~/platformengr/tf-platform/envs/dev

terraform init

Expect Plan: 1 to import, 7 to add, 1 to change, 0 to destroy., all under module.payments. Apply it.

terraform plan

terraform apply

Same AGE as before:

kubectl get namespace payments-dev

3. Then prod. No import: payments-prod does not exist yet.

Add to ~/platformengr/tf-platform/envs/prod/main.tf at the end of the file


module "payments" {
  source = "../../modules/team-namespace"

  team          = "payments"
  environment   = var.environment
  quota         = { pods = 30 }
  extra_viewers = { audit = "team-finance-audit" }
}

Change in ~/platformengr/tf-platform/envs/prod/checks.tf

- [module.checkout, module.catalog]
+ [module.checkout, module.catalog, module.payments]

Create ~/platformengr/tf-platform/envs/prod/outputs.tf

output "checkout_namespace" {
  description = "The namespace team-checkout deploys into."
  value       = module.checkout.namespace
}

output "viewer_groups" {
  description = "Every group with read access to a namespace in this environment."
  value       = flatten([for m in [module.checkout, module.catalog, module.payments] : m.viewer_groups])
}

cd ~/platformengr/tf-platform/envs/prod

terraform init

Expect Plan: 8 to add, 0 to change, 0 to destroy. Apply it.

terraform apply

4. Prove it. Expect yes, then no:

kubectl auth can-i list pods -n payments-prod --as=x --as-group=team-finance-audit

kubectl auth can-i list pods -n checkout-prod --as=x --as-group=team-finance-audit

5. Write it down and ship it. Fill in the three sections in your own words.

Create ~/platformengr/tf-platform/docs/onboarding.md

# Onboarding a team

## Platform team

1.

## Product team

-

## When a team leaves

1.

cd ~/platformengr/tf-platform && git add -A && git commit -qm "Onboard team-payments; module: extra_viewers" && git push -qu origin payments

gh pr create --fill

gh pr checks --watch

gh pr merge --squash --delete-branch && git switch main && git pull -q
The five questions, answered
What failed?
If you skip the moved block: every team's team-view binding is destroyed and recreated, so each team loses read access for a moment, in prod too.
How was it detected?
The plan: 1 to add, 1 to destroy per team.
Who responds?
The platform team.
How is it corrected?
The moved block.
How is it prevented?
The bar for a module change is an empty plan for every existing caller.

Simplified for the course

Five teams in hand-written module blocks is still readable. At twenty, teams usually become a map in a tfvars file and one module block with for_each; the keys are the team names, which are static.

The audit group gets the built-in view role. A real audit role is narrower and comes from the security team.

Break it on purpose: Removing a team would delete its namespace

team-reporting is shut down. A colleague deletes its module block, its import block and its two list entries. They expect the module's guard to stop any deletion of the namespace.

You practise: what a guard written in code can protect, and what it cannot: it leaves with the code.

1. Break it

Delete from ~/platformengr/tf-platform/envs/dev/main.tf

- 
- module "reporting" {
-   source = "../../modules/team-namespace"
- 
-   team          = "reporting"
-   environment   = var.environment
-   name_override = "reporting"
- }

Delete from ~/platformengr/tf-platform/envs/dev/imports.tf

- 
- import {
-   to = module.reporting.kubernetes_namespace_v1.this
-   id = "reporting"
- }

Change in ~/platformengr/tf-platform/envs/dev/checks.tf

- module.search, module.reporting, module.payments]
+ module.search, module.payments]

Change in ~/platformengr/tf-platform/envs/dev/outputs.tf

- module.search, module.reporting, module.payments]
+ module.search, module.payments]

cd ~/platformengr/tf-platform/envs/dev

terraform plan

2. What you see

What your colleague expects: Error: Instance cannot be destroyed, the module's prevent_destroy protecting the namespace. What happens: Plan: 0 to add, 0 to change, 7 to destroy., and one of the seven is module.reporting.kubernetes_namespace_v1.this. An apply would delete the namespace and everything running in it. Do not apply.

3. Fix it yourself

Take team-reporting out of Terraform's state without Terraform deleting anything, then delete the namespace yourself, as the deliberate last step. Done when terraform plan says No changes., terraform state list has no module.reporting address, and kubectl get namespace reporting says NotFound.

Hint 1

prevent_destroy is read from the configuration of the resource. The module block is gone, so for Terraform there is no configuration and no guard. A destroy is not the only way an address leaves the state (lab 2).

Hint 2

removed { from = module.reporting lifecycle { destroy = false } } in envs/dev.

The fix

Add this to docs/onboarding.md under "When a team leaves", so the next person does not remove the guard from the module to get past it.

Create ~/platformengr/tf-platform/envs/dev/removed.tf

# team-reporting left. Terraform forgets its objects; deleting the namespace is a separate, deliberate step.
removed {
  from = module.reporting

  lifecycle {
    destroy = false
  }
}

Expect seven lines that say the object will no longer be managed by Terraform, but will not be destroyed, and 0 to destroy. Apply it.

terraform plan

terraform apply

Still there: Terraform only forgot it.

kubectl get namespace reporting

The deliberate last step, once team-reporting has confirmed nothing in it is needed:

kubectl delete namespace reporting

Expect No changes.

terraform plan

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "Retire team-reporting" && git push -q)
After you fixed it: the five questions
What failed?
The guard was not there to fail: prevent_destroy lives in the block that was deleted, so the plan proposed deleting the namespace with no error.
How was it detected?
At plan time, by a person reading the last line and the namespace line before typing yes. Nothing else would have stopped it.
Who responds?
The platform team removes the team from the code; deleting the namespace is a decision someone signs off, after team-reporting confirms nothing in it is needed.
How is it corrected?
Forget in Terraform, delete by hand.
How is it prevented?
Read every plan that destroys something, line by line, and never apply one you did not expect. Write the removed-block path under "When a team leaves", so nobody deletes a module block to retire a team.

Quiz

  1. You add for_each to a resource that existed without it. What does the plan show without a moved block?
    • The old instance destroyed and a new, keyed instance created
    • The old instance updated in place, with its new key added
    • An error that asks you to choose a key for the old instance
  2. How do you add an input to a module that four teams already use, safely?
    • Add it as required, so that every caller has to think about it once
    • Make it optional with a default that keeps today's behaviour, and check that every existing caller's plan is empty
    • Add it to one environment's copy of the module first, then to the other
  3. What does removed { from = module.reporting lifecycle { destroy = false } } do on apply?
    • It destroys the module's objects and then removes them from the state
    • It marks the module as deprecated and prints a warning on every plan
    • It deletes the module's objects from state and leaves the real objects running
Answers
  1. The old instance destroyed and a new, keyed instance created view and view["team"] are different addresses.
  2. Make it optional with a default that keeps today's behaviour, and check that every existing caller's plan is empty An empty plan for existing callers is the proof that the change is additive.
  3. It deletes the module's objects from state and leaves the real objects running destroy = false means forget; the objects become unmanaged.

Field note

The second time I onboard a team I write the checklist. The third time I change the module so the checklist gets shorter. If the checklist grows with every team, the module is missing an input.

Report a problem with this module