Module 04 · State, modules and environments · 45 min · You are: Platform team

One module, two teams

team-catalog needs the same kit. Copying main.tf and replacing checkout with catalog works today and fails slowly: the next fix is made in one copy only. A module is the kit written once. The risk is in getting there: Terraform identifies objects by address, and moving a block into a module changes its address, so the naive plan destroys the namespace and creates it again. You will read that plan, and then make it say what you meant.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

Module

A directory of .tf files with variables in and outputs out. A module block calls it: module "checkout" { source = "../../modules/team-namespace" ... }.

Address

Inside a module the same resource is module.checkout.kubernetes_namespace_v1.this. A new address looks like a new object to Terraform, and the old address like one to remove.

moved block

moved { from = A to = B } says "the object recorded under A is now B". The plan shows has moved to, and no destroy.

A module does not configure providers

It only says which providers it needs and which versions it works with (a range). The root module configures the provider and pins the exact version.

prevent_destroy

A lifecycle setting. Any plan that would destroy that resource fails with an error. A guard for objects whose loss is expensive.

The module's interface

Outputs: namespace, deployer_service_account, viewer_groups.

What each subject may do

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• modules/team-namespace/variables.tf (reference, your repo): The module's inputs, as they end up. Today you write the first three.
  1. A team name the cluster will accept, checked before any plan L1–7
  2. An object with a default for each attribute L21–32
• modules/team-namespace/main.tf (reference, your repo): The kit, as it ends up. The guard is the lifecycle block.
  1. "this": the module has one namespace. prevent_destroy stops any plan that would delete it L13–23
• modules/team-namespace/rbac.tf (reference, your repo): Who may do what in the team's namespace.
  1. The deployer role: no secrets, no roles L30–47
• modules/team-namespace/versions.tf (reference, your repo): What the module needs.
  1. A range: the caller picks the exact version L7
• envs/dev/moved.tf (reference, your repo): How Terraform learned that checkout's objects moved and were not replaced.
  1. Old address, new address L2–4

Steps

1. Create the module from your own resources

A new directory with the module's versions and inputs. Then cut the locals block and the three resources out of envs/dev/main.tf and paste them into the module, renamed from checkout to this and reading the module's inputs. The tier label goes: it was checkout's own, not part of the kit.

mkdir -p ~/platformengr/tf-platform/modules/team-namespace

Create ~/platformengr/tf-platform/modules/team-namespace/versions.tf

terraform {
  required_version = ">= 1.11.0, < 2.0.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 3.0.0, < 4.0.0"
    }
  }
}

Create ~/platformengr/tf-platform/modules/team-namespace/variables.tf

variable "team" {
  description = "Team name without the team- prefix, for example checkout."
  type        = string

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{1,20}$", var.team))
    error_message = "team must be lower case letters, digits and dashes, 2 to 21 characters, starting with a letter."
  }
}

variable "environment" {
  description = "dev or prod."
  type        = string

  validation {
    condition     = contains(["dev", "prod"], var.environment)
    error_message = "environment must be dev or prod."
  }
}

variable "quota" {
  description = "What the team may use in this namespace."
  type = object({
    pods   = optional(number, 10)
    cpu    = optional(string, "2")
    memory = optional(string, "2Gi")
  })
  default = {}

  validation {
    condition     = var.quota.pods >= 1 && var.quota.pods <= 50
    error_message = "quota.pods must be between 1 and 50."
  }
}

Create ~/platformengr/tf-platform/modules/team-namespace/main.tf
The locals and the three resources from envs/dev/main.tf, renamed

locals {
  namespace = "${var.team}-${var.environment}"
  labels = {
    team                           = var.team
    environment                    = var.environment
    "app.kubernetes.io/managed-by" = "terraform"
  }
}

resource "kubernetes_namespace_v1" "this" {
  metadata {
    name   = local.namespace
    labels = local.labels
  }
}

resource "kubernetes_resource_quota_v1" "this" {
  metadata {
    name      = "team-quota"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
  spec {
    hard = {
      pods              = tostring(var.quota.pods)
      "requests.cpu"    = var.quota.cpu
      "requests.memory" = var.quota.memory
    }
  }
}

resource "kubernetes_limit_range_v1" "this" {
  metadata {
    name      = "team-limits"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
  spec {
    limit {
      type            = "Container"
      default         = { memory = "256Mi" }
      default_request = { cpu = "100m", memory = "128Mi" }
    }
  }
}

Create ~/platformengr/tf-platform/modules/team-namespace/outputs.tf

output "namespace" {
  description = "The namespace name."
  value       = kubernetes_namespace_v1.this.metadata[0].name
}

Now envs/dev calls the module instead. main.tf becomes one module block; checkout_quota leaves variables.tf (the module has its own quota); checkout.tf and outputs.tf take the namespace from the module's output.

Replace the whole file ~/platformengr/tf-platform/envs/dev/main.tf

module "checkout" {
  source = "../../modules/team-namespace"

  team        = "checkout"
  environment = var.environment
}

Delete from ~/platformengr/tf-platform/envs/dev/variables.tf

- 
- variable "checkout_quota" {
-   description = "What team-checkout may use in this environment."
-   type = object({
-     pods   = optional(number, 10)
-     cpu    = optional(string, "2")
-     memory = optional(string, "2Gi")
-   })
-   default = {}
- 
-   validation {
-     condition     = var.checkout_quota.pods >= 1 && var.checkout_quota.pods <= 50
-     error_message = "pods must be between 1 and 50."
-   }
- }

Replace the whole file ~/platformengr/tf-platform/envs/dev/checkout.tf
Both namespace lines now read module.checkout.namespace

resource "kubernetes_config_map_v1" "checkout_contact" {
  metadata {
    name      = "team-contact"
    namespace = module.checkout.namespace
  }
  data = {
    oncall = "#checkout-oncall"
  }
}

ephemeral "random_password" "checkout_db" {
  length  = 24
  special = false
}

resource "kubernetes_secret_v1" "checkout_db" {
  metadata {
    name      = "checkout-db"
    namespace = module.checkout.namespace
  }
  data_wo = {
    password = ephemeral.random_password.checkout_db.result
  }
  data_wo_revision = 1
}

Change in ~/platformengr/tf-platform/envs/dev/outputs.tf

- value       = kubernetes_namespace_v1.checkout.metadata[0].name
+ value       = module.checkout.namespace

2. Install the module, then read the plan. Do not apply

A new module block is an init job, like a new provider.

cd ~/platformengr/tf-platform/envs/dev

Expect Error: Module not installed:

terraform plan

Expect - checkout in ../../modules/team-namespace:

terraform init

Read the last line first, then find the namespace twice: # kubernetes_namespace_v1.checkout will be destroyed with # (because kubernetes_namespace_v1.checkout is not in configuration), and # module.checkout.kubernetes_namespace_v1.this will be created. Same object, two addresses. Applied, this deletes checkout-dev with everything in it, then tries to create it again. Do not apply.

terraform plan

3. Tell Terraform they are the same objects

One moved block per object: old address, new address.

Create ~/platformengr/tf-platform/envs/dev/moved.tf

moved {
  from = kubernetes_namespace_v1.checkout
  to   = module.checkout.kubernetes_namespace_v1.this
}

moved {
  from = kubernetes_resource_quota_v1.checkout
  to   = module.checkout.kubernetes_resource_quota_v1.this
}

moved {
  from = kubernetes_limit_range_v1.checkout
  to   = module.checkout.kubernetes_limit_range_v1.this
}

Find three lines like # kubernetes_namespace_v1.checkout has moved to module.checkout.kubernetes_namespace_v1.this, and the last line Plan: 0 to add, 1 to change, 0 to destroy.: the namespace loses tier = "web".

terraform plan

Apply it (type yes):

terraform apply

The three addresses now start with module.checkout.:

terraform state list

Same AGE as before: the namespace was never deleted.

kubectl get namespace checkout-dev

4. Make that plan impossible

Put a guard on the namespace, in the module, so every team gets it.

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

-     labels = local.labels
-   }
- }
+     labels = local.labels
+   }
+ 
+   lifecycle {
+     prevent_destroy = true
+   }
+ }

Try a dangerous change to see the guard. A typo in the team name means a new namespace name:

Change in ~/platformengr/tf-platform/envs/dev/main.tf

- team        = "checkout"
+ team        = "checkouts"

Expect Error: Instance cannot be destroyed, naming module.checkout.kubernetes_namespace_v1.this. The Plan: summary may print just before it; the error wins, and nothing can be applied.

terraform plan

Change it back:

Change in ~/platformengr/tf-platform/envs/dev/main.tf

- team        = "checkouts"
+ team        = "checkout"

5. Add RBAC to the kit

Four more objects per team: a read-only RoleBinding for the team's group, a deployer ServiceAccount, and the Role and RoleBinding that let it deploy. Kubernetes detail, written once in the module. Then two outputs, so a caller can see who has access.

Create ~/platformengr/tf-platform/modules/team-namespace/rbac.tf

# Who may do what in the team's namespace. Same model as the Kubernetes course:
# the team's people read, the deployer ServiceAccount deploys, nobody but the platform team reads Secrets.

resource "kubernetes_role_binding_v1" "view" {
  metadata {
    name      = "team-view"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "view"
  }
  subject {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Group"
    name      = "team-${var.team}"
  }
}

resource "kubernetes_service_account_v1" "deployer" {
  metadata {
    name      = "deployer"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
}

resource "kubernetes_role_v1" "deployer" {
  metadata {
    name      = "deployer"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
  rule {
    api_groups = ["apps"]
    resources  = ["deployments"]
    verbs      = ["get", "list", "watch", "create", "update", "patch", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["services", "configmaps"]
    verbs      = ["get", "list", "watch", "create", "update", "patch", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["pods"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_role_binding_v1" "deployer" {
  metadata {
    name      = "deployer"
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role_v1.deployer.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account_v1.deployer.metadata[0].name
    namespace = kubernetes_namespace_v1.this.metadata[0].name
  }
}

Add to ~/platformengr/tf-platform/modules/team-namespace/outputs.tf at the end of the file

output "deployer_service_account" {
  description = "The ServiceAccount the team's pipeline deploys with."
  value       = kubernetes_service_account_v1.deployer.metadata[0].name
}

output "viewer_groups" {
  description = "Every group with read access to the namespace."
  value       = ["team-${var.team}"]
}

Apply. Expect Plan: 4 to add, 0 to change, 0 to destroy. and type yes.

terraform apply

Ask the cluster, as someone else. A member of team-checkout may read. Expect yes:

kubectl auth can-i list pods -n checkout-dev --as=jane --as-group=team-checkout

The deployer may not read Secrets. Expect no:

kubectl auth can-i get secrets -n checkout-dev --as=system:serviceaccount:checkout-dev:deployer

6. Onboard team-catalog

This one you write: a second call of the module. catalog is smaller than checkout and asks for 5 pods.

Add to ~/platformengr/tf-platform/envs/dev/main.tf at the end of the file

module "catalog" {
  source = "../../modules/team-namespace"

  team        = "catalog"
  environment = var.environment
  quota       = { pods = 5 }
}

terraform init

Apply. Expect Plan: 7 to add, 0 to change, 0 to destroy. and type yes.

terraform apply

Expect pods: 0/5 in the REQUEST column:

kubectl get resourcequota -n catalog-dev

checkout's people see nothing of catalog. Expect no:

kubectl auth can-i list pods -n catalog-dev --as=jane --as-group=team-checkout

7. Commit

Save it.

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "team-namespace module; checkout moved into it; catalog onboarded" && git push -q)

Simplified for the course

The module lives in the same repo as its callers, so a merge changes it for every caller at once. Teams that need dev and prod on different versions publish the module with a Git tag or in a registry and write source = "git::https://github.com/ORG/REPO.git//modules/team-namespace?ref=v1.2.0".

moved blocks may be deleted once every state that had the old address has been applied. In a module used by other repos you keep them much longer. Keep yours: the lab checks read them.

prevent_destroy also blocks removing a team on purpose, and in Terraform it cannot be switched by a variable. Taking a team out is a removed block: the break in the capstone.

Users and groups come from --as here. In a company they come from the identity provider.

Break it on purpose: catalog asked for more pods; checkout's quota changes too

team-catalog needs 20 pods. A platform engineer finds the number 10 in the module, changes it there, and opens a pull request titled "catalog: 20 pods". Do this one, fix included: the next steps build on what the fix leaves in your repo.

You practise: reading which callers a module change touches, and putting a difference where it belongs.

1. Break it

Change in ~/platformengr/tf-platform/modules/team-namespace/variables.tf

- pods   = optional(number, 10)
+ pods   = optional(number, 20)

catalog has the default now, so its own number goes:

Delete from ~/platformengr/tf-platform/envs/dev/main.tf

-   quota       = { pods = 5 }

terraform plan

2. What you see

Expected: Plan: 0 to add, 1 to change, 0 to destroy. (catalog only). You get: Plan: 0 to add, 2 to change, 0 to destroy. Read the two addresses that change.

3. Fix it yourself

Give catalog 20 pods and leave every other team exactly as it was. Done when the plan changes only module.catalog.kubernetes_resource_quota_v1.this and git diff modules/ is empty. Apply it.

- pods   = optional(number, 20)
+ pods   = optional(number, 10)

-   team        = "catalog"
-   environment = var.environment
+   team        = "catalog"
+   environment = var.environment
+   quota       = { pods = 20 }

terraform plan

terraform apply

(cd ~/platformengr/tf-platform && git add -A && git commit -qm "catalog: 20 pods" && git push -q)

Quiz

1. You move a resource block into a module and run plan with no other change. What does Terraform propose?
   • To destroy the object at the old address and create one at the new address
   • Nothing at all: Terraform recognises the moved block by its content
   • To update the object in place, with its new address added as a label
2. Where is the provider configured for resources inside modules/team-namespace?
   • In the module's own providers.tf, written once for each module
   • In the root module that calls it; the module only declares a version range
   • Nowhere: a module always inherits the current kubectl context
3. A plan fails with Instance cannot be destroyed. What does that tell you?
   • The cluster refused the delete call because the namespace still has a finalizer
   • The state lock is held by someone else, so nothing can be destroyed yet
   • The plan would destroy a resource with prevent_destroy: find out why first

Field note

Before I approve a change inside a shared module, I ask for the plan of every caller, not only the one in the title. The title says what the author meant. The plans say what will happen.

Report a problem with this module