Module 06 · Safe change · 45 min · You are: Platform and product

Tests, checks and a pipeline on every pull request

So far the only test was you, reading a plan on your laptop. That does not scale to product teams proposing their own quotas, and it depends on the laptop being clean (module 02). A pipeline gives every change the same checks from a clean checkout. This pipeline cannot reach your laptop's cluster or state store, which forces an honest split: what can be checked from the files alone, and what needs the real state. The first group runs here. The second group is the plan, and you will see where it goes in a company.

Open this module in the interactive course to track progress and switch tracks.

Loop practised: Write → Plan → Review → Apply → State

Concepts

{TF} test

Runs *.tftest.hcl files. Each run block plans or applies the module with given variables and checks assert conditions. A failing assertion fails the command.

mock_provider

Replaces a provider with a fake that accepts any configuration and invents computed values. No cluster, no credentials. Values you set yourself are real; values the API would return are not.

expect_failures

A run that passes only if a named validation fails. It tests that bad input is refused.

check block

An assertion in normal configuration. When it fails, plan and apply print a warning and carry on. For things worth knowing that should not block.

Policy check

Rules about the files themselves, written once by the platform team and run on every pull request. Here conftest reads the HCL and applies policy/tenancy.rego.

Four ways to say no

What CI can and cannot see

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• modules/team-namespace/tests/unit.tftest.hcl (reference, your repo): The module's tests, as they end up in the capstone.
  1. No cluster needed L2
  2. Passes only if the validation refuses staging L27–34
• envs/dev/checks.tf (reference, your repo): A budget that warns, it does not block.
  1. Every team's pod quota, added up, against what the cluster can run L1–4
• .github/workflows/terraform.yaml (reference, your repo): The checks every pull request runs.
  1. Only when Terraform files, the policy or this workflow change L4–6
  2. No state, and the lock file must already be right L32
  3. The module's tests, from a clean checkout L44–45
  4. The policy, on the files L51
• policy/tenancy.rego (reference, your repo): Two rules the platform team keeps for every pull request.
  1. No ClusterRoleBinding anywhere in this repo L7–10
  2. Every S3 backend sets the lock L13–17

Steps

1. Write the first test

A test file next to the module. It plans the module with a fake provider and checks one promise: the namespace's name.

mkdir -p ~/platformengr/tf-platform/modules/team-namespace/tests

Create ~/platformengr/tf-platform/modules/team-namespace/tests/unit.tftest.hcl

# Plans the module against a fake provider: no cluster, no credentials.
mock_provider "kubernetes" {}

variables {
  team        = "checkout"
  environment = "dev"
}

run "names_the_namespace_team_dash_environment" {
  command = plan

  assert {
    condition     = kubernetes_namespace_v1.this.metadata[0].name == "checkout-dev"
    error_message = "The namespace must be named <team>-<environment>."
  }
}

A module directory needs its own init to run tests. The lock file it writes there is ignored by .gitignore: a reusable module does not pin exact versions. Its init picks the newest version in the module's range, so the tests may run on a newer provider than the one envs/dev pins; that is the point of a range.

cd ~/platformengr/tf-platform/modules/team-namespace && terraform init

Run it. Expect run "names_the_namespace_team_dash_environment"... pass and Success! 1 passed, 0 failed.

terraform test

No namespace was created. Expect the same four as before: catalog-dev, catalog-prod, checkout-dev and checkout-prod.

kubectl get namespaces -l app.kubernetes.io/managed-by=terraform

2. Pin the default and test a refusal

Two more promises: a team that sets no quota gets 10 pods (the default module 04's break almost changed), and an unknown environment is refused.

Add to ~/platformengr/tf-platform/modules/team-namespace/tests/unit.tftest.hcl at the end of the file

run "quota_defaults_to_ten_pods" {
  command = plan

  assert {
    condition     = kubernetes_resource_quota_v1.this.spec[0].hard["pods"] == "10"
    error_message = "A team that sets no quota gets 10 pods."
  }
}

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "staging"
  }

  expect_failures = [var.environment]
}

Expect Success! 3 passed, 0 failed.

terraform test

3. Watch a test fail

Swap the two parts of the name. validate would pass this, and a plan in dev would show a replacement. The test says it first.

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

- namespace = "${var.team}-${var.environment}"
+ namespace = "${var.environment}-${var.team}"

Find run "names_the_namespace_team_dash_environment"... fail, Error: Test assertion failed, your own message, and Failure! 2 passed, 1 failed.

terraform test

Change it back. Expect Success! 3 passed, 0 failed. again.

Change in ~/platformengr/tf-platform/modules/team-namespace/main.tf

- namespace = "${var.environment}-${var.team}"
+ namespace = "${var.team}-${var.environment}"

terraform test

4. Add a check that warns

Do the teams' pod quotas, added up, fit on the cluster? Worth knowing on every plan; not worth blocking one. The module gives its pod quota as an output, each environment gets a capacity, and a check adds them up.

Add to ~/platformengr/tf-platform/modules/team-namespace/outputs.tf at the end of the file

output "quota_pods" {
  description = "The team's pod quota, for the environment's capacity check."
  value       = var.quota.pods
}

Add to ~/platformengr/tf-platform/envs/dev/variables.tf, ~/platformengr/tf-platform/envs/prod/variables.tf at the end of each file

variable "cluster_pod_capacity" {
  description = "How many pods the cluster can run: what one kind node allows."
  type        = number
  default     = 110
}

Create ~/platformengr/tf-platform/envs/dev/checks.tf

check "quotas_fit_the_cluster" {
  assert {
    condition     = sum([for m in [module.checkout, module.catalog] : m.quota_pods]) <= var.cluster_pod_capacity
    error_message = "The teams' pod quotas add up to more than the cluster can run."
  }
}

cd ~/platformengr/tf-platform/envs/dev

Expect No changes.: 10 and 20 pods fit in 110.

terraform plan

Now make it fail without changing a file. Find Warning: Check block assertion failed and your message, and see that the plan still ends normally:

terraform plan -var cluster_pod_capacity=25

The same check in prod, where the numbers are bigger. Expect No changes. and no warning: 30 and 20 fit in 110.

cp ~/platformengr/tf-platform/envs/dev/checks.tf ~/platformengr/tf-platform/envs/prod/

cd ~/platformengr/tf-platform/envs/prod

terraform plan

5. Run the policy check on your laptop

Two rules the platform team wants on every pull request: nothing binds cluster-wide, and every backend is locked. conftest reads the .tf files as data; it never runs Terraform.

mkdir -p ~/platformengr/tf-platform/policy && cp ~/platformengr/terraform-platform/policy/tenancy.rego ~/platformengr/tf-platform/policy/

Expect a last line with 0 failures. Every .tf file was tested; none broke a rule.

cd ~/platformengr/tf-platform && docker run --rm -v "$PWD:/project" -w /project openpolicyagent/conftest:v0.71.1 test --parser hcl2 --ignore '\.terraform' -p policy envs modules

A rule you have never seen fail proves nothing. Add a cluster-wide binding to the module, in a scratch file:

Create ~/platformengr/tf-platform/modules/team-namespace/scratch.tf

resource "kubernetes_cluster_role_binding_v1" "oops" {
  metadata {
    name = "oops"
  }
}

Expect FAIL - modules/team-namespace/scratch.tf - main - kubernetes_cluster_role_binding_v1.oops: cluster-wide bindings are not allowed in this repo and a last line with 1 failure:

cd ~/platformengr/tf-platform && docker run --rm -v "$PWD:/project" -w /project openpolicyagent/conftest:v0.71.1 test --parser hcl2 --ignore '\.terraform' -p policy envs modules

Delete the scratch file:

rm ~/platformengr/tf-platform/modules/team-namespace/scratch.tf

6. Add the workflow

Three jobs on every pull request: static (format, init without the backend, validate) once per environment, test, and policy. The actions are pinned by commit, with the tag in a comment. The pull request template asks for the last line of the plan, which only a person with access can paste.

mkdir -p ~/platformengr/tf-platform/.github/workflows

cp ~/platformengr/terraform-platform/.github/workflows/terraform.yaml ~/platformengr/tf-platform/.github/workflows/terraform.yaml

cp ~/platformengr/terraform-platform/.github/workflows/terraform.tofu.yaml ~/platformengr/tf-platform/.github/workflows/terraform.yaml  # the OpenTofu version, same name

cp ~/platformengr/terraform-platform/.github/pull_request_template.md ~/platformengr/tf-platform/.github/

Before pushing, check the one job that most often fails on a laptop that works: format. Expect no output.

cd ~/platformengr/tf-platform && terraform fmt -check -recursive

7. Send a change through it

Everything so far goes on a branch and into a pull request.

The static job runs init -lockfile=readonly on Linux: it accepts the lock file only if it has a checksum for the Linux build of each provider. init wrote one for your machine only. Add the platforms CI and your colleagues use, in both environments:

cd ~/platformengr/tf-platform/envs/dev && terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64

cd ~/platformengr/tf-platform/envs/prod && terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64

Expect Success! Terraform has updated the lock file. twice (OpenTofu: Success! OpenTofu has updated the lock file.). Then the branch:

cd ~/platformengr/tf-platform && git switch -c ci && git add -A && git commit -qm "Tests, a check and CI" && git push -qu origin ci

gh pr create --fill

Watch the checks. Expect four rows, static (envs/dev), static (envs/prod), test and policy, all pass. None of them contacted your cluster.

gh pr checks --watch

Merge it and go back to main:

gh pr merge --squash --delete-branch && git switch main && git pull -q

Simplified for the course

No job plans against real state: a GitHub runner cannot reach a cluster and a state store on your laptop. Where the backend is reachable, a fourth job runs plan -out, posts the plan on the pull request, and a separate, approved job applies that exact file. docs/cloud.md in the reference repo has both jobs.

A mocked plan proves what you wrote, not what the API will answer. Teams add a second kind of test that applies the module against a throwaway cluster and destroys it.

One policy file with two rules. Larger setups run the policy on the JSON of a real plan, which can see computed values, and add a linter such as tflint.

The repo has no branch protection. With it, the three jobs are required and main only moves through a pull request.

Break it on purpose: All checks are green, and the deployer can read Secrets

team-checkout's pipeline cannot read the checkout-db Secret, so a developer proposes a one-word change to the module. It is formatted, valid and breaks no test. Do this one, fix included: the next steps build on what the fix leaves in your repo.

You practise: turning a rule that lives in people's heads into a test, and proving the test works by seeing it fail first.

1. Break it

cd ~/platformengr/tf-platform && git switch -c deployer-secrets

Change in ~/platformengr/tf-platform/modules/team-namespace/rbac.tf

- resources  = ["services", "configmaps"]
+ resources  = ["services", "configmaps", "secrets"]

git commit -qam "deployer: read secrets" && git push -qu origin deployer-secrets

gh pr create --fill

gh pr checks --watch

2. What you see

What you want: a red check. Since module 04 the rule is that deployer cannot read Secrets, in any team, in any environment. You get: four green checks. Nothing in the repo knows the rule.

3. Fix it yourself

On this branch, make the pipeline fail because of this change, then make it pass again by removing the cause. Done when the pull request shows a failed test run followed by a passing one, and rbac.tf on main has no secrets. Merge it: the test is worth keeping.

run "deployer_cannot_read_secrets" {
  command = plan

  assert {
    condition     = !contains(flatten([for r in kubernetes_role_v1.deployer.rule : r.resources]), "secrets")
    error_message = "The deployer role must not include secrets."
  }
}

git commit -qam "test: deployer cannot read secrets" && git push -q

gh pr checks --watch

gh run view --log-failed

- resources  = ["services", "configmaps", "secrets"]
+ resources  = ["services", "configmaps"]

git commit -qam "deployer: no secrets" && git push -q

gh pr checks --watch

gh pr merge --squash --delete-branch && git switch main && git pull -q

Quiz

1. What does mock_provider "kubernetes" {} change in a test?
   • The provider is replaced by a fake, so the test needs no cluster; values the API would compute are invented
   • The test runs against a temporary cluster that is created first and deleted afterwards
   • Assertions on resources of that provider are skipped, so only variables are checked
2. A check block's assertion fails during apply. What happens?
   • The apply stops before making any change and prints the message
   • Terraform prints a warning and the apply completes
   • The resources named in the check are rolled back to their state before
3. Why does this workflow have no plan job?
   • plan is not allowed on pull requests that come from the same repository
   • validate already does everything that plan does, only faster
   • A plan needs the real state and the real API, and the runner can reach neither

Field note

I do not trust a new test until I have seen it fail for the reason it was written. A test that has only ever been green might be asserting nothing.

Report a problem with this module