Module 03 · When it happens · 50 min · You are: Platform team
Incident response
In this module
- You learn: Preserve evidence and contain a compromised dependency, workflow or image.
- You do: Work three seeded incident cases: moved action reference, unknown workflow and denied image. Disable the affected path, rotate the fake credential, capture evidence and update the register.
- You end with: Three incident records name the surface, timeline, containment and follow-up control.
Containment without evidence makes the next question harder: which runs were exposed and what capability did they have?
Open this module in the interactive course to track progress.
Change journey practised: Surface → Test → Measure → Prove
Concepts
- Surface: delivery evidence and alert
- The lab incident is a controlled simulation with no exfiltration or real credential.
- Trust boundary
- A failed admission can be evidence that a boundary is working.
- Evidence
- Rotate credentials after preserving the minimum useful run metadata.
Files to know
Open them in your editor or on GitHub; the bullets say what to look at.
start/docs/incidents/001.md(reference): Reference file for the runtime surface.- This line shows the trust boundary or the evidence shape. L3
start/.github/workflows/ghost-action-lab.yml(reference): Bounded lab fixture for this module.- Use this fixture to test the hypothesis safely. L1
Steps
1. Investigate: preserve evidence before containment
Choose one seeded incident. Find the first affected commit and the workflows that could read credentials. Disable the affected path, rotate only fake lab credentials and write docs/incidents/001.md.
cd devsecops-lab && git log --diff-filter=A -- .github/workflows/
Preserve the commit id, workflow run id and action reference before disabling the workflow.
gh workflow list --repo YOUR_GITHUB_USER/devsecops-lab
2. Prove: record the stopping control
For the denied image, inspect cluster events and the policy report. Decide whether the refusal is an incident or the control working, then add the evidence and retest date.
kubectl get events -A --sort-by=.lastTimestamp
Simplified for the workshop
The metadata service is a local mock, and its sample credentials open nothing. The CI token verifier uses claims supplied by the workflow and does not contact a cloud provider.
The lab checks parse files only. They never run code from the learner repository.
Break it on purpose: Break it: an unexpected workflow file appears
A change appears to work because the easiest test is blocked by another control. Lift only the named lab control for one test and isolate the boundary under study.
You practise: You practise separating controls, predicting signals and restoring the runtime to its safe default.
1. Break it
cd devsecops-lab && git diff -- deploy/ infra/
2. What you see
Expected: the tested path stops at this control. You get a request or identity accepted that should have been denied. Compare the observed log with the pre-test prediction.
3. Fix it yourself
Restore the control and prove the same path is denied. Done when the register names the result, owner, evidence and retest date.
Hint 1
Trace the request from application to destination, including the network path.
Hint 2
Test each independent control alone, then restore all of them.
The fix
Follow the module policy files and reapply the default deny before continuing.
cd devsecops-lab && ./scripts/validate.sh
After you fixed it: the five questions
- What failed?
- A runtime or identity boundary allowed an unintended pivot.
- How was it detected?
- The bounded request or verifier returned an unexpected success.
- Who responds?
- The owning team in the register: product for the app, platform for cluster and cloud policy.
- How is it corrected?
- Close the specific boundary and repeat the request or identity test.
- How is it prevented?
- Keep an exact parse-only rule and a dated retest in the shared evidence register.
Quiz
- Why does the metadata mock belong in this lab?
- It provides a bounded local target whose sample credentials open no cloud account.
- It proves that real cloud credentials are safe to expose.
- It removes the need to inspect network policy.
- What does an exact OIDC subject condition do?
- Limits which repository or environment identity may assume the role.
- Encrypts credentials in GitHub logs.
- Proves that every commit is reviewed.
- What is the first response to an unexpected workflow?
- Preserve run and commit evidence, disable the path, and rotate exposed credentials.
- Delete every workflow and erase the repository history.
- Wait until a scanner confirms data was stolen.
Answers
- It provides a bounded local target whose sample credentials open no cloud account. The target must be contained and disposable.
- Limits which repository or environment identity may assume the role. Trust conditions define eligible identities.
- Preserve run and commit evidence, disable the path, and rotate exposed credentials. Containment and evidence both matter; rotate credentials that could be exposed.
Field note
I test the same hypothesis after each control. That shows which layer stopped the path and whether another layer still works.