Module 06 · Share the cluster · 50 min · You are: Platform team

Walls: NetworkPolicy and Pod Security

RBAC decides what people may do to the API. It says nothing about what a running container may do: by default it can open a connection to any pod in the cluster, and it may ask to run as root with the node's filesystem mounted. A product team rarely needs either. These two fences are the platform team's, and both have the same trap: they are applied to a namespace that already works, and they break it silently if you enforce before you look.

Open this module in the interactive course to track progress.

Concepts

Default allow

With no NetworkPolicy, all pods accept all traffic. The first policy that selects a pod flips that pod to "deny everything not allowed", for the directions the policy names.

Policies add up

Each policy allows something. A connection is permitted if any policy allows it. There is no order and no deny rule.

Both ends must agree

A call from A to B needs A's egress rules and B's ingress rules to allow it. Replies are allowed automatically.

The network plugin enforces

A NetworkPolicy is only data. The network plugin enforces it (kindnet here). On a cluster whose plugin ignores policies, they apply without error and do nothing.

Pod Security Admission

Built into the API server. Labels on a namespace pick a level (privileged, baseline, restricted) for three modes: enforce rejects pods, warn tells the person applying, audit writes to the audit log.

What `restricted` asks of a pod

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• tenants/checkout/networkpolicy.yaml (reference, your repo): checkout-dev's network walls, in their final form.
  1. Selects every pod, allows nothing in L6–11
  2. From the traefik namespace, to the pod's port 9898, not the Service's 80 L29–42
  3. The way out: DNS first L47–63
• tenants/checkout/namespace.yaml (reference, your repo): Pod Security is three pairs of labels.
  1. The level, and the Kubernetes version whose definition of it applies L8–9
• apps/checkout/deployment.yaml (reference, your repo): What team-checkout changes to pass restricted.
  1. For the pod: a numeric non-root user, the default syscall filter L20–25
  2. For the container: no escalation, no capabilities L54–58

Steps

1. Before: a flat network

Call checkout from inside a catalog pod, by its cluster name <service>.<namespace>.

Expect "message": "checkout (dev)" (or your banner). Nothing stopped catalog from calling another team's app.

kubectl -n catalog-dev exec deployment/catalog -- wget -qO- -T 3 http://checkout.checkout-dev/api/info | grep message

2. Deny everything into checkout-dev

One policy that selects every pod (podSelector: {}) and allows no incoming traffic.

Create ~/platformengr/k8s-platform/tenants/checkout/networkpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: checkout-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress

cd ~/platformengr/k8s-platform && kubectl apply -f tenants/checkout/networkpolicy.yaml

The same call from catalog. Expect wget: download timed out: closed, as intended.

kubectl -n catalog-dev exec deployment/catalog -- wget -qO- -T 3 http://checkout.checkout-dev/api/info

And the front door. Expect 000 or 504 after five seconds: the ingress controller is just another pod in another namespace.

curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://checkout.localtest.me:8090/

Yet the pods stay 1/1: probes come from the kubelet on the pod's own node, which policies do not block.

kubectl -n checkout-dev get pods

3. Allow what is needed

Two more policies in the same file: pods may talk to pods in their own namespace, and the ingress controller may reach port 9898. Each allows something; together they are the allow list.

Add to ~/platformengr/k8s-platform/tenants/checkout/networkpolicy.yaml at the end of the file

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: checkout-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: checkout-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
      ports:
        - protocol: TCP
          port: 9898

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: deny incoming, allow same namespace and the ingress controller" && git push -q

kubectl apply -f tenants/checkout/networkpolicy.yaml

Expect 200:

curl -s -o /dev/null -w '%{http_code}\n' http://checkout.localtest.me:8090/

The call from catalog still times out:

kubectl -n catalog-dev exec deployment/catalog -- wget -qO- -T 3 http://checkout.checkout-dev/api/info

Three policies:

kubectl -n checkout-dev get networkpolicy

4. Write catalog's policies yourself

The same three policies for catalog-dev, in tenants/catalog/networkpolicy.yaml. Write it from checkout's file, changing only the namespace.

Create ~/platformengr/k8s-platform/tenants/catalog/networkpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: catalog-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: catalog-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: catalog-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
      ports:
        - protocol: TCP
          port: 9898

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "catalog-dev: deny incoming, allow same namespace and the ingress controller" && git push -q

kubectl apply -f tenants/catalog/networkpolicy.yaml

checkout to catalog: times out.

kubectl -n checkout-dev exec deployment/checkout -- wget -qO- -T 3 http://catalog.catalog-dev/api/info

catalog through the front door: 200.

curl -s -o /dev/null -w '%{http_code}\n' http://catalog.localtest.me:8090/

5. Now do Break it on purpose, then come back

Outgoing traffic is closed in Break it on purpose below. Do it now: the rest of the course expects both namespaces to have the restrict-egress policy its fix leaves behind. Then continue here with Pod Security.

6. Pod Security: look before you enforce

Ask the API server what would happen if checkout-dev enforced restricted, without changing anything.

Expect two warnings: existing pods in namespace "checkout-dev" violate the new PodSecurity enforce level "restricted:latest", and one that names the pods and four things they lack: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile.

kubectl label --dry-run=server --overwrite namespace checkout-dev pod-security.kubernetes.io/enforce=restricted

7. Platform team: warn first

warn and audit block nothing. They tell the team, every time it applies, what would be refused.

Add to ~/platformengr/k8s-platform/tenants/checkout/namespace.yaml under labels:, after team: checkout

    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.37
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.37

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: warn about Pod Security restricted" && git push -q

kubectl apply -f tenants/checkout/namespace.yaml

Send checkout's Deployment to the API server again as an update (a dry run: nothing changes). Read the Warning: line: the same four things, now addressed to whoever deploys.

kubectl replace --dry-run=server -f apps/checkout/deployment.yaml

8. team-checkout: fix the pod

runAsNonRoot needs a numeric user, or the kubelet cannot prove the user is not root. Ask the image who it runs as.

Expect uid=100(app) gid=101(app):

docker run --rm --entrypoint id ghcr.io/stefanprodan/podinfo:6.17.0

For the pod:

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the pod spec:, above serviceAccountName:

      securityContext:
        runAsNonRoot: true
        runAsUser: 100
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault

For the container. readOnlyRootFilesystem is not required by restricted; it costs nothing here, because podinfo writes no files.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, above resources:

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: restricted security context" && git push -q

Apply it. Expect no Warning: this time:

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

Expect 200:

curl -s -o /dev/null -w '%{http_code}\n' http://checkout.localtest.me:8090/

And the dry run from before prints no warning now:

kubectl label --dry-run=server --overwrite namespace checkout-dev pod-security.kubernetes.io/enforce=restricted

9. Platform team: enforce

Now that every pod passes, turn on enforce.

Add to ~/platformengr/k8s-platform/tenants/checkout/namespace.yaml under labels:, after team: checkout

    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.37

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: enforce Pod Security restricted" && git push -q

kubectl apply -f tenants/checkout/namespace.yaml

Try to start what an attacker, or a hurried engineer, would: a root shell. Expect Error from server (Forbidden): pods "root-shell" is forbidden: violates PodSecurity "restricted:v1.37", followed by the four things it lacks.

kubectl -n checkout-dev run root-shell --image=busybox:1.38.0 --restart=Never -- sleep 3600

10. Do the same for catalog, on your own

Warn, fix apps/catalog/deployment.yaml, check with the dry run, then enforce. The same labels and the same two security contexts as checkout.

Done when:

• tenants/catalog/namespace.yaml has the warn, audit and enforce labels for restricted, each with its -version label
• kubectl -n catalog-dev run root-shell --image=busybox:1.38.0 --restart=Never -- sleep 3600 is refused with violates PodSecurity
• curl -s -o /dev/null -w '%{http_code}\n' http://catalog.localtest.me:8090/ prints 200

    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.37
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.37
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.37

      securityContext:
        runAsNonRoot: true
        runAsUser: 100
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "catalog: Pod Security restricted" && git push -q

kubectl apply -f apps/catalog/deployment.yaml

kubectl -n catalog-dev rollout status deployment/catalog --timeout=120s

kubectl apply -f tenants/catalog/namespace.yaml

Simplified for the course

kube-system and traefik stay at the default (privileged). System components such as the network plugin need host access. A platform team sets baseline wherever it can and lists the exceptions.

Pod Security Admission knows three fixed levels. Rules of your own (allowed registries, required labels, no latest tag) need an admission policy engine: ValidatingAdmissionPolicy, Kyverno or OPA Gatekeeper.

NetworkPolicy works on pod and namespace labels and ports. Rules on DNS names or HTTP paths need a network plugin with its own policy type (Cilium, Calico) or a service mesh.

The *-version labels are pinned to the cluster's version, so an upgrade does not change the rules under running teams. Bumping them is part of the platform's upgrade checklist.

Break it on purpose: Locking the way out

After closing incoming traffic, the platform team closes outgoing traffic the same way: one default-deny policy, this time for Egress. It is the mirror image of the policy that worked an hour ago.

You practise: telling a name that does not resolve from a connection that is blocked, and building an allow list one error message at a time.

1. Break it

Add to ~/platformengr/k8s-platform/tenants/checkout/networkpolicy.yaml at the end of the file

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-egress
  namespace: checkout-dev
spec:
  podSelector: {}
  policyTypes:
    - Egress

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: deny egress" && git push -q

kubectl apply -f tenants/checkout/networkpolicy.yaml

2. What you see

The site still answers: curl gives 200, because replies to incoming requests are always allowed. Now let a checkout pod call its own Service, which worked a minute ago: kubectl -n checkout-dev exec deployment/checkout -- wget -qO- -T 3 http://checkout/healthz.

Expected: {"status": "OK"}. You get: wget: bad address 'checkout', after a pause. Read the error: it is not timed out. The pod did not get as far as connecting.

3. Fix it yourself

Make the call work again while egress stays denied by default. checkout may talk to DNS and to pods in its own namespace, nothing else. You will fix one thing and get a different error: that is progress. Done when the wget prints the status, and kubectl -n checkout-dev exec deployment/checkout -- wget -qO- -T 3 http://catalog.catalog-dev/healthz still fails. Then give catalog the same policy.

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - podSelector: {}

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: egress to DNS and the namespace" && git push -q

kubectl apply -f tenants/checkout/networkpolicy.yaml

kubectl -n checkout-dev exec deployment/checkout -- wget -qO- -T 3 http://checkout/healthz

kubectl -n checkout-dev exec deployment/checkout -- wget -qO- -T 3 http://catalog.catalog-dev/healthz

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-egress
  namespace: catalog-dev
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - podSelector: {}

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "catalog: egress to DNS and the namespace" && git push -q

kubectl apply -f tenants/catalog/networkpolicy.yaml

Quiz

1. A namespace has no NetworkPolicy. Which pods can connect to its pods?
   • Only pods in the same namespace, since a namespace is a network boundary
   • Every pod in the cluster, from any namespace
   • None of them, until a policy is written that allows a source
2. Pods in A may call pods in B only if what is true?
   • A's egress rules and B's ingress rules both allow it (or no policy selects that pod for that direction)
   • B has an ingress rule that allows A; egress rules only apply to traffic leaving the cluster
   • Either side allows it, since all policies add up across namespaces
3. A namespace enforces restricted. You apply a Deployment whose pods run as root. What happens?
   • The API server rejects the Deployment itself with a Forbidden error
   • The pods are created and then stopped by the kubelet when they start as root
   • The Deployment is accepted with a warning, and its ReplicaSet fails to create any pod

Field note

I turn on warn and audit months before enforce, and I do the same for network policy, in the sense that I write the policy next to a test that calls through it. The cluster was working before I arrived. Whatever I add has to prove it did not change that.

Report a problem with this module