Module 05 · Share the cluster · 50 min · You are: Platform team

Two teams, one cluster: namespaces and RBAC

Until now you were cluster-admin, like everyone on a cluster that grew without a plan. A namespace on its own separates names, not people: without RBAC every user can change every team's objects. The platform team's job is to hand out the smallest set of permissions that lets a team work, bound to its namespace, and to be able to show an auditor that the boundary holds. You will not take anyone's word for it, including your own YAML: you will ask the API server.

Open this module in the interactive course to track progress.

Concepts

Subject

Who is asking: a user, a group, or a ServiceAccount. Users and groups are not objects in Kubernetes; they are names that come from the login (a certificate, your company's single sign-on). ServiceAccounts are objects, and are for programs.

Role and ClusterRole

A list of rules: API group, resource, verbs. A Role lives in a namespace. A ClusterRole is cluster-wide and can also be reused inside namespaces.

RoleBinding and ClusterRoleBinding

A RoleBinding grants a Role, or a ClusterRole, inside its own namespace. A ClusterRoleBinding grants a ClusterRole everywhere. One word of difference decides whether a team sees its namespace or the whole cluster.

Additive only

There is no deny rule. A subject may do everything any of its bindings allows. You remove access by removing a binding.

Impersonation

An admin can ask "as someone else" with --as and --as-group. It is how you test permissions without having that person's login.

What each subject may do

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• tenants/checkout/rbac.yaml (reference, your repo): Who may do what in checkout-dev: five objects.
  1. A RoleBinding to the built-in ClusterRole view: read access, in this namespace only L7–15
  2. The deployer's rules: no secrets, no roles, no quotas, no network policies L5–44
  3. The workload gets no API credential L65
• tenants/catalog/rbac.yaml (reference, your repo): The same file for team-catalog.
  1. One name changed: the group L10–12

Steps

1. Onboard team-catalog: the platform half

A second team arrives. Copy checkout's tenant files inside your own repo, then change the names.

cd ~/platformengr/k8s-platform && mkdir -p tenants/catalog apps/catalog
cp tenants/checkout/namespace.yaml tenants/checkout/quota.yaml tenants/checkout/limits.yaml tenants/catalog/

In all three new files, change the namespace name. In namespace.yaml change the team label too:

Change in ~/platformengr/k8s-platform/tenants/catalog/namespace.yaml

-   name: checkout-dev
-   labels:
-     team: checkout
+   name: catalog-dev
+   labels:
+     team: catalog

Change in ~/platformengr/k8s-platform/tenants/catalog/quota.yaml

-   namespace: checkout-dev
+   namespace: catalog-dev

Change in ~/platformengr/k8s-platform/tenants/catalog/limits.yaml

-   namespace: checkout-dev
+   namespace: catalog-dev

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "Onboard team-catalog: namespace, quota, defaults" && git push -q

Apply the namespace first. kubectl reads a folder in alphabetical order, and limits.yaml would come before the namespace it lives in.

kubectl apply -f tenants/catalog/namespace.yaml

Then the whole folder (the namespace again shows unchanged):

kubectl apply -f tenants/catalog/

Two team namespaces. Expect checkout-dev and catalog-dev:

kubectl get namespaces -l team

2. Onboard team-catalog: the team half

team-catalog runs the same demo app. Copy checkout's five app files and turn them into catalog's.

cd ~/platformengr/k8s-platform && cp apps/checkout/deployment.yaml apps/checkout/service.yaml apps/checkout/ingress.yaml apps/checkout/configmap.yaml apps/checkout/pdb.yaml apps/catalog/

In each of the five files in apps/catalog/, replace every checkout with catalog (your editor's find and replace, whole folder). That covers names, labels, the namespace, the host catalog.localtest.me and the banner catalog (dev). Then three edits by hand:

catalog has no database, so remove the password from its Deployment:

Delete from ~/platformengr/k8s-platform/apps/catalog/deployment.yaml

-           env:
-             - name: DB_PASSWORD         # a reference to a Secret by name; the value is not here
-               valueFrom:
-                 secretKeyRef:
-                   name: catalog-db
-                   key: password

A new team starts its settings at version 1: in the ConfigMap, and in the two places the Deployment names it.

Change in ~/platformengr/k8s-platform/apps/catalog/configmap.yaml

-   name: catalog-settings-v2
+   name: catalog-settings-v1

Change in ~/platformengr/k8s-platform/apps/catalog/deployment.yaml
Both places: under envFrom: and under volumes:.

- catalog-settings-v2
+ catalog-settings-v1

If you kept the Black Friday banner in checkout, set catalog's to catalog (dev) now. Then save and apply:

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "Onboard team-catalog: the app" && git push -q

kubectl apply -f apps/catalog/

kubectl -n catalog-dev rollout status deployment/catalog --timeout=120s

Expect "message": "catalog (dev)":

curl -s http://catalog.localtest.me:8090/api/info | grep message

3. Who are you?

Before you hand out permissions, look at your own.

Expect Username kubernetes-admin and a group that ends in cluster-admins: the identity kind wrote into your kubeconfig.

kubectl auth whoami

May you do everything, everywhere? Expect yes:

kubectl auth can-i '*' '*' --all-namespaces

4. Before: a team member can do nothing

alice is a developer in team-checkout. She does not exist as an object: her name and group would come from her login. Ask the API server as her. Access starts at zero.

Expect no:

kubectl auth can-i list pods -n checkout-dev --as=alice --as-group=team-checkout

5. Grant: copy rbac.yaml for checkout

Five objects: a RoleBinding that gives group team-checkout the built-in ClusterRole view in this namespace; a ServiceAccount deployer for the pipeline, with a Role and a RoleBinding; and a ServiceAccount checkout for the workload, with no token. Read it in Quick look above before you apply it.

cd ~/platformengr/k8s-platform && cp ~/platformengr/kubernetes-platform/tenants/checkout/rbac.yaml tenants/checkout/

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: RBAC" && git push -q

kubectl apply -f tenants/checkout/rbac.yaml

6. Prove it, one question per line

Each question gets one answer, yes or no. The ones that must say no matter more than the ones that say yes.

alice may list pods in her namespace: yes.

kubectl auth can-i list pods -n checkout-dev --as=alice --as-group=team-checkout

She can read logs: yes.

kubectl auth can-i get pods --subresource=log -n checkout-dev --as=alice --as-group=team-checkout

She cannot read Secrets: no. view leaves Secrets out on purpose: reading one is reading the password.

kubectl auth can-i get secrets -n checkout-dev --as=alice --as-group=team-checkout

People read; the pipeline writes. no:

kubectl auth can-i delete deployments -n checkout-dev --as=alice --as-group=team-checkout

And nothing in the other team's namespace: no.

kubectl auth can-i list pods -n catalog-dev --as=alice --as-group=team-checkout

The deploy account may create Deployments here: yes.

kubectl auth can-i create deployments -n checkout-dev --as=system:serviceaccount:checkout-dev:deployer

A team cannot raise its own quota: no.

kubectl auth can-i update resourcequotas -n checkout-dev --as=system:serviceaccount:checkout-dev:deployer

Or give itself more rights: no.

kubectl auth can-i create rolebindings -n checkout-dev --as=system:serviceaccount:checkout-dev:deployer

Or deploy into another team's namespace: no.

kubectl auth can-i create deployments -n catalog-dev --as=system:serviceaccount:checkout-dev:deployer

The whole list for one subject, as a table of resources and verbs:

kubectl auth can-i --list -n checkout-dev --as=system:serviceaccount:checkout-dev:deployer

can-i asks. This one does it, and shows the error a team member would paste into a ticket. Read its four parts: who, verb, resource, namespace. Expect secrets is forbidden: User "alice" cannot list resource "secrets" in API group "" in the namespace "checkout-dev".

kubectl -n checkout-dev get secrets --as=alice --as-group=team-checkout

7. Take the API token out of the workload

Every pod gets a ServiceAccount token mounted by default. checkout never calls the Kubernetes API, so it should not hold a credential for it.

Before. Expect ca.crt, namespace and token:

kubectl -n checkout-dev exec deployment/checkout -- ls /var/run/secrets/kubernetes.io/serviceaccount

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the pod spec:, above topologySpreadConstraints:

      serviceAccountName: checkout

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: run as ServiceAccount checkout" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

After. Expect No such file or directory (and an exit code: the folder is gone):

kubectl -n checkout-dev exec deployment/checkout -- ls /var/run/secrets/kubernetes.io/serviceaccount

8. Write catalog's rbac.yaml yourself

The same five objects for catalog-dev, group team-catalog, workload ServiceAccount catalog. Try writing it from checkout's before you look at this one.

Create ~/platformengr/k8s-platform/tenants/catalog/rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-view
  namespace: catalog-dev
subjects:
  - kind: Group
    name: team-catalog
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: catalog-dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: catalog-dev
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployer
  namespace: catalog-dev
subjects:
  - kind: ServiceAccount
    name: deployer
    namespace: catalog-dev
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: catalog
  namespace: catalog-dev
automountServiceAccountToken: false

Add to ~/platformengr/k8s-platform/apps/catalog/deployment.yaml under the pod spec:, above topologySpreadConstraints:

      serviceAccountName: catalog

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "catalog-dev: RBAC, and catalog runs as its own ServiceAccount" && git push -q

kubectl apply -f tenants/catalog/rbac.yaml

kubectl apply -f apps/catalog/deployment.yaml

kubectl -n catalog-dev rollout status deployment/catalog --timeout=120s

Prove the mirror image with bob, in team-catalog. His own namespace: yes.

kubectl auth can-i list pods -n catalog-dev --as=bob --as-group=team-catalog

checkout's: no.

kubectl auth can-i list pods -n checkout-dev --as=bob --as-group=team-catalog

catalog's deploy account in checkout's namespace: no.

kubectl auth can-i create deployments -n checkout-dev --as=system:serviceaccount:catalog-dev:deployer

Simplified for the course

alice and bob are impersonated. Real users log in through your identity provider (OIDC), which supplies their name and groups; the bindings stay exactly the same.

deployer stands in for a pipeline that applies manifests. With Argo CD no pipeline holds cluster credentials at all: Argo CD applies, and its AppProjects decide what each team may deploy (Argo CD, module 09).

kubectl --token is not used to test the ServiceAccount: kind's kubeconfig logs in with a client certificate, which wins over a token, so you would be testing yourself. --as is the reliable test.

People get read-only access. Many teams also allow pods/exec and pods/portforward in dev; that is a decision to write down, not a default.

Break it on purpose: ClusterRole, so ClusterRoleBinding

A new platform engineer gives team-catalog's auditors read access. The role is the ClusterRole view, so they reach for a ClusterRoleBinding: the names match, the apply succeeds, and catalog can see catalog. The check they ran passes.

You practise: finding which binding grants a permission that should not exist, and removing access from a cluster that does not forget what you delete from Git.

1. Break it

Create ~/platformengr/k8s-platform/tenants/catalog/auditors.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: team-catalog-auditors
subjects:
  - kind: Group
    name: team-catalog
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "catalog: read access for auditors" && git push -q

kubectl apply -f tenants/catalog/auditors.yaml

The check they ran. yes, as intended:

kubectl auth can-i list pods -n catalog-dev --as=bob --as-group=team-catalog

2. What you see

Expected: kubectl auth can-i list configmaps -n checkout-dev --as=bob --as-group=team-catalog prints no. You get: yes.

Try -n kube-system and -n traefik: yes and yes. Nothing failed and nothing warned. Every check from earlier in this module still gives the expected answer, because none of them asked this question.

3. Fix it yourself

Find the object that gives team-catalog access outside its namespace, and remove that access from Git and from the cluster. bob must keep read access in catalog-dev. Done when the can-i lines for checkout-dev, kube-system and traefik print no and the one for catalog-dev prints yes.

cd ~/platformengr/k8s-platform && git rm -q tenants/catalog/auditors.yaml && git commit -qm "Remove cluster-wide view for catalog" && git push -q

kubectl delete clusterrolebinding team-catalog-auditors

kubectl auth can-i list configmaps -n checkout-dev --as=bob --as-group=team-catalog

kubectl auth can-i list configmaps -n kube-system --as=bob --as-group=team-catalog

kubectl auth can-i list configmaps -n traefik --as=bob --as-group=team-catalog

kubectl auth can-i list configmaps -n catalog-dev --as=bob --as-group=team-catalog

Quiz

1. A RoleBinding in checkout-dev refers to the ClusterRole view. Where does it grant read access?
   • In every namespace, because a ClusterRole is cluster-wide wherever it is bound
   • In checkout-dev only: a RoleBinding never reaches beyond its own namespace
   • Nowhere: a RoleBinding can only refer to a Role, so the API server rejects it
2. How do you take a permission away from a group?
   • Add a Role with the verb deny for that resource and bind it to the group
   • Bind the group to the ClusterRole view, which overrides any wider roles it has
   • Remove the binding, or the rule in the role, that grants it: RBAC has no deny rules
3. Why give the checkout pods a ServiceAccount with automountServiceAccountToken: false?
   • So the containers hold no credential for the Kubernetes API that an attacker inside them could use
   • So the pods start faster, since mounting the token delays every container on start
   • So the Deployment can be applied by deployer, which is not allowed to create tokens

Field note

Every RBAC review I have done found the problem in the tests that were missing, not in the roles that were written. People check that the right person gets in. Write down the three things that must be refused and check those.

Report a problem with this module