Cheat notes · Kubernetes

Kubernetes: cheat notes

Every module on one page, cut to what you need to remember: the idea, the commands, the pitfalls and a few questions to test yourself. Made for review after you have done the course, not instead of it. Back to the course.

00 Orientation: what a platform team owns in Kubernetes

This module only · Full module

The idea

Kubernetes stores what you declare and runs controllers that make the cluster match it. A platform team shares one cluster between teams, so every object has an owner: the platform team owns the namespace and the rules around a team, the team owns what runs inside.

Remember

Declare, then reconcile: kubectl apply sends an object to the API server, which stores it in etcd. Controllers and kubelets then make it true.
The API server is the only door. kubectl, controllers, the scheduler and every kubelet talk to it and to nothing else.
Namespace: the unit a platform team hands to a product team. Quotas, permissions and network rules attach to it.
Two folders, two owners: tenants/<team>/ (platform team), apps/<team>/ (product team). Kubernetes ignores folders; people do not.
A kind node is a Docker container: k8s-control-plane, k8s-worker, k8s-worker2. The kube context is kind-k8s.
Every namespace gets a label for free: kubernetes.io/metadata.name=<name>, which network rules select on.

Pitfalls

A node goes NotReady with no error anywhere: only a status. You see it only if you look, or if an alert does.
Changing the cluster from the wrong context. Check kubectl config current-context before every change.

Commands

`kubectl config current-context`	Which cluster you are about to change
`kubectl get nodes`	Nodes and their status
`kubectl get pods -n kube-system -o wide`	The cluster's own parts, with their nodes
`kubectl apply --dry-run=server -f FILE`	Would the API server accept it? Changes nothing
`kubectl get namespace NS --show-labels`	A namespace and its labels

tenants/checkout/namespace.yaml (module 00)

apiVersion: v1
kind: Namespace
metadata:
  name: checkout-dev
  labels:
    team: checkout

Test yourself

Answer in your head first, then open the answer.

What does kubectl apply -f namespace.yaml actually do?

It sends the object to the API server, which stores it; controllers then make the cluster match.

A node shows NotReady. What does that tell you?

The control plane stopped hearing from that node's kubelet. It says nothing yet about the pods.

Why are namespaces under tenants/ and Deployments under apps/?

Different owners: the platform team sets the rules around a team, the team decides what runs inside.

Where module 00 leaves you: A cluster kind-k8s with three Ready nodes. Your repo on GitHub holds cluster/kind.yaml and tenants/checkout/namespace.yaml. The namespace checkout-dev exists and is empty.

01 One app, three objects: Deployment, Service, Ingress

This module only · Full module

The idea

A Deployment keeps N identical pods running through a ReplicaSet per version; a Service finds Ready pods by label and gives them one name; an Ingress routes a hostname to a Service, through an ingress controller. Labels are the only glue, and nothing checks them.

Remember

One ReplicaSet per pod template version. A rollout scales the new one up and the old one down; the old one stays at 0 for a rollback.
Service selector: matches pod labels; the matching Ready pods are its endpoints (EndpointSlice).
targetPort: http names the container port, so the Service follows the pod if the number changes.
An Ingress is only data until the ingress controller with its ingressClassName reads it. Traefik runs on the control-plane node.
Always set metadata.namespace in the file, or it lands wherever your context points.
***.localtest.me points at 127.0.0.1;** kind maps port 8090 to Traefik's NodePort 30080.

Pitfalls

A Service selector that matches no pod: every object is green, the site answers 503. Look at the endpoints.
A Deployment's selector cannot be changed after creation. Relabel by adding the new label to the template first, then switching the Service.
kubectl apply never deletes what you removed from your files.

Commands

`kubectl -n NS rollout status deployment/NAME`	Wait for a rollout
`kubectl -n NS get deployment,replicaset,pod -o wide`	The whole chain, with nodes
`kubectl -n NS get endpointslices -l kubernetes.io/service-name=SVC`	Which pods the Service sends to
`kubectl -n NS port-forward service/SVC 9898:80`	A private tunnel, for you only
`kubectl diff -f FOLDER`	What an apply would change

apps/checkout/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: checkout-dev
spec:
  selector:
    app.kubernetes.io/name: checkout
  ports:
    - name: http
      port: 80
      targetPort: http

Test yourself

Answer in your head first, then open the answer.

You change the image and apply. What does Kubernetes create?

A new ReplicaSet for the new pod template, scaled up while the old one is scaled down.

A Service has no endpoints but its pods are Running and Ready. Most likely cause?

Its selector does not match the pods' labels.

Who makes an Ingress do something?

The ingress controller that matches its ingressClassName.

Why does the site answer 503 instead of 404 when a Service has no pods?

Traefik is configured with allowEmptyServices: the route exists, it has nowhere to send the request.

Where module 01 leaves you: checkout runs in checkout-dev with 2 pods on image 6.17.0, and answers at http://checkout.localtest.me:8090. Your repo holds cluster/traefik.yaml and three files in apps/checkout/.

02 Configuration and secrets, and where the line is

This module only · Full module

The idea

A ConfigMap gives a pod settings as environment variables (read once at start) or files (updated in place). A Secret is a ConfigMap with base64 and its own permissions, not encryption, so its value never goes in Git, and a rebuilt cluster needs a list of what Git does not hold.

Remember

Env is read once, files follow: changing a ConfigMap does not change a running container's environment.
Versioned, immutable ConfigMaps: checkout-settings-v2 with immutable: true. A config change is a new name, a rollout, and a clean revert.
base64 is an encoding. Anyone who may get the Secret reads the value.
Git history keeps everything. A pushed Secret means a new password, not a delete.
A missing Secret leaves new pods in CreateContainerConfigError; the old pods keep serving.
docs/secrets.md lists each Secret, its keys, its owner and where the value lives.

Pitfalls

Editing a ConfigMap in place and waiting for the change: pods only restart when the pod template changes.
"It is only encoded": a Secret in Git is readable by everyone who can read the repo, for ever.
Forgetting to delete the old ConfigMap after renaming: kubectl apply leaves it behind.

Commands

`kubectl -n NS exec deployment/NAME -- printenv VAR`	What the container's environment says
`kubectl -n NS create secret generic NAME --from-literal=KEY=VALUE`	A Secret typed in, not committed
`kubectl -n NS get secret NAME -o jsonpath='{.data.KEY}' \| base64 -d`	Read a Secret's value back
`git show HEAD~1:PATH`	A file as it was one commit ago

apps/checkout/deployment.yaml (container, excerpt)

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: checkout-db
        key: password
envFrom:
  - configMapRef:
      name: checkout-settings-v2

Test yourself

Answer in your head first, then open the answer.

You change a ConfigMap used through envFrom. What happens to running pods?

Nothing: they keep the old values until they are replaced.

What protects a Secret's value?

Access control on secrets and optional encryption at rest. The value itself is only base64.

A Secret was pushed, then deleted in the next commit. What must happen?

Change the credential: the old value stays in the history.

How do you make a config change roll out from Git?

Give the ConfigMap a new versioned name, make it immutable, and point the Deployment at it.

Where module 02 leaves you: checkout reads its banner from an immutable ConfigMap checkout-settings-v2 and a password from the Secret checkout-db. The Secret exists only in the cluster; your repo has no Secret and has docs/secrets.md saying so.

03 Sizing: requests, limits, and a quota per team

This module only · Full module

The idea

A request is what the scheduler reserves; a limit is what the node enforces. A ResourceQuota caps the sum for a namespace at admission, and a LimitRange fills in defaults. A pod refused by a quota never exists, so the reason is on the ReplicaSet, not on a pod.

Remember

Request: used to place the pod. Memory limit: over it the container is killed (OOMKilled, exit 137). CPU limit: throttles; many platforms leave it off.
QoS: BestEffort (nothing set) goes first under memory pressure, then Burstable, then Guaranteed.
A quota on requests.cpu refuses any pod that states no CPU request, unless a LimitRange gives a default.
Three places for "not running": no pod (ReplicaSet events, FailedCreate), Pending (pod events, FailedScheduling), restarts (Last State).
Size quotas for peak plus surge: a rolling update needs room for one extra pod.
checkout: requests 100m and 64Mi, limit 192Mi; quota 500m CPU, 8 pods, no NodePorts.

Pitfalls

Looking for a Pending pod when a quota said no: there is no pod. Read the ReplicaSet's FailedCreate events.
A quota set to exactly what is used: the next rollout cannot create its surge pod and times out, while the site stays up.
Expecting logs from an OOMKilled container: the kernel killed it, it printed nothing.

Commands

`kubectl -n NS describe resourcequota team-quota`	Used and hard, side by side
`kubectl -n NS get events --field-selector reason=FailedCreate`	Pods refused at admission
`kubectl -n NS describe pod POD \| grep -A4 'Last State'`	Why the last container died
`kubectl describe node NODE \| grep -A7 'Allocated resources'`	What the scheduler has given away on a node

tenants/checkout/quota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: checkout-dev
spec:
  hard:
    requests.cpu: 500m
    requests.memory: 512Mi
    limits.memory: 1Gi
    pods: "8"
    services.loadbalancers: "0"
    services.nodeports: "0"

Test yourself

Answer in your head first, then open the answer.

A Deployment asks for 6 replicas, the quota has room for 5. What do you see?

Five running pods and no sixth at all; the reason is in the ReplicaSet's events.

What happens to a container over its memory limit?

The kernel kills it and the kubelet restarts it: OOMKilled, exit code 137.

What is a request used for?

Placing the pod: the scheduler puts it only where the sum of requests fits.

Why did the rollout hang after the quota was right-sized?

A rolling update needs room for one extra pod, and the quota had none.

Where module 03 leaves you: Each checkout pod requests 100m CPU and 64Mi memory, with a 192Mi memory limit. The namespace has a ResourceQuota team-quota (500m CPU requested, 8 pods) and a LimitRange container-defaults. kubectl describe quota shows 200m of 500m used.

04 Probes and disruption budgets: staying up while things change

This module only · Full module

The idea

Readiness decides traffic, liveness decides restarts, startup gives time to start. A PodDisruptionBudget tells drains how many pods may be away at once. Written well they keep the app up through every node upgrade; written badly they kill a healthy app or block the upgrade.

Remember

Readiness fails: out of the Service's endpoints, not restarted.
Liveness fails failureThreshold times: the kubelet kills and restarts the container (same pod, RESTARTS +1).
Startup: holds the other two until it passes; failureThreshold × periodSeconds is the start budget.
Liveness checks the process only, never a database or another service.
kubectl drain evicts through the eviction API, which respects budgets. A dying node is involuntary: no budget helps.
maxUnavailable: 1 lets a drain through; minAvailable equal to the replica count blocks it for ever.
Spread: topologySpreadConstraints on kubernetes.io/hostname. Nothing moves pods back after a node returns.

Pitfalls

minAvailable: 2 with 2 replicas: every eviction fails and the drain retries without end.
Getting a stuck drain through with kubectl delete pod: it bypasses the budget, the outage the budget was meant to stop.
A long initialDelaySeconds on liveness instead of a startup probe.

Commands

`kubectl drain NODE --ignore-daemonsets --delete-emptydir-data`	Evict a node's pods, respecting budgets
`kubectl uncordon NODE`	Schedulable again
`kubectl -n NS get poddisruptionbudget`	ALLOWED DISRUPTIONS
`kubectl -n NS rollout restart deployment/NAME`	Replace every pod (respreads them)

apps/checkout/pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
  namespace: checkout-dev
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout

Test yourself

Answer in your head first, then open the answer.

A pod's readiness probe fails. What does Kubernetes do?

It stops sending the Service's traffic to that pod and leaves it running.

Why should liveness not check the database?

A slow database would restart every pod at once, turning a slowdown into an outage.

2 replicas and minAvailable: 2: what happens on a drain?

The eviction is refused and the drain retries without end.

Who changes the budget when a drain hangs?

The team that owns it; the platform team calls them rather than editing it.

Where module 04 leaves you: checkout has startup, readiness and liveness probes, a spread constraint and a PodDisruptionBudget checkout with maxUnavailable: 1. Both workers are schedulable again and the pods are on different nodes.

05 Two teams, one cluster: namespaces and RBAC

This module only · Full module

The idea

RBAC is additive: a subject may do whatever any of its bindings allows. A RoleBinding grants a role inside its own namespace, even a ClusterRole; a ClusterRoleBinding grants it everywhere. Prove the boundary by asking the API server with kubectl auth can-i --as, especially the questions that must say no.

Remember

Subjects: users and groups come from the login and are not objects; ServiceAccounts are objects, for programs.
The binding sets the scope, not the role. view bound with a RoleBinding stops at the namespace.
No deny rules. Remove access by removing a binding.
The kit per team: RoleBinding team-view (group, ClusterRole view), ServiceAccount deployer with a Role, workload ServiceAccount with automountServiceAccountToken: false.
view leaves out Secrets on purpose.
The deployer may not touch Secrets, roles, bindings, quotas, limits or network policies.
kubectl apply does not delete: removing a binding from Git leaves it in the cluster until you delete it.

Pitfalls

"ClusterRole, so ClusterRoleBinding": the team can now read every namespace. Nothing warns.
Testing only that the right people get in. Over-permission never produces an error; ask the no questions too.
Testing a ServiceAccount with --token on kind: the client certificate wins, so you test yourself. Use --as.

Commands

`kubectl auth whoami`	Who the API server thinks you are
`kubectl auth can-i list pods -n NS --as=alice --as-group=team-checkout`	Ask as someone else
`kubectl auth can-i create deployments -n NS --as=system:serviceaccount:NS:deployer`	Ask as a ServiceAccount
`kubectl auth can-i --list -n NS --as=SUBJECT`	Everything a subject may do
`kubectl get clusterrolebindings -o wide`	Cluster-wide grants and their subjects

tenants/checkout/rbac.yaml (first object)

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-view
  namespace: checkout-dev
subjects:
  - kind: Group
    name: team-checkout
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

Test yourself

Answer in your head first, then open the answer.

A RoleBinding in checkout-dev refers to ClusterRole view. Where does it grant access?

In checkout-dev only.

How do you take a permission away?

Remove the binding or the rule that grants it; RBAC has no deny.

Why automountServiceAccountToken: false on the workload?

So the container holds no API credential an attacker could use.

You deleted a ClusterRoleBinding's file and the access is still there. Why?

kubectl apply never deletes; delete the object too.

Where module 05 leaves you: Two tenants, checkout-dev and catalog-dev, each with a quota, defaults, a running app and an rbac.yaml. Members of team-checkout can read checkout-dev and nothing in catalog-dev, and the reverse. Neither team can read Secrets. The checkout and catalog pods carry no API token.

06 Walls: NetworkPolicy and Pod Security

This module only · Full module

The idea

The pod network is flat until a NetworkPolicy selects a pod; then only what some policy allows gets through, and both the caller's egress and the callee's ingress must allow it. Pod Security Admission checks pods against a level set by namespace labels: warn and audit first, enforce once the pods pass.

Remember

The first policy changes the default: a pod no policy selects accepts all traffic. Once one policy selects it, it accepts only what some policy allows, in the directions (ingress, egress) that policy names.
Policies add up, with no order and no deny rule. Replies are always allowed.
The front door is a pod too: allow the traefik namespace to the pod port 9898, not the Service's 80.
Deny egress breaks DNS first: allow CoreDNS (k8s-app: kube-dns in kube-system, port 53 UDP and TCP).
One dash matters: namespaceSelector and podSelector in one list item means both; two items means either.
restricted needs: runAsNonRoot with a numeric user, allowPrivilegeEscalation: false, capabilities.drop: [ALL], seccompProfile: RuntimeDefault.
enforce rejects pods, not Deployments. Workloads only get a warning; their ReplicaSet then fails to create pods.

Pitfalls

bad address is DNS, download timed out is a blocked connection: different fixes.
Enforcing Pod Security before looking: dry-run the label first, then warn, then fix, then enforce.
Default deny ingress also closes the front door: the ingress controller needs its own allow rule.

Commands

`kubectl -n NS exec deployment/NAME -- wget -qO- -T 3 http://SVC.NS/healthz`	Test a call from inside a pod
`kubectl -n NS get networkpolicy`	The policies in a namespace
`kubectl label --dry-run=server --overwrite namespace NS pod-security.kubernetes.io/enforce=restricted`	Which pods would fail, without changing anything
`kubectl replace --dry-run=server -f FILE`	Run admission on an update, and see its warnings

tenants/checkout/networkpolicy.yaml (restrict-egress)

spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - podSelector: {}

Test yourself

Answer in your head first, then open the answer.

A namespace has no NetworkPolicy. Who can connect to its pods?

Every pod in the cluster.

When may pods in A call pods in B?

When A's egress and B's ingress both allow it (or no policy selects that pod for that direction).

A namespace enforces restricted; you apply a Deployment running as root. What happens?

The Deployment is accepted with a warning; its ReplicaSet cannot create pods.

Why do probes keep working after default deny?

They come from the kubelet on the pod's own node, which policies do not block.

Where module 06 leaves you: Both team namespaces accept traffic only from their own pods and from the ingress controller, and may only call DNS and their own pods. Both enforce Pod Security restricted. checkout and catalog run as a non-root user with no privileges, and still answer through the ingress.

07 Debugging labs: five ways a pod does not run

This module only · Full module

The idea

A pod that is not running is stuck at one stage: not created, not scheduled, not started, crashing, or killed from outside. Say the stage first, then read the one place that explains it.

Remember

No pod at all: admission refused it (quota, LimitRange, Pod Security). Read the ReplicaSet's FailedCreate events.
Pending: the scheduler. Read the pod's events (FailedScheduling).
ErrImagePull, CreateContainerConfigError: the registry or the kubelet. Pod events.
CrashLoopBackOff: the process exited. kubectl logs --previous and Last State.
Restarts with clean logs: something stopped it, usually liveness. Unhealthy and Killing events.
Events last about an hour. Look while it is failing.
Old pods keep serving during a failed rollout, which is why these are found late.

Pitfalls

Reading kubectl logs of a container that just restarted: the failed one is --previous.
Missing the Warning: line that kubectl apply printed: Pod Security told you before the rollout hung.
Deleting a file from Git and forgetting the object: delete both.

Commands

`kubectl -n NS get events --field-selector reason=FailedCreate`	Refused at admission
`kubectl -n NS describe pod POD`	Events, state, last state
`kubectl -n NS logs POD --previous`	Logs of the container that died
`kubectl -n NS get events --field-selector reason=Unhealthy`	Probe failures
`kubectl -n NS debug -it POD --image=busybox:1.38.0 --target=checkout --profile=restricted`	A temporary shell next to a container

Test yourself

Answer in your head first, then open the answer.

2/3 and no third pod in any state: where is the reason?

In the ReplicaSet's events: admission refused the pod.

CrashLoopBackOff and kubectl logs shows a clean start. Next?

kubectl logs --previous and the Last State in describe pod.

Pods restart every half minute with clean shutdowns after a successful rollout. Why?

The liveness probe fails; the kubelet kills them. Look at Unhealthy events.

ImagePullBackOff, message ends in not found. Cause?

The tag does not exist. unauthorized means credentials; no such host means the registry name.

Where module 07 leaves you: checkout is healthy again: 2 pods, the probes and image from module 06, nothing extra. Your repo has docs/runbook.md with one real entry.

08 Capstone: a third team, on your own

This module only · Full module

The idea

Onboarding a team is the same kit every time, in an order that lets you test each layer: namespace, quota and defaults, RBAC, the app, network policies last. An internal service has no Ingress and one named caller, and the caller's egress must allow the call too.

Remember

Order: namespace, quota and LimitRange, RBAC, app, network policies, with a check after each.
Internal only: a ClusterIP Service, no Ingress, ingress allowed only from the caller's pods.
One list item, both selectors: namespaceSelector for checkout-dev and podSelector for app.kubernetes.io/name: checkout.
Both ends: checkout's restrict-egress needs a rule to payments on 9898.
Quota for 3 replicas at 100m: room for 4 pods during a rollout, so 400m to 600m.
A budget that allows a drain: maxUnavailable: 1, never minAvailable equal to the replica count.
docs/onboarding.md: each item with a command and the answer you expect, including some no.

Pitfalls

Writing payments' ingress rule and forgetting checkout's egress: the call times out.
Two dashes instead of one in from:: either selector would match, which lets in all of checkout-dev or every checkout pod anywhere.
Applying the tenant folder before the namespace: limits.yaml comes first alphabetically and fails.

Commands

`kubectl apply -f tenants/TEAM/namespace.yaml`	The namespace before its folder
`kubectl label --dry-run=server --overwrite namespace NS pod-security.kubernetes.io/enforce=restricted`	Ready for enforce?
`kubectl -n NS exec deployment/NAME -- ls /var/run/secrets/kubernetes.io/serviceaccount`	No token mounted?
`diff -r REF/tenants/payments tenants/payments`	Compare with the reference

tenants/payments/networkpolicy.yaml (allow-from-checkout)

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: checkout-dev
        podSelector:
          matchLabels:
            app.kubernetes.io/name: checkout
    ports:
      - protocol: TCP
        port: 9898

Test yourself

Answer in your head first, then open the answer.

payments allows checkout in, but the call from checkout times out. First check?

checkout's egress rules.

Which budget lets a node drain with 3 replicas?

maxUnavailable: 1.

In what order do you build a tenant?

Namespace, quota and defaults, RBAC, app, network policies, testing after each.

Where module 08 leaves you: A third tenant, payments-dev, runs payments with 3 replicas, its own quota, RBAC, policies and Pod Security. checkout can call it; catalog and the ingress controller cannot. Your repo has docs/onboarding.md. The cluster is deleted; your repo keeps everything.