Module 03 · Keep it healthy · 45 min · You are: Platform and product

Sizing: requests, limits, and a quota per team

On a shared cluster, a pod with no requests is invisible to the scheduler, and a pod with no memory limit can take a node down with it. One team's mistake becomes another team's outage. Requests and limits are the team's promise about one container; a quota is the platform's cap on the sum. A quota also changes how failure looks: a pod that does not fit the quota is not Pending. It does not exist, and you have to know where to look.

Open this module in the interactive course to track progress.

Concepts

Request

What the scheduler reserves on a node for the container. A pod is placed only where the sum of requests fits. Nothing checks that the container really uses it.

Limit

What the node enforces. Over the CPU limit, the container is slowed down. Over the memory limit, it is killed (OOMKilled, exit code 137).

QoS class

Derived from requests and limits: Guaranteed (requests equal limits), Burstable, BestEffort (none set). When a node runs out of memory, BestEffort pods go first.

ResourceQuota

A cap on the total of a namespace: requested CPU and memory, memory limits, number of pods, Services of a kind. Checked when a pod is created, not later.

LimitRange

Per-container defaults and bounds in a namespace. A container with no request gets the default; one above max is refused.

Not running: three different places

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• apps/checkout/deployment.yaml (reference, your repo): The container's promise.
  1. Requests for the scheduler, a memory limit for the node, and no CPU limit: see the first step L59–64
• tenants/checkout/quota.yaml (reference, your repo): The platform team's cap on the whole namespace.
  1. Sums for the namespace; the last two stop a team opening ports on the nodes L9–15
• tenants/checkout/limits.yaml (reference, your repo): Defaults for containers that state nothing.
  1. Filled in when a container has no request L10–12
  2. Filled in when a container has no limit L13–14
  3. A container asking for more is refused L15–16
• labs/memhog.yaml (reference, your repo): A lab pod you apply from the reference clone and never commit.
  1. Loads 200 MB into memory at start L12
  2. With a 64Mi limit L17–18

Steps

1. Before: nobody promised anything

checkout states no resources. Look at what that means to Kubernetes.

The pod's QoS class. Expect BestEffort: the first to go when a node runs out of memory.

kubectl -n checkout-dev get pod -l app.kubernetes.io/name=checkout -o jsonpath='{.items[0].status.qosClass}'; echo

What the scheduler thinks k8s-worker has given away. Find the cpu and memory rows under Requests: checkout adds nothing to them.

kubectl describe node k8s-worker | grep -A7 'Allocated resources'

2. team-checkout sizes its container

100m is a tenth of a CPU core. There is a memory limit and no CPU limit, on purpose. Memory cannot be taken back from a process, so a hard wall is the only protection. CPU can: a container without a CPU limit may use idle CPU, and under pressure it is held to its share by its request. Many platform teams therefore require memory limits and CPU requests, and leave CPU limits off. Know the rule your platform chose.

Add to ~/platformengr/k8s-platform/apps/checkout/deployment.yaml under the container, after the volumeMounts: block (same depth as volumeMounts:)

          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              memory: 128Mi

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: requests and a memory limit" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

Expect Burstable now: it has requests, and its limits are not equal to them.

kubectl -n checkout-dev get pod -l app.kubernetes.io/name=checkout -o jsonpath='{.items[0].status.qosClass}'; echo

3. Watch a memory limit do its job

A lab pod asks for 200 MB with a 64Mi limit. It comes from the reference clone and never goes in your repo.

kubectl apply -f ~/platformengr/kubernetes-platform/labs/memhog.yaml

Watch STATUS go Running, OOMKilled, CrashLoopBackOff, with RESTARTS climbing. Press Ctrl+C after two restarts.

kubectl -n checkout-dev get pod memhog -w

The proof is under Last State: Reason: OOMKilled, Exit Code: 137. The logs show nothing about it: the kernel killed the process, it did not get to say goodbye.

kubectl -n checkout-dev describe pod memhog | grep -A4 'Last State'

kubectl -n checkout-dev delete pod memhog

4. Platform team: a quota for the namespace

Now you are the platform team. The quota caps the sum of what team-checkout's pods may ask for. Copy it from the reference and read it.

cd ~/platformengr/k8s-platform && cp ~/platformengr/kubernetes-platform/tenants/checkout/quota.yaml tenants/checkout/

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: quota" && git push -q

kubectl apply -f tenants/checkout/quota.yaml

Used and Hard, side by side. Expect requests.cpu 200m 500m and pods 2 8: the two pods that already run are counted.

kubectl -n checkout-dev describe resourcequota team-quota

5. Platform team: defaults, written by you

A quota on requests.cpu has a side effect: from now on a pod that states no CPU request is refused outright. A LimitRange turns that refusal into a default.

Create ~/platformengr/k8s-platform/tenants/checkout/limits.yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: checkout-dev
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 50m
        memory: 64Mi
      default:
        memory: 128Mi
      max:
        memory: 512Mi

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout-dev: container defaults" && git push -q

kubectl apply -f tenants/checkout/limits.yaml

Start a pod with no resources at all:

kubectl -n checkout-dev run defaults-demo --image=ghcr.io/stefanprodan/podinfo:6.17.0 --restart=Never

Expect requests of 50m and 64Mi and a 128Mi limit, filled in at admission:

kubectl -n checkout-dev get pod defaults-demo -o jsonpath='{.spec.containers[0].resources}'; echo

kubectl -n checkout-dev delete pod defaults-demo

6. Hit the quota

team-checkout scales to 6. Six pods at 100m is 600m; the quota is 500m.

Change in ~/platformengr/k8s-platform/apps/checkout/deployment.yaml

-   replicas: 2
+   replicas: 6

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: scale to 6" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

Expect 5/6 under READY:

kubectl -n checkout-dev get deployment checkout

Now look for the sixth pod. Expect five rows and nothing Pending: the pod was refused before it existed.

kubectl -n checkout-dev get pods

The refusal is recorded on the object that tried to create it, the ReplicaSet. Read the message: what was requested, what is used, what the limit is. Expect exceeded quota: team-quota and, for each resource the quota caps, three lists: requested, used and limited. limits.memory comes first; find requests.cpu in each list: requested: ...requests.cpu=100m, used: ...requests.cpu=500m, limited: ...requests.cpu=500m. CPU is what ran out.

kubectl -n checkout-dev get events --field-selector reason=FailedCreate | tail -n 2

Undo the scale-up with a revert, and apply:

cd ~/platformengr/k8s-platform && git revert --no-edit HEAD && git push -q

kubectl apply -f apps/checkout/deployment.yaml

Expect 2/2:

kubectl -n checkout-dev get deployment checkout

7. Compare: a pod that exists and cannot be placed

Another lab pod asks for 64 CPU cores. No quota applies in the default namespace, so it is created. Then the scheduler has to find a node.

kubectl apply -f ~/platformengr/kubernetes-platform/labs/too-big.yaml

Expect Pending, for as long as you wait:

kubectl -n default get pod too-big

This time the pod exists and the scheduler explains itself in the pod's own events. Expect 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu, followed by a note about preemption.

kubectl -n default describe pod too-big | tail -n 3

kubectl -n default delete pod too-big

Simplified for the course

There is no metrics-server on this cluster, so kubectl top does not work and the numbers are guesses. In production you set requests from measured use (metrics-server, Prometheus, the Vertical Pod Autoscaler in recommendation mode) and review them.

One quota per namespace, set by hand. With many teams the platform generates quotas from a size (small, medium) in one file per team: see Fleet GitOps.

Quotas cap what teams ask for, not what the nodes have. A cluster autoscaler adds nodes; that needs a cloud.

Break it on purpose: The quota was exactly right

The platform team reviews quotas and sees checkout using 200m of 500m. To stop teams hoarding, an engineer sets requests.cpu to what is in use: 200m. Everything keeps running. A week later team-checkout raises its memory limit to 192Mi, a harmless change.

You practise: reading a stuck rollout from the ReplicaSets down, and finding a cause that is not in the file that was changed.

1. Break it

As the platform team:

Change in ~/platformengr/k8s-platform/tenants/checkout/quota.yaml

-     requests.cpu: 500m
+     requests.cpu: 200m

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "Right-size checkout quota" && git push -q

kubectl apply -f tenants/checkout/quota.yaml

A week later, as team-checkout:

Change in ~/platformengr/k8s-platform/apps/checkout/deployment.yaml

-               memory: 128Mi
+               memory: 192Mi

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "checkout: memory limit 192Mi" && git push -q

kubectl apply -f apps/checkout/deployment.yaml

kubectl -n checkout-dev rollout status deployment/checkout --timeout=60s

2. What you see

Expected: deployment "checkout" successfully rolled out. You get: Waiting for deployment "checkout" rollout to finish: 0 out of 2 new replicas have been updated... and then error: timed out waiting for the condition.

The site still answers 200: the old pods are untouched, with the old limit. kubectl -n checkout-dev get pods shows two Running pods and nothing else. Look at kubectl -n checkout-dev get replicaset: one of them wants a pod it does not have.

3. Fix it yourself

Get the new limit running without taking checkout down and without deleting the quota. Decide whose change it is. Done when rollout status succeeds and kubectl -n checkout-dev get pod -l app.kubernetes.io/name=checkout -o jsonpath='{.items[*].spec.containers[0].resources.limits.memory}' prints 192Mi 192Mi.

-     requests.cpu: 200m
+     requests.cpu: 500m

cd ~/platformengr/k8s-platform && git add -A && git commit -qm "Quota: leave room for a rollout" && git push -q

kubectl apply -f tenants/checkout/quota.yaml

kubectl -n checkout-dev rollout restart deployment/checkout

kubectl -n checkout-dev rollout status deployment/checkout --timeout=120s

Quiz

1. A Deployment asks for 6 replicas and the namespace quota has room for 5. What do you see?
   • Five running pods and a sixth in Pending until some quota is freed
   • Five running pods and no sixth pod at all; the reason is in the ReplicaSet's events
   • Six pods, with the sixth restarted by the kubelet each time it goes over quota
2. What happens to a container that uses more memory than its limit?
   • It is slowed down until its memory use drops below the limit again
   • The pod is moved by the scheduler to a node with more free memory
   • The kernel kills it and the kubelet restarts it; the pod shows OOMKilled with exit code 137
3. What is a request used for?
   • The scheduler uses it to decide which node has room for the pod
   • The kubelet uses it as the most the container may ever use
   • The quota controller uses it to restart pods that use less than they asked for

Field note

Teams argue about the numbers. The numbers matter less than having them. A cluster where every container states a request and a memory limit, even rough ones, behaves predictably. One where half do not is a lottery, and the team that loses is rarely the one that left them out.

Report a problem with this module