The whole cluster runs on one laptop with 12 GB of RAM. Three VMs get 10 GB of it, and the hypervisor keeps the rest. Every memory decision in this series has come back to that number.
I wanted this post to be about measurements, not a tour of Kubernetes resource settings. So I pulled the live numbers from the cluster this morning (19 November, with all the apps from part 13 running) and compared three things on each node: what the pods ask for, what they’re allowed, and what they actually use.
The comparison turned up something I hadn’t checked before, and it was fixed the same afternoon. That’s at the end, with the numbers from before and after.
The split #
| Node | VM memory | Allocatable | What runs there |
|---|---|---|---|
| Control plane | 3 GB | 2.9 GiB (before the fix) | API server, etcd, controllers. Tainted, so no workloads |
| Worker 1 | 3.5 GB | 3.33 GiB (before the fix) | Gateway data plane, databases, apps |
| Worker 2 | 3.5 GB | 3.33 GiB (before the fix) | ArgoCD and the platform controllers |
The control plane started at 2 GB. I raised it to 3 GB on 29 October, and the same day Ansible started enforcing its taint, both at install and on every run:
- name: Ensure control plane taint is applied
ansible.builtin.command: >
k3s kubectl taint nodes {{ inventory_hostname }} node-role.kubernetes.io/control-plane:NoSchedule --overwriteThe worker split is a preference, not a rule. The shared pod defaults from part 13 prefer worker 1 for apps, and the scheduler can still use worker 2 if worker 1 is full.
Requests, limits and reality #
Measured with kubectl describe node and kubectl top nodes, before the fix at the end of this post:
| Node | Memory requests | Memory limits | Actually used |
|---|---|---|---|
| Control plane | 268 Mi (9%) | 586 Mi (19%) | 1,873 Mi (63%) |
| Worker 1 | 1,252 Mi (36%) | 3,568 Mi (104%) | 1,689 Mi (49%) |
| Worker 2 | 1,056 Mi (30%) | 3,280 Mi (96%) | 1,639 Mi (48%) |
Three things stand out.
The control plane uses seven times what its pods request. K3s runs the API server, etcd and the controllers inside one server process on the host, not as pods. The scheduler only counts pods, so to it, that node is 9% full. The real figure is 63%. Today’s 1,873 Mi would barely have fit in the original 2 GB. It’s also why the taint matters: a scheduler that thinks the node is nearly empty would happily put a workload there.
Limits add up to more than the node. Worker 1’s memory limits total 104% of what it has, and its CPU limits total 286%. That’s normal and fine. Limits are ceilings for each container, not reservations, and the containers don’t all peak at once. CPU overcommit is especially cheap, because a container at its CPU limit is slowed down, not killed.
Requests are the honest number. Requests are what the scheduler guarantees, and on the workers they sit at about a third of the node, with real usage at about half. Some pods use more than they asked for: Blinko requests 192 Mi and was using 251 Mi this morning. That’s allowed, because it’s under its 512 Mi limit, but it changes who gets killed first if memory runs out.
Limits that measurement corrected #
The current limits weren’t all right the first time. Three were raised after a kill or a measurement:
| Component | Old limit | New limit | Why |
|---|---|---|---|
| MetalLB controller | 64 Mi | 128 Mi | OOMKilled during a disaster-recovery drill’s reconnect storm. It idles at about 52 Mi |
| ArgoCD application controller | 512 Mi | 768 Mi | Raised with the bootstrap-release fix (ade060d). It was using 267 Mi this morning |
| Alloy’s config reloader | 48 Mi | 64 Mi | Grew from 6 to 39 Mi in its first day (b32e5ba) |
The MetalLB one is the pattern worth remembering:
limits:
cpu: 100m
memory: 128Mi # HARD-18: OOMKilled at 64Mi in the DR-05 reconnect storm; idles ~52MiAt idle, 64 Mi looked generous. The drill caused a reconnect storm, and that burst is what the limit has to cover. A limit sized from idle usage is sized for the one moment it doesn’t matter.
The same check found one more candidate: the MetalLB speakers were at 58 to 63 Mi against a 64 Mi limit. They now request 64 Mi with a 96 Mi limit.
Who dies first #
Every pod gets a QoS class from its requests and limits, and on this cluster that’s 38 Burstable pods, 2 BestEffort and no Guaranteed. One of the BestEffort pods is K3s’s own local-path-provisioner, which ships with no resources set. The other was a test pod that had finished days earlier, since deleted.
The QoS class matters because it sets each container’s OOM score. When a node truly runs out of memory, the kernel’s OOM killer picks the process with the highest score: BestEffort first, then Burstable pods in order of how far they are over their requests, with Guaranteed last.
I also have priority classes from part 13 (platform, apps, batch), with a comment saying the platform is “evicted last”. That’s only partly true, and the next section is why.
What I found while writing this #
The kubelet works out a node’s allocatable memory as capacity minus three things: memory reserved for Kubernetes daemons, memory reserved for the OS, and the memory eviction threshold. On every node here, allocatable equalled capacity. The kubelet’s live config showed why:
evictionHard: imagefs.available 5%, nodefs.available 5%
kubeReserved: none
systemReserved: noneK3s’s defaults set disk eviction thresholds and nothing for memory. So:
- Nothing was set aside for the OS, containerd or the kubelet. The scheduler could promise all of a node’s memory to pods.
- The kubelet never evicted for memory. Its node-pressure eviction, the one that respects priority classes and evicts politely, had no memory threshold to trigger it.
- The kernel’s OOM killer was the only backstop. It goes by OOM score, so QoS decided who died. Priority classes didn’t.
So the “platform is evicted last” comment held for scheduling (a high-priority pod can preempt a lower one to get placed) but not for a node running out of memory. Nothing had gone wrong, because requests sat at about a third of each worker. But the safety net I thought I had wasn’t there.
The fix, the same afternoon #
A K3s config drop-in, managed by Ansible, sets the reservations and a memory eviction threshold on each worker (ADR-003 addendum):
- name: Kubelet reservations and memory eviction
ansible.builtin.copy:
dest: /etc/rancher/k3s/config.yaml.d/40-kubelet-reservations.yaml
content: |
# Managed by Ansible (workers.yml), task 4.32
kubelet-arg:
- "system-reserved=cpu=100m,memory=256Mi"
- "kube-reserved=cpu=100m,memory=384Mi"
- "eviction-hard=memory.available<200Mi,nodefs.available<5%,imagefs.available<5%"eviction-hard replaces K3s’s defaults rather than adding to them, so the disk thresholds are restated at their old 5%. The 384 Mi for K3s on a worker covers the agent and containerd, which use about 420 Mi together. The control plane reserves 1.5 GiB for K3s instead, because its server process holds the API server, etcd and the controllers.
The rollout restarted one worker at a time (serial: 1). A restarted kubelet stays Ready for its lease time, so Ready alone proves nothing. Before moving on, the playbook waits until the node reports an allocatable value below its capacity: the proof that the new kubelet is the one answering.
| Node | Allocatable before | Allocatable after | Requests after | Limits after |
|---|---|---|---|---|
| Control plane | 2.9 GiB | 0.96 GiB | 300 Mi (30%) | 618 Mi (63%) |
| Worker 1 | 3.33 GiB | 2.51 GiB | 1,300 Mi (50%) | 3,600 Mi (139%) |
| Worker 2 | 3.33 GiB | 2.51 GiB | 1,088 Mi (42%) | 3,312 Mi (128%) |
Nothing about the workload changed. The percentages went up because they’re now measured against memory the node can actually spare. kubectl top nodes shows the control plane at 158% after the fix, which looks alarming and isn’t: the percentage is against allocatable, and most of what that node uses is the K3s server, which never counted as pods.
Priority classes now do what they say. Under memory pressure the kubelet evicts pods using more than their request first, then lower priority first. So a platform pod is protected only while it stays within its request. Two changes followed. The Envoy proxy and the tailnet gateway proxy got the platform class, which the class description had always claimed for “the gateway” (before, only the two databases had it). And the gateway proxy’s request went from 32 to 48 Mi, because it was using 35. The class description now says “evicted last under memory pressure while within its requests”, which is what actually happens.
What I learned #
- Measure all three numbers per node. Requests, limits and real usage each tell a different story, and the gaps between them are where the risk is.
- On K3s, the control plane is invisible to the scheduler. Size it from real usage, and taint it so nothing else lands there. (
6b24fec) - Size limits for the burst, not the idle. The MetalLB controller was fine at idle and OOMKilled in the one drill that mattered.
- Check allocatable against capacity. If they’re equal, nothing is reserved and there’s no memory eviction, whatever your priority classes say. (ADR-003 addendum)
- Priority classes govern scheduling, not the OOM killer. Without a memory eviction threshold, QoS decides who dies. With one, priority only protects pods that stay within their requests, so set requests from real use.
- Wait for proof, not
Ready, when rolling kubelet changes. A restarted kubelet looksReadybefore it is.
Architecture takeaways #
Why CPU throttles and memory kills #
CPU is compressible: the kernel can give a process less CPU time and it just runs slower. A container at its CPU limit is throttled by the CFS scheduler and keeps working. Memory isn’t compressible: the kernel can’t take back memory a process is using without breaking it. A container that goes over its memory limit is killed by its cgroup’s OOM killer (exit code 137). That’s why overcommitting CPU limits is routine and overcommitting memory needs a plan for what happens when the bet loses.
Why QoS classes exist #
Kubernetes needs a fair rule for who loses when a node runs out of memory, and the requests and limits a pod declares are the only promise it has. Guaranteed pods (requests equal limits) asked for exactly what they use, so they’re protected most. Burstable pods asked for less than they may use, so they’re at risk in proportion to how far over their request they are. BestEffort pods asked for nothing, so they go first. The class is translated into an OOM score the kernel understands, which is why the ordering still holds when Kubernetes itself has no time to act.
Why allocatable isn’t capacity #
A node isn’t only running pods. The kernel, systemd, containerd and the kubelet all need memory, and on K3s the control plane runs on the host too. Allocatable is meant to be capacity minus those reservations minus an eviction buffer, so the scheduler never promises memory that’s already spoken for. When the reservations are left empty, the scheduler’s view of the node is optimistic by exactly the amount everything else is using. Here that was about 2 GiB on the control plane and over 800 MiB on each worker.
Previous: A Free Outside Observer · All posts in the series