“Day 2” is everything after the cluster is built: patching, upgrades, reboots, key rotation. On 19 November I wrote the runbooks for three of those jobs before any of them was urgent:
- Node maintenance, with a script that predicts what a drain will do.
- K3s upgrades, with a playbook that swaps one binary per node.
- SOPS key rotation, for the keys behind every secret in the repo.
A runbook that has never run is a guess written in the imperative. On 20 November the upgrade one ran for real, from K3s v1.36.4 to v1.36.6, one node at a time. The playbook itself took about nine minutes across three nodes. It found three things the runbook had wrong (INC-2026-019).
Predicting a drain #
Draining a node evicts its pods so they start somewhere else. On a cluster with spare capacity that’s the end of it. Mine has two workers with about 2.5 GiB of allocatable memory each, single replicas and no PodDisruptionBudgets, so a drain moves everything at once into whatever room the other worker has.
So before writing the procedure I wrote a read-only check (drain-check.py). It reads every pod’s requests, priority class and volumes, places them highest priority first as the scheduler would, and gives each pod a verdict:
fits <node> the scheduler has room for its memory and CPU requests there
preempts <pods> no room, but its priority class evicts lower-priority pods there
PENDING no room and nothing it may preempt: down until the node is back
PINNED its volume is local-path on this node: down until the node is backWhat it said on 19 November:
| Drain | Moves | Stays down until the node is back |
|---|---|---|
| Worker 2 | ArgoCD, the NFS driver’s controller, cert-manager’s webhook | Paperless (needs 640 MiB, 632 MiB free) and most of the controllers |
| Worker 1 | Redis, the gateway proxy, the tailnet gateway and most apps | PostgreSQL and CouchDB, pinned by their local volumes. Linkwarden preempts all four ArgoCD pods on worker 2 |
Neither worker can hold the other’s pods, and draining the database worker is a planned outage, never a failover. Knowing that before starting is the point.
The runbook’s other rule came from reading the node config: always drain, never just reboot. The kubelet here has no graceful node shutdown configured, so a bare reboot kills PostgreSQL mid-write and leaves it to WAL recovery. A drain sends it SIGTERM and gives it 60 seconds to shut down cleanly.
Upgrading K3s by swapping one binary #
K3s is one binary. Its install script can upgrade it, but rerunning the script rewrites the systemd unit and needs the cluster’s join token, which lives in escrow. So the upgrade playbook uses the other method K3s documents: replace the binary and restart the service. The unit, its flags and the config files stay exactly as they are.
For each node, the playbook:
- Skips the node if it’s already on the target, so a rerun after a failure carries on where it stopped.
- On the control plane, takes an etcd snapshot named after the version it’s leaving.
- Drains the node.
- Downloads the new binary and checks it against the release’s SHA-256 list.
- Keeps the running binary for a rollback, then swaps in the new one.
- Restarts K3s, waits until the node is Ready and reports the new version, and uncordons it.
- name: Download the release binary, checked against its SHA-256 list
ansible.builtin.get_url:
url: "https://github.com/k3s-io/k3s/releases/download/{{ k3s_target | urlencode }}/k3s"
checksum: "sha256:https://github.com/k3s-io/k3s/releases/download/{{ k3s_target | urlencode }}/sha256sum-amd64.txt"
dest: /usr/local/bin/k3s.new
- name: Keep the running binary for a rollback
ansible.builtin.copy:
src: /usr/local/bin/k3s
dest: /usr/local/bin/k3s.previous
remote_src: trueThe order is control plane first, then worker 2, then worker 1 last, because its drain takes the databases down. Rolling back a patch release is putting k3s.previous back. Rolling back a minor release is restoring the snapshot, because a newer Kubernetes may have written data the older one can’t read.
Renovate tracks the version: one pull request bumps the pin in both playbooks, and a minor release waits for my approval. Merging it changes nothing on the cluster. A person runs the playbook, in a window that avoids the night’s backups.
The first real run #
| Node | Playbook time | Notes |
|---|---|---|
| Control plane | 2.5 min | Snapshot first. K3s replaced CoreDNS, a single replica, so cluster DNS was down for about 20 seconds |
| Worker 2 | 1.6 min | Finding 2 |
| Worker 1 | 4.5 min | The planned database outage. PostgreSQL was ready about five minutes after the drain started. Finding 3 |
No rollback was needed. The whole job, including putting things right, took from 12:00 to 12:29 UTC. Three findings came out of it.
1. The documented one-node command had never worked #
The runbook suggests running one stage at a time, with --limit to a single node. The first --limit to a worker failed before touching it: the workers’ play read its target version from the control plane’s play, and --limit to a worker skips that play.
It cost nothing, because the playbook checks the version before it does anything else. The fix was for each play to read the version pins itself. Every command form a runbook documents is a claim, and this one had never been run.
2. The receiving node evicted its own load-balancer agent #
Worker 2’s drain put Paperless on worker 1. The check had said Paperless would stay Pending, 8 MiB short. Instead, the scheduler made room by preempting the MetalLB speaker on worker 1, the per-node agent that announces the cluster’s LAN address. Its replacement then sat Pending with “Insufficient memory”, because a DaemonSet’s pod can only run on its own node.
Worker 1 runs the gateway proxy, so for 4.5 minutes nothing announced the address every app is reached on. The apps kept answering only because devices on the LAN still had the address in their cache. A device that had to look it up again in that window would have failed, and nothing in the cluster would have reported an error.
The cause was a gap in the priority design. I’d given priority classes to the apps and the platform pods, but the cluster’s own per-node agents had none, so they ranked below every app. The drain check made the same assumption from the other side: it never counts a DaemonSet pod as something a moving pod can preempt.
Kubernetes ships a priority class for exactly this, so the speaker, node exporter and Alloy got it:
speaker:
# A per-node agent: without it a node's LoadBalancer IPs aren't announced.
priorityClassName: system-node-criticalThe speaker was back within minutes, after evicting the database exporters, which also have no class. Restarting Paperless sent it back to worker 2, which it prefers, and the exporters returned.
Reference: Scheduling in my knowledge base covers PriorityClasses, how preemption picks its victims, and the built-in system classes.
3. Pods don’t move back #
The worker 1 drain did what the check predicted: Linkwarden moved to worker 2 and preempted ArgoCD. The check didn’t say how long that would last. Linkwarden’s image wasn’t cached on worker 2, so it spent minutes pulling, and ArgoCD, which only runs on worker 2, waited behind it. A planned five-minute outage for Linkwarden became about twelve, and ArgoCD was down for about nine.
The scheduler never moves a pod back after an uncordon. The runbook had said so and called it fine, because the next rollout spreads things out again. With two full workers it isn’t fine: the next drain starts from a full node. The step it now has, after every node:
kubectl cordon <node the pod moved to> # cordoning evicts nothing
kubectl -n <ns> delete pod <pod> # its Deployment recreates it on the other node
kubectl uncordon <node the pod moved to>The checks after an upgrade gained two rows: every DaemonSet has a pod on every node, and every pod is back on its preferred node.
One more thing showed up afterwards. Draining makes each VM touch its full memory, and the hypervisor doesn’t take it back, so the host has sat at its 12 GB ceiling since. That’s the case for the RAM upgrade, which gets its own post.
Rotating keys in a public repo #
The third runbook (SOPS key rotation) hasn’t run for real yet, and I’d rather say so than write it up as proven. Writing it still changed how I think about the keys from part 7:
- In a public repo, re-encrypting doesn’t undo a leak. Every encrypted file in every old commit stays readable by anyone holding a key it was encrypted for. If a key may have been read, the fix is changing the secret values where they’re issued, and the runbook lists every one, highest risk first.
- The cluster gets both keys before any file changes. KSOPS in ArgoCD can only decrypt with the key in its Secret, and an age key file can hold several identities. So the new key goes into the cluster alongside the old one, then the files are re-keyed, then the old key is retired. ArgoCD can decrypt at every step.
updatekeysisn’t always enough. It re-wraps each file’s data key for the new recipients, but if the old key leaked, it can still unwrap a data key it saw in an old commit. Then every file needs a new data key too (sops rotate).
The one piece that has run is the check that came with it. A script now fails any commit where an encrypted file lacks one of the two recipients. Its first run found a secret that had been encrypted to only one key for three weeks, which is the correction now in part 7.
What I learned #
- Writing the runbook is half the drill. Running it is the other half. Writing them found the drain predictions and a secret encrypted to one key. The first real run found three more problems in nine minutes. (INC-2026-019)
- Run every documented command form once, including the “one stage at a time” variant, before calling a runbook operational. (
1b8f2f1) - Give every per-node agent
system-node-critical. An app should never be able to take a node’s networking, metrics or logs away. (8654089) - On a cluster with no spare room, a drain isn’t over at the uncordon. Put back what moved before the next node.
- Upgrade one node at a time and check in between. Each problem showed up while the next node was still untouched.
Architecture takeaways #
Why drain uses the Eviction API #
kubectl drain doesn’t delete pods. It asks the Eviction API, which refuses an eviction that would break a PodDisruptionBudget, so a drain can’t take a replicated service below the replicas it promised to keep. This cluster has no budgets, because everything is a single replica and a budget would only block the drain. That’s an honest trade for a homelab, and it’s why the drain check and the runbook exist instead: when nothing stops a drain, you need to know in advance what it will take down.
Why the control plane goes first #
Kubernetes’ version skew policy lets a kubelet be older than the API server, never newer. Upgrading the control plane first keeps every step inside that rule: for a few minutes the workers run an older kubelet against a newer API server, which is supported. The reverse order would break that rule, and for a minor release the policy also says to move one minor at a time, never skipping one.
Why priority and preemption exist #
When a cluster runs out of room, the scheduler has to decide who loses. Priority classes are how you decide in advance, and preemption is the scheduler acting on that decision. Kubernetes reserves two classes above anything a user can create: system-cluster-critical for cluster add-ons and system-node-critical for pods a node can’t work without. My mistake was ranking the apps carefully and never ranking the agents, so the scheduler ranked them for me, last.
Previous: Diagrams as Code · All posts in the series