Part 18 ended on a list of things I hadn’t proven. One of them was the DMZ observer: the free Google Cloud VM from part 16 that watches everything else and serves the public status page. Its recovery target was 15 minutes, and every drill of it had passed.
Every one of those drills kept the machine. Stop the containers, redeploy, check the monitors. On 19 November I deleted the VM and ran the documented rebuild for real, with a stopwatch. It took 48 minutes and hit three breaks, none of which a drill on the same machine could have reached (INC-2026-016).
This post is that drill: the three breaks, the fixes, why the target went up afterwards, and an older incident that’s the same lesson from the other side.
The setup #
The runbook is DR Scenario 05: delete the VM, recreate it with Terraform, deploy it with Ansible, restore Uptime Kuma’s data. The only state on the machine that isn’t in Git is Kuma’s database: every monitor, its history, and the push tokens other hosts report to.
Before deleting anything, I took a copy of that database with Kuma stopped and ran SQLite’s integrity check on it. Kuma’s SQLite runs with a write-ahead log, and a copy of a running database can be inconsistent. A backup you can’t open isn’t one. The check passed with all 22 monitors, the Terraform plan was clean, and I deleted the VM.
| Time (UTC) | What happened |
|---|---|
| 15:06:53 | VM deleted. The clock starts |
| 15:07:51 | Terraform creates the new VM, still with no public IPv4, so still free |
| 15:11:24 | Its startup script fails to join the mesh |
| ~15:24 | Joined by hand through the break-glass path |
| 15:26–15:42 | The Ansible deploy: 16 minutes, mostly image pulls, then a crash at the last stage |
| 15:43–15:52 | Kuma restored, deploy rerun cleanly |
| 15:53 | Every monitor green except the Watchdog heartbeat |
| 15:55:18 | Watchdog received. Recovery complete: 48 min 25 s |
Break 1: the mesh said no #
The new VM booted, and its startup script ran tailscale up with the shared enrollment credential. Tailscale refused:
requested tags [tag:gcp-dmz] are invalid or not permittedThe VM was up, but not on the mesh, so nothing could reach it. I found it from the serial console about six minutes later.
The cause was a rule I’d already written about. The edge VM and the cluster nodes join with one OAuth client, which holds three tags, and each machine asks for one of them. As part 4 covered, Tailscale lets a client mint a subset of its tags only if one of its own tags owns each requested tag. Two of the three tags were owned by admins only, so every single-tag request was refused.
I’d fixed exactly this rule on 11 November, for a different client. Nobody checked whether another client relied on it, because this one is only used when a machine is created. It was set up on 11 October, when the nodes first joined, and the drill was the first time it was tested since.
The way back in was the break-glass path from part 16: Google’s IAP SSH, which doesn’t need the mesh. I made a one-time auth key with the right tag in the admin console and joined the VM by hand. The permanent fix was a small policy change: the edge and cluster tags now own themselves, as the cloud host’s tags already did. Self-ownership only lets a holder of a tag mint that same tag, so it grants nothing new.
This break would also have stopped a rebuilt cluster node from joining. No drill had covered that either.
Break 2: a script assumed yesterday #
The deploy ran for 16 minutes, pulling every image onto a fresh 1 GB VM, then stopped at the last stage. A small script that reads Kuma’s database for the status page’s backup ages crashed with FileNotFoundError. On a fresh VM the database doesn’t exist until the restore, and the runbook restores after the deploy.
The fix is to treat that as a normal state, not an error:
def main():
# A rebuilt DMZ has no Kuma database until the restore (DR Scenario 05 §3.5).
# Not an error: skip this run, so the deploy that installs this timer completes
# and the next run after the restore fills the page (5.53 drill finding).
if not os.path.exists(DB) or not os.path.getsize(DB):
print(f"{DB} missing or empty (fresh VM, Kuma not restored yet): skipping this run")
returnIt’s a small bug, and exactly the kind that only a machine with no history finds.
Break 3: the address was inside a secret #
After the restore and a clean rerun, every Kuma monitor went green except one: the Watchdog heartbeat from part 15, the always-firing alert that proves the alerting path works. Alertmanager’s log said contextDeadlineExceeded. It was still posting to the old VM.
Every heartbeat sender used the VM’s mesh IP address, and a new machine gets a new one. That’s Alertmanager’s Watchdog, Kestra, and four backup stages on the hypervisor, all rendered from one shared secret. The runbook did say to look for the old address, and a search of the repo found nothing, because the address was inside an encrypted file. A plain-text search can’t see it.
During the drill, I gave the new machine its old mesh address in the Tailscale console, and the Watchdog arrived within seconds. The real fix is to stop depending on the address at all: senders now use the VM’s mesh DNS name, which survives a rebuild. I checked that the name resolves and connects from every sender, including from inside the Alertmanager and Kestra containers, because part 16 taught me the host and its containers can disagree.
The heartbeat design is what made this a two-minute finding. In a real loss at night, the rebuilt VM would have looked healthy while every backup heartbeat failed to arrive, and the morning would have started with a run of false “backup failed” alerts.
Proving the join without a rebuild #
Break 1 could come back whenever someone edits the policy, and I don’t want to delete a VM to find out. So the repo now has a check that proves a machine could rejoin, without touching one (check-tailnet-join.sh). It starts a throwaway Tailscale node in a container, logs in with the same stored credential and tag the real machine uses, and reports the result:
if OUT=$(docker exec "$NAME" tailscale up --authkey="$KEY" --advertise-tags="$TAG" \
--hostname="$NAME" --timeout=40s 2>&1); then
unset KEY
echo "OK: the $1 secret joins with $TAG"
else
unset KEY
echo "FAIL: the $1 secret can't join with $TAG: $(echo "$OUT" | tail -1)"
exit 1
fiBefore the policy change, both tags were refused. After it, both joined. The check is now a step in the two DR scenarios that rebuild machines. One gotcha: the node is ephemeral, but Tailscale can keep listing it offline for a while after it logs out, so it sometimes needs deleting by hand.
The number, and the target that went up #
The 48 minutes split roughly like this:
| Part | Minutes | Fixed since? |
|---|---|---|
| First deploy on a fresh e2-micro, mostly image pulls | 16 | No: this is the machine’s speed |
| Break 1: noticing the failed join and joining by hand | about 13 | Yes |
| Break 2: the crash, the restore and the rerun | about 10 | Yes |
| Break 3: finding the address | about 3 | Yes |
| VM creation, boot and checks | the rest | No |
The 16 minutes is the uncomfortable part. The 15-minute target was never achievable, even with no breaks at all. A deploy on that VM takes longer than the whole target allowed. Nobody knew, because no one had ever deployed it onto an empty machine and watched the clock.
So the target went up to 30 minutes (9ecebf3). That felt like losing, but a target that the hardware can’t meet isn’t ambitious, it’s untested. The recovery objectives file, which feeds the public status page, records the measured 48 minutes next to the 30-minute target, and the gap stays visible until a rerun closes it.
Reference: Disaster recovery in my knowledge base covers how a post-mortem is structured, the format each break here was written up in.
The reverse case: machines that thought they were new #
The rebuild broke because a new machine didn’t have the old one’s identity. Two weeks earlier, the opposite happened (INC-2026-010).
After an unclean hypervisor restart, all three K3s VMs regenerated their SSH host keys on the same boot. cloud-init picks a VM’s instance ID from a hash of the configuration the hypervisor hands it. The VMs didn’t set a DNS search domain, so Proxmox filled it in from the hypervisor’s own /etc/resolv.conf. The mesh client rewrites that file, so what it said depended on whether the client had started yet. After the unclean boot it said something different, every VM got a new instance ID, and cloud-init treated each one as a brand-new machine.
That night, the first stage of the etcd backup chain connected to the control VM over SSH, saw a changed host key and refused. That’s the right behaviour. But the two later stages ran on the previous day’s copy and reported success, the same pattern as Green Is Not a Status. I checked the new key over the mesh, where the peer is authenticated, before trusting it. The fix was a fixed search domain (home.arpa) on every VM, so the configuration no longer depends on the hypervisor’s runtime state.
One case is a machine that kept its name and lost its identity. The other is a machine that kept everything and decided it was someone else. Identity is state, and it changes when you’re not looking.
What I learned #
- A drill that keeps the machine tests the software, not the recovery. Identity, credentials and addresses only break when the machine is really replaced. (INC-2026-016)
- A credential used only at creation time rots silently. When a fix changes how a rule works for one credential, check every other credential that depends on the same rule. (
0657e1d) - Search the decrypted secrets for an address, not just the plain files.
- Set recovery targets from a measurement. Sixteen of the 48 minutes were the machine’s speed, so the old target was impossible. (Scenario 05)
- A blank setting gets filled by someone else’s default, and here the default depended on the host’s runtime state. (INC-2026-010)
Architecture takeaways #
Name things, don’t address them #
Kubernetes makes pods talk to a Service’s DNS name, never to a pod IP, because pods are replaced and get new addresses. The name is the contract and the address is an implementation detail. Break 3 is what happens without that rule: six senders held a copy of the address, and replacing the machine silently broke all of them. A mesh DNS name does for VMs what a Service name does for pods.
Joining is a privileged moment, so test it #
Kubernetes gives node joins their own short-lived bootstrap token, separate from everything a node does day to day, and every new node exercises it. The enrollment credential here was the same idea without the exercise: nothing tested it between 11 October and the drill, so a policy change broke it and nothing noticed. A credential that only matters during recovery needs a scheduled test that uses it, like the join check.
Cattle needs identity to be automatic #
“Cattle, not pets” only holds if a replacement can get its identity without a person. In Kubernetes, the join creates the Node object, its certificate and its name. Each manual step in this rebuild (a hand-made auth key, a hand-set address) was a pet. Two of them are now automated, and the third, the restore, is ordered so the deploy no longer depends on it.
Previous: Disaster Recovery You Can Point To · All posts in the series