ArgoCD keeps the cluster matching Git all day (part 8). Nothing did the same for the cloud: the Oracle VM, the Google Cloud DMZ and the Backblaze bucket are all Terraform, and a change made in a console would sit there unnoticed until the next time I happened to run a plan.
The fix is a nightly terraform plan in GitHub Actions, which fails when reality and Git disagree. That’s a few lines of YAML. Doing it safely in a public repository took more thought, because my first version broke two rules at once: it gave GitHub a long-lived cloud key, and it printed the plan into a log anyone can read.
This post is the rebuilt version: keyless, read-only, printing only what drifted, and proven with a change I knew it had to catch (ADR-015 addendum).
What was wrong with the first version #
The original job authenticated with a service-account JSON key stored as a GitHub secret. Its role on the state bucket was objectAdmin, enough to rewrite or delete state. And on drift, it wrote the plan body into the job summary:
terraform plan -detailed-exitcode -no-color | tee plan_output.txtTwo problems. A JSON key never expires, so anyone who extracts it (a compromised action, a leaked log, a mistake in a workflow) keeps it until I notice. And a plan body contains attribute values: names, addresses, rule contents. In a public repo, the plan output is a secret, even when no single value in it looks like one.
Reference: Terraform state management in my knowledge base covers why state holds sensitive values and how to protect it.
Keyless: GitHub’s token, exchanged for a short-lived one #
GitHub Actions can issue each run an OIDC token that says which repository, branch and workflow it is. Google Cloud’s Workload Identity Federation can trust that token and exchange it for short-lived credentials, with a condition on who may do so. So the setup in my control-plane project is:
| Resource | What it does |
|---|---|
| Workload identity pool and provider | Trusts GitHub’s OIDC issuer. Its attribute_condition admits only this repository on refs/heads/main |
workloadIdentityUser binding |
Lets that repository’s identity act as the drift service account |
objectViewer on the state bucket |
Read state, nothing else |
cryptoKeyDecrypter on the SOPS key |
Decrypt the variables files (part 7) |
| Viewer and a small metadata-only custom role | Read what the plan needs to compare |
Forks and pull-request runs aren’t on main, so they get nothing. The workflow asks for an ID token and has no secret to read:
permissions:
contents: read
id-token: write # OIDC token for Workload Identity FederationOnce the federation worked, the JSON key was deleted, along with its GitHub secret. The bucket role dropped from objectAdmin to objectViewer, and plans run with -lock=false, because a lock is a write and this account can’t write.
The honest part: the real power of this identity isn’t the state bucket. It’s KMS decrypt, which opens every SOPS file in the repo. That’s why tokens are limited to main and short-lived, and why the drift account is one of the two principals the SOPS rotation runbook lists.
Printing only what drifted #
terraform plan -detailed-exitcode exits 0 for no changes, 2 for drift and 1 for an error. The job keeps the plan in a file, never in the log, and on drift prints only the action and resource address of each change:
2)
echo "⚠️ ${{ matrix.dir }}: drift detected. Resources that would change:" | tee -a "$GITHUB_STEP_SUMMARY"
# Addresses and actions only; values stay out of this public log.
terraform show -json tfplan \
| jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "- `\(.change.actions | join(","))` \(.address)"' \
| tee -a "$GITHUB_STEP_SUMMARY"
;;
*)
# Show only Terraform's error lines, not the plan body.
echo "❌ ${{ matrix.dir }}: plan failed." | tee -a "$GITHUB_STEP_SUMMARY"
grep -E '^(Error|│ Error)' plan.log | head -20 || true
;;“update google_compute_firewall.allow_icmp_ipv6” tells me where to look. The values are one authenticated terraform plan away on my own machine. The decrypted variables file is deleted straight after the plan, and the plan and log files at the end of the step.
The OCI root needed one change to be plannable in CI at all: its provider now takes the API key as private_key from SOPS, instead of a path to a key file on disk. One gotcha there: the key file Oracle generates ends with an OCI_API_KEY label line after the PEM block, and the provider rejects it until that line is removed.
Reference: Testing IaC in my knowledge base covers preview testing: running terraform plan in CI.
Proving it can fail #
The textbook drift test is “add a label in the console and watch the plan notice”. It passes silently here: since v5, the Google provider manages only the labels declared in config, and the DMZ declares none, so a console label isn’t drift. A drift test that can’t fail proves nothing.
So the test changed something Terraform fully owns: the description of a firewall rule. The next run failed on exactly that address. I reverted it, and the run after that was green. Both outcomes were seen, which is the only way to know the check works.
When a scheduled run fails, GitHub emails me. That’s the alert for now. Sending it to Telegram like everything else is still an open question in the ADR.
The environment I left out #
There are four Terraform roots. The nightly job plans three. The fourth, the Proxmox homelab, is planned by hand once a month, by choice.
The bpg/proxmox provider needs VM.Config.Disk, the permission to attach, detach and resize disks, just to plan. I tested it on 13 November: a read-only PVEAuditor token could read (200), couldn’t write (403), and the plan failed with 403. Giving CI that permission would mean a GitHub runner holding disk-write rights on my hypervisor, plus a network path into my home. A monthly manual plan is the better trade.
Since this was set up, two more guardrails joined it. The main branch is protected from force-pushes and deletion, so every commit hash the docs and the status page quote stays valid. And every push runs a validation workflow with the same pre-commit hooks I run locally, now including shellcheck and yamllint.
What I learned #
- The plan body is a secret in a public repo. Print actions and addresses, never values. (
terraform-drift.yml) - Replace stored cloud keys with OIDC federation, and condition it on the repository and branch, so forks and PRs get nothing. (runbook)
- Know what the identity can really do. Read-only state access looked like the whole story. KMS decrypt is the part that matters.
- Pick a drift test that can fail. Change a field you know Terraform owns, and see the run go red and then green.
- Leaving something out of automation can be the secure choice, when automating it needs a permission you wouldn’t give a stranger. (ADR-015 addendum)
Architecture takeaways #
Short-lived, audience-bound credentials #
Kubernetes moved from long-lived ServiceAccount token Secrets to bound, projected tokens for the same reason this job moved from a JSON key to OIDC. A projected token is issued for one audience, expires, and is tied to the pod that requested it. A stolen one is worth little and not for long. Workload Identity Federation gives a CI run the same shape: a credential that exists only for this run, from this repository, on this branch.
Drift detection is the read-only half of a controller #
Every Kubernetes controller runs the same loop: compare desired state with actual state, then act. ArgoCD does it for the cluster and fixes what it finds. A nightly plan is the same comparison without the second half, which is deliberate here: a CI job that could apply would need write access to everything it watches. Detection with a human in the loop is the safer half to automate first.
Previous: A Public Status Page That Can’t Leak, and Can’t Flatter Me Either · All posts in the series