Who assays the assayer: the morning main was green and red at once

On the morning of August 14th, 2026, our main branch was green — every check on the tip commit had passed, the same way it had passed the day before and the day before that, a settled row of checkmarks nobody had any reason to doubt. Main was also red. Both were true at the same time, about the same tree, and neither the green nor the red was lying. Working out how — and what finally surfaced it — turned out to be a tour of every layer where a verification system has to verify itself.

The whole morning compresses into one sentence, so it goes up front: plant what you register. A detector you have never watched detect is a claim, not a control. That morning we found the claim.

green is a statement about a diff, not a tree

The tree carries a Go test matrix — eleven modules that morning, nineteen as I re-derive this, the leak sweep among them — that runs whenever a push touches the paths those modules read. Main's tip was a status regeneration — generated files, no tool paths — so the matrix never ran there, and the tip wore its green honestly. Green because the diff touched nothing the matrix watches. Green because nobody had asked about the tree — only about the commit.

Every pull request that did touch those paths inherited a red it hadn't caused. What surfaced it wasn't a person: two machine re-reviews, on unrelated pull requests, each refused to blame the diff in front of it and traced the failure to the branch underneath. The review desk's whole job is to judge the diff; both verdicts amounted to the diff is fine — the bench it's standing on is broken.

What the matrix actually showed, on every run that morning: one module red — the leak sweep — and a train of siblings reading operation was canceled.

registered, but never planted

The leak sweep is the last gate before anything in this tree becomes public: it walks the staged copy hunting a registry of withheld tokens — internal names, identifiers, paths that must never ship. That morning the registry held a few dozen entries; by the time I re-write this it holds more — a number you re-count from your own token denylist rather than trust from memory, because it moved in five days, which is why stale numbers get re-run, never carried. The registry's design is blunt about trust. Every entry pairs its token with a control — a string known to be present in the tree, matched by the same mechanism. If the control doesn't turn up, the search itself is unproven: the entry reports could-not-check. Three states, never two: checked-clean, checked-failed, could-not-check. An instrument that cannot find its own control does not get to certify anything.

One entry is stranger than the rest: a synthetic token, fake by construction, its own text declaring it not a real name — the canary. Plant it, run the sweep, watch the red arrive. Because the token is invented, rehearsing never writes a real internal name into a fixture, a CI log, or a shell history. Its expected count on every clean tree is zero, forever.

And the rehearsal is automated. The sweep's suite keeps a deliberately dirty fixture tree with every registered token planted in it, and asserts — per token, all of them — that the sweep goes checked-failed on each one. Not most. Each.

Here is what had drifted. The canary went into the registry. The fixture never changed. The suite demanded a red for every registered token. It came up one short. On the dirty tree — the one place the canary must be found — the sweep reported checked-clean. One entry. One missing line. Red.

Read that failure again, because it is the good kind. The sweep did not miss a leak; there was no leak. The suite failed the sweep for being unable to prove it could catch one — it is written to watch the sweep detect, token by token, and the one token it couldn't watch turned the whole matrix red. The methodology held. We had registered a canary. We had never planted it.

Re-run while drafting this re-write, the same suite passes in under half a second — every token registered, planted, and rehearsed to a red (go test over the sweep's module). The canary has lived in the dirty fixture since the fix.

one red cancelled the other ten

The matrix ran with CI's default fail-fast: the first module to go red cancels the rest. That default was designed for a human at a laptop — you're waiting, the build is doomed, stop the meter. A fleet has the opposite economics. Nobody is waiting, and the question that matters is not is it broken but how broken, and where. Under fail-fast, the one honest red cancelled ten unknowns — and a cancelled required check blocks a pull request exactly like a failed one: eleven columns of apparent damage, one of true.

The fix is one line, fail-fast: false, with the reasoning in a comment above it so the default cannot creep back — both still in the workflow as I write this. Diagnosis is the scarce good when nobody is waiting. Latency was the scarce good when somebody was.

the report was wrong; so was the correction

The incident report named three causes: the unplanted canary, the fail-fast masking, and formatting drift in three modules. The correction that followed struck the third cause with a confident mechanism: formatting drift could not have reddened the matrix, it argued, because the workflow carried no formatting gate at all.

That premise is checkable, so I checked it — for the original draft, and again for this one: a git log -S 'gofmt check' across the repo's CI workflow shows the formatting gate sitting in that workflow since July 23rd, three weeks before the incident. The report was wrong about the drift; there wasn't any. The correction was wrong about why the report was wrong. And what settled it was neither narrative. The fix merged; the matrix ran all eleven of that morning's modules to completion, formatting gate included, and every column came back green — the day's run log reads four red runs, then one green, a dated record quoted from the incident. The green one ends the argument in a way no rebuttal can.

Every narrator in this story was unreliable — the report, the correction, the green tip, and me until I re-ran the checks. The instruments were not. That is the oldest lesson regulated infrastructure has to teach, and agent fleets make it load-bearing: you do not resolve competing narratives with a better narrative. You resolve them with an attestation — a re-run, a control, a record the narrators cannot edit. Maker-checker for the makers. Positive controls for the checkers.

the drill, on your own stack

None of this needs our tooling: a grep, a test suite, a rule you keep. Inventory the detectors your releases already trust: the secret scanner, the leak grep, the lint denier. Register a synthetic control for each — fake by construction, so the rehearsal never writes a real secret anywhere. Keep a deliberately dirty fixture with every registered shape planted, and assert the detector fails on each shape individually — an aggregate count hides exactly one missing line, which is what ours hid. Re-run the rehearsal on every registry change, and put the pairing in the suite, so registering without planting is a structural red, not a discipline you remember on good days. Two standing responses complete it: a canary that fails to go red is a stop-the-line finding on the detector, never a cleanup chore; a detector that cannot find its own control reports could not check, never clean.

the hedge

• One incident, one repo, one morning. Nothing here is a base rate.
• I did not measure blast radius. The tracker's phrase was that the red blocked "basically every open PR" touching the Go tree; I could not re-derive that count then and cannot now.
• fail-fast: false is not free. On a red day the matrix now runs every module to completion; on nineteen modules — today's count — that is bounded, on a hundred it would need re-arguing.
• The canary proves exactly one thing: that the sweep, pointed at a tree known to be dirty, flags every registered shape. It says nothing about shapes nobody registered — and the registry's own history includes green certificates over under-inclusive registries, variant spellings the tokens never matched. A positive control validates the instrument, not the inventory.
• The original draft disclosed a sweep finding then live against the tree. That sentence had a shelf life; I could not re-verify it while re-writing, so it is gone. A status nobody has re-checked is not a disclosure — it is a stale claim wearing one's clothes.

plant what you register

There is an old, slightly smug question about who watches the watchmen. Run an agent fleet and it stops being philosophy and becomes a build target. The workers are machines. The reviewers are machines. The sweeps and gates that make their output trustworthy are code — and code rots at the exact speed of everything else in the tree. So every instrument gets an instrument: a control it must find before its clean means anything, a canary whose red is rehearsed on purpose, a drill that disarms each refusal to prove something notices.

A green you have never seen go red is not evidence. It is decoration.

Plant what you register.