The day Assay's repository went public, the question holding the button was not the license. The license had been settled for weeks — a lawyer's afternoon, long done. What stood between the tree and the internet was a checklist of controls, and every control on it had already failed us at least once in private: a sweep that had certified a gap, a review that had waved through what it should have caught, a gate that had gone green for the wrong reason. Each had to be green again, at head, on the exact tree going out. The automated leak sweep was one row of that list. It came back clean. That was the least interesting fact of the week.
The repository had been written substantially by agents — commits, review comments, pull-request bodies, issue threads, produced at machine speed over months, and no human had read that output end to end. We do not have a defensible number for how much of it went unread, and the honest form of the claim is about process, not volume: nothing in our pipeline guaranteed a human pair of eyes on any given sentence before it entered history. Any one of those sentences could carry a secret, an internal identifier, or a path into the private tree it came from.
The week compresses to one sentence, and it belongs at the top rather than the end: checked clean is evidence about the checker, not the tree. Everything below is that sentence, hit from a different angle.
the exhaust ships with the code
Open-sourcing a human-written repository, you worry about the source: a key committed
in a config, a credential in a test fixture. Those failure modes are old and the
scanners for them are good. An agent-written repository moves the surface. The source
files are the most-reviewed artifacts in the tree — they are what the checks check and
the reviews review. The exhaust around them is not: commit messages, review comments,
pull-request narratives, issue bodies, the prose a fleet writes about its own work.
It is produced faster than it is read, it is written by processes that have seen the
private tree and will cite it without ceremony, and when a repository goes public, all
of it goes too. git log is part of the release.
Volume is what changes the threat model, not intent. A small human team produces exhaust a reviewer can plausibly have skimmed on its way in. A fleet produces it at a rate no skim survives, and the working assumption has to flip — from "someone probably saw this" to "assume nobody did." Every sentence in the history is unreviewed until a control says otherwise.
what the sweep is, and what it cannot be
The control we trust most is deliberately dumb. The leak sweep is the last gate before anything in the tree ships public: a pattern matcher that walks the outgoing copy hunting a registry of withheld tokens — internal names, identifier shapes, credential prefixes, paths that must never appear. Its design is blunt about its own fallibility. Every entry pairs its token with a control string the sweep must first find in the tree, and an entry whose control is missing reports could-not-check rather than clean. One entry is a synthetic canary, fake by construction, planted so the sweep's red can be rehearsed on purpose. We have watched this green go red. That is the only reason the green means anything — we wrote up the mechanism, and the morning it failed, separately.
And still: it matches tokens. It does not read. Three classes of leak are invisible to it by construction. A name nobody registered — the registry is an inventory, and inventories run under-inclusive. A cross-reference — an identifier that is an inert string to the matcher and a working pointer to anyone who has seen the tree it resolves in. And context — a sentence harmless on its own and revealing next to the one above it, which no per-token matcher will ever notice, because meaning is not a substring. The sweep's clean certifies exactly one thing: none of the registered strings appear. It has never certified more, however much a green checkmark invites you to read into it.
twice, in opposite directions
The reason we shipped anyway is that the sweep was never asked to be sufficient. Before the cut, an independent review pass went back through material the sweep had already passed — a different instrument asking a different question: not "does a registered string appear" but "does this sentence depend on anything a stranger cannot see." It found the class the matcher structurally cannot: prose written for an internal reader. References that resolve only in the private tree. Working vocabulary that maps our internals. Breadcrumbs individually innocent and jointly a map. All of it sitting under an honest green. The sweep had not failed; it had answered its question, and its question was too small.
Then the same lesson arrived pointing the other way. On later public-bound work, a model-driven review read a change, considered it, and passed it — and the mechanical sweep went red on the same content. The pattern matcher, which considers nothing, flagged what the reader's judgment had smoothed over. We did not overrule the red because a smarter layer had already said yes. The red stood until a person resolved it, which is the standing rule: a machine's clean never clears another machine's alarm.
That pair of catches, in opposite directions, is the whole argument for layering. The reader catches meaning the matcher cannot see. The matcher catches literals the reader's attention slides past — reading is exactly the faculty that normalizes what it reads. Neither layer outranks the other; each can veto; disagreement escalates to a person. Controls that trust each other collapse into one control with extra steps.
the button hardened into a ruleset
The cut itself was a day. What it left behind is an invariant. The public repository's main branch now requires the leak sweep as a branch rule — not a habit, not a line on somebody's checklist, a condition the platform enforces on every change regardless of who or what authored it. The one-time publishing control became a standing one, because the exposure is standing: the fleet keeps writing, the history keeps growing, and every future commit message is a future public artifact.
The review layer left a standing rule too: anything shipped in the public tree must be self-contained — no reference that resolves only in the private tree, no identifier whose meaning lives somewhere the reader cannot go. That rule is enforced the expensive way, by review, because it is a rule about meaning and we do not have a scanner for meaning. We would like one. We do not pretend the review pass is one.
withheld on purpose
Some artifacts did not go out, and not because a control flagged them. The file describing how our deployment is actually wired — which systems, where, connected how — stays private by decision, permanently, not "until cleaned up." The public tree ships a schema-only example in its place: the shape of the file, none of its contents. Stating the boundary up front is itself a control. A public tree with a declared perimeter can be audited against it; a tree that implies it withheld nothing invites the reader to find out otherwise.
the drill, on your own stack
None of this needs our tooling — a grep, a review habit, and a branch rule cover it. If agents write in your repositories and any of that work will ever be published or exposed:
- Treat exhaust as shipped content. Commit messages, review comments, and issue threads go public with the code; point every publishing control at them, not only at files.
- Run a literal scanner over the outgoing tree: a registry of strings that must never appear — internal hostnames, project names, identifier shapes, credential prefixes. Plant a synthetic canary in it, so you have watched its green go red before you trust one.
- Add a reader pass asking the other question: would this sentence mean anything it shouldn't to someone outside — and does it depend on anything they cannot see.
- Give both layers a veto, and let neither clear the other's red. A red from the dumb layer stands until a person resolves it, whatever the smart layer concluded.
- Make the gate standing: a required check on the public branch, so the invariant outlives the launch.
- Decide what stays private, write the boundary down, and ship schemas where the contents would have gone.
the hedge
- One repository, one cut, one team's controls. An existence proof of the failure classes, not a rate for any of them.
- "Substantially written by agents" is qualitative and stays that way. We do not have a defensible number for the split, and are not inventing one here.
- Both catches are told here without dates, artifacts, or identifiers — deliberately, because the records that would let you re-derive them resolve only in the private tree, and this article's own rule is that public content must not lean on those. That is an abstraction cost paid on purpose: you are getting the mechanism, not the attestation, and you should weight the anecdotes accordingly.
- The two opposite-direction catches are the ones a layer surfaced. What no layer caught, we cannot count — by construction. Today's clean public tree is the same species of claim this article warns about; our layered checkers are still checkers.
- The sweep's clean is case-sensitive and tokens-only, and its registry has run under-inclusive before — honest greens over real gaps, an inventory problem, not a bug.
- The self-contained rule runs on reviewer attention, which is finite and known to smooth things over. That is precisely why the mechanical layer keeps its veto.
evidence about the checker
Anyone open-sourcing anything an agent touched has inherited this problem, noticed or not. The history is part of the release; the history was written at machine speed; and the tool that says it is safe is a tool, with a question it can answer and an enormous space of questions it cannot. The day we shipped, the green sweep was real, necessary, and nowhere near the point. The point was the stack of instruments that disagree in different directions — and the rule that no single one of them, however smart, gets to say yes alone.
Checked clean is evidence about the checker. Layer the checkers, and make them distrust each other.