The same work item was implemented twice, eighty-one minutes apart, by two workers in the same agent fleet. The rule that should have prevented it existed and was written down: claim before dispatch, never dispatch a claimed item. The model running the loop had read it. But the rule lived as prose in the model's attention, and under a full pool of in-flight workers the attention went to the work — and the schedule fell on the floor.
A second loop, separately, reported itself idle while dispatched work was still in flight. "Is the queue empty" and "are the workers done" are two different facts, and the model was tracking one of them.
Neither failure is a reasoning error inside an item. Both are scheduling errors, and they generalize: any loop with a model near its outer edge will eventually drop a fact left in that attention. The lesson: anything the loop must not forget belongs in code, not in the model's attention.
orchestration collapses into attention
Give one model two jobs — decide each item on its merits, and keep the whole schedule in its head — and under load the schedule is what it drops. Attention is a fixed budget. Every slot of it spent remembering "worker three is still running, worker one landed, refill it, don't re-dispatch item nine" is a slot not spent on the judgment the item needs. The pool grows, the bookkeeping grows with it, and the attention does not. The predictable failures follow: a forgotten refill, a duplicate dispatch, a drain declared finished while workers are still out.
The eighty-one minutes is that collapse landing: two implementations of one item, built in parallel, because the no-re-dispatch rule was a sentence the model was supposed to keep holding, and under a full pool it did not hold. The false idle report is the same collapse from the other side: one fact tracked, the other dropped. Both are scheduling facts. Neither is judgment. Both belong in code.
the fix: a scheduler in code, judgment in the model
The correction is a separation of concerns, almost boring once stated. The scheduler is a deterministic state machine: read the queue, hold a pool of N, claim before dispatch, land each result, refill, poll for idle. None of that is a judgment call; all of it is what code is good at and attention is bad at. So the scheduler became code — a small engine — and the model kept exactly one job per item: the judgment inside the dispatched agent.
The engine decides which dispatch to make and when; the agent decides what the answer is. The engine never reasons about an item's content; the model never tracks a slot, a claim, or a refill. One honest caveat: in its interim form our engine does not spawn the worker itself — it emits an exact dispatch instruction, an operator executes it verbatim, and a structured result is fed back. Even so, zero scheduling state lives in the model's attention; the later upgrade to a native child process swaps one method and touches nothing else.
Here is one drain in the engine's own words — five stand-in items, pool capped at two, a synchronous stand-in dispatcher (the trace reads serially), run for this piece against a reference build:
CLAIM item-1 DISPATCH item-1 LAND item-1 ok=true; RELEASE CLAIM item-2 DISPATCH item-2 LAND item-2 ok=true; RELEASE [items 3 and 4 drain the same way] CLAIM item-5 DISPATCH item-5 LAND item-5 ok=true; RELEASE IDLE — stop
Every line is a scheduling decision recorded as it was made. No model narrated it afterward. The trace is the run.
six methods, and a seventh is a design review
A scheduler serving more than one role needs a contract, and the contract is where such projects go wrong: every role wants one more hook, and a year later the engine is a second home for per-role policy. Ours is frozen at six — six in the reference form below and six in the production engine, re-counted for this piece — and a proposed seventh is a design review, never a patch.
type Loop interface {
Name() string // identity: logging and the claim namespace
SelectQueue() ([]Item, error) // where items come from
TierPolicy(Item) Tier // routing policy; a hold routes to a human
Dispatch(Item) (Handle, error) // carry out one dispatch
Land(Item, Result) error // record one result, release what it held
OnIdle() (bool, error) // what "empty" means: poll again, or stop
}Claiming is deliberately its own small interface — claim, release — because the dedupe rests on it: an item is in flight at most once, for every dispatcher that routes through the claim. Skipping the claim puts a dispatcher outside that property by construction, not by policy. This is the eighty-one-minute failure made structural: the rule stopped being a sentence and became the only path to a dispatch.
Everything role-specific lives in TierPolicy and
Land; the pool arithmetic, the claim-and-refill dance, and the
drain-continues-on-failure discipline are the engine's, written once. Two disciplines
it enforces rather than requests. A failed dispatch does not abort the drain: it is
landed as failed, its claim released, the slot refilled — one bad item cannot freeze
the pool. And failure has three outcomes, not two — retry, refuse, cannot-classify —
because a two-state taxonomy must guess on an unrecognized failure, and both guesses
are defects: retry a refusal forever, or silently drop a transient one. The unclassifiable failure routes to a human — the same three-state
honesty as checked-clean, checked-failed,
could-not-check.
the worked case: a pipeline that reports its own progress
One family of these loops drains a delivery pipeline: a merged change compiles to a versioned artifact, the artifact ships, the target system has to accept it, and a reconciler has to converge the environment onto the declared state. Each stage is an item, and its judgment — did the target actually accept it, did the reconciler actually converge, or is it reporting progress it has not made — is real per-item work for an agent.
Verifying a deployment is exactly the task where the tempting shortcut is to trust the report: the pipeline says converged, so mark it converged. With the scheduler in code, that shortcut is unavailable: the only way an item leaves the queue is through dispatch to a verifier that queries the target directly. There is no "checked it in passing" path — the queue-runner is code, and code that can only dispatch cannot wave anything through. Most of the fleet's work is ordinary software items; the pipeline is where the separation pays the largest dividend.
what it buys, and the limit of the claim
Three things. Attribution: every claim, dispatch, land, and release is journalled as it happens — the record of the run is derived from the run. Evidence: a landed result is rendered from the runner's structured output — command, exit code, key lines — never from free text a model wrote about its own work. And structural separation: the entity that schedules cannot be the entity that judges.
What it does not buy is a result that cannot be faked. A caller with the right signing material can still forge one. The stronger close — the author of a change being unable to also approve it — belongs to distinct signing identities and branch protection, outside the scheduler entirely. The failure mode of a piece like this is to let "the scheduler cannot cut corners" inflate into "the results cannot be faked"; those are different claims. The engine attributes; it does not certify.
the takeaway, made operable
The rule is the one planted at the top: anything the loop must not forget belongs in code, not in the model's attention. On your own stack, with tools you already have:
- Write the outer loop as a deterministic engine holding a constant pool of N; the model never learns the pool exists.
- Make claim-before-dispatch the only path to a dispatch; a claim store can start as small as exclusive file creation — ours did.
- Land every outcome, failures included, and release the claim in the same motion; the drain continues past any single item.
- Classify failures three ways — retry, refuse, cannot-classify — and route the third to a human instead of guessing.
- Freeze the contract small; a new hook is a design review. Six methods have covered our three roles so far.
The standing response when a loop misbehaves: ask which scheduling fact the model was holding, and move it into the engine. The cadence: re-ask at every new role, because each arrives wanting a seventh method.
the hedge
- The eighty-one minutes is the gap between the duplicate implementations' pull-request creation times — re-measured for this piece at eighty minutes and thirty seconds, recorded rounded to eighty-one; we keep the recorded figure, derivation stated.
- The reference engine quoted here is under five hundred lines with its tests; its claim store is deliberately minimal — single-machine, exclusive-create, no stale-claim reclaim. Production needs reclaim, a durable cross-machine lock, and an evidence probe before granting a claim.
- The dedupe is a chokepoint, not a perimeter: it binds dispatchers that route through the claim, and a process that bypasses the engine can still double-dispatch. Ours did — which is why the claim moved into code.
- Today the engine emits dispatch instructions that an operator relays; the scheduling state has left the model, but the loop is not yet unattended end to end.
- None of this is novel: moving deterministic work out of a non-deterministic component is old. With agents, the component bad at determinism is the same one good at judgment — that coincidence is the only reason to write it down.
The move is small. Take the schedule out of the place that is bad at holding it — because right up until the pool is full, that place will insist it can do both, and then the schedule will quietly fall on the floor again.