People keep asking me for my leverage number. I run a fleet of AI coding agents, the fleet merges hundreds of pull requests a month, and the question arrives in the same shape every time from people deciding what to believe about agent-run software work before deciding what to buy or build: how many engineers is it worth?
I don't have a number. I have a rule about why I don't have one.
A metric whose producer is also its beneficiary is a claim, not a measurement. That sentence is the whole position. What follows is what it did to my own dashboard when I tried to live by it — twice in one month.
most published figures fail their own audit
The shift that forced the rule is dated and concrete. In July, before publishing our own operating numbers, I put the widely cited AI-productivity figures through the same adversarial audit we apply to vendor claims: trace the number to a unit, a baseline, and a defect-escape rate, or fail it. Most failed. One much-quoted success-rate figure could not be traced to any of the three — no unit of success, no baseline to beat, no companion figure for the failures. It circulated with a decimal point, which is the costume precision wears when the derivation is missing. I am not reprinting it here: a number I cannot trace is a number I do not republish, and that rule does not bend for numbers I am criticizing.
An assayer does not test the stamp on a gold bar; the stamp is the mint's claim about its own metal. Most of what is published about AI productivity is stamps — produced by the party that benefits, quoted by parties who did not re-derive them.
the numbers I published in July had drifted by August
So in July we published only what a command could regenerate: 297 pull requests merged in the trailing 28 days, median open-to-merge 1.3 hours, median implemented-to-done 2.5 days. Each number carried its generating query. Each was true when it ran.
Re-derived on 2026-08-19, from the raw ledger of the fleet's busiest repository —
merged-PR records and bug-issue records pulled with a stock CLI
(gh pr list --state merged --json createdAt,mergedAt,
gh issue list --label bug --json createdAt) and reduced with a few lines
of arithmetic:
- 427 pull requests merged in the trailing 28 days
- median open-to-merge 2.8 hours
- median implemented-to-done 3.3 days, from the work ledger's status transitions
- 305 bug issues filed in the same window — 71 bug filings per 100 merged PRs, a partial, unflattering proxy for change-failure rate that we publish anyway, because a series that only keeps its good weeks is an advertisement
One July figure survived to the digit: re-run for its own July window, the open-to-merge median still lands on 1.3 hours. The rest had moved. July wasn't wrong; a number is a photograph, and republishing July's photograph as August's state would have been exactly the stamp I refuse to buy from anyone else.
The oldest number in the set is the one I keep republishing because it re-derives every time and flatters nobody. In the week of 2026-06-29, before the review gates existed, the fleet merged 8 pull requests and filed 29 bug issues against its own work: 362 bug filings per 100 merges. That is ungated agent throughput, measured. The weekly series since the gates came on runs 33, 39, 61, 109, 64, 40 — noisy, inflated mid-series by old defects surfacing late, and traceable line by line. It is a worse story than a single glossy percentage. It is also checkable, which the glossy percentage was not.
then my two instruments disagreed
The second incident is why the rule applies to tooling, not just to vendors. Re-deriving for this piece, I asked the same question two ways. Our roll-up tool reported 500 PRs merged in the window. The direct ledger query said 427. Same repository, same 28 days, 73 phantom merges. The cause was dull. The roll-up fetches its PR sample through a listing call capped at 500 records. The fleet's volume grew past the cap. The cap bound. The count was the cap.
No malice, no misbehaving agent — a tool we built, reading our own ledger, over-reporting our throughput in the flattering direction, with nothing in its output saying so. If I had not been re-deriving for publication, I would have quoted it. Producer-benefits bias does not require intent. An unexamined derivation is enough, and the beneficiary is the last person positioned to notice.
the strongest objection: everyone self-measures
If a self-interested producer disqualifies a metric, then every engineering organization's numbers are disqualified — DORA dashboards are computed by the teams they describe. The objection is correct, and it is why the rule is not "never self-measure." It is a rule about how much weight a self-measured number can bear before something independent has to hold it up.
There is an ordering. Self-measured and self-published is a claim. Publishing the
derivation next to the number raises it a rung. Keeping the unflattering rows in the
series raises it another. A re-derivation by someone who does not benefit from the
answer is what finally makes it a measurement. Our own numbers have cleared the first
two rungs and not the last: no outside party has reproduced them. Two of our own DORA
rows — failed-deploy recovery time and rework rate — we still print as
unknown, because the instrumentation does not exist and omitting the rows
would imply it did. And this piece was drafted with an AI agent inside the same fleet
the numbers describe, reviewed and stood behind by a human; on this subject, that
disclosure is part of the derivation.
Which is the answer to the opening question. A leverage number — "worth N engineers" — is the figure with the steepest incentive gradient toward its own producer and the weakest available independent check: no agreed unit, no shared baseline, no defect-escape companion. By the ordering above it cannot currently be more than a claim, whoever publishes it. So I won't.
what to do with the next number you are shown
None of this requires our tooling, or any tooling — the checks fit in four questions asked of whoever brings the number. What is the unit? What is the baseline? Where is the failure figure that travels with the success figure? Who ran the derivation, and do they gain from the answer?
And when you publish your own: ship the query with the number, date the window, keep the weeks that embarrass you, and state your caps before they bind — ours did, silently, at 500.
Until someone who does not benefit re-derives it, file every AI-productivity figure — mine included — under claims. A metric whose producer is its beneficiary is not a measurement yet. It is a stamp waiting for an assay.