An engineering lead can count more merged pull requests while delivery waits longer. More completed briefs can coexist with a user workflow that has barely improved. A useful measurement starts with the decision it should change.
Assay organizes work into bounded requests called briefs and records implementation, review and independent verification as different events. It is developing resource measurement and planning later user-outcome assessment. Together they define an approach to following work from its request into use; the observation needed depends on the question.
Choose the measurement boundary
The running case is Desk tools brief 24, an audit-ledger improvement used by deskpost. Deskpost posts reviews after checking authority and safeguards. The brief reduces shared read costs while preserving write limits, previous-post detection and recovery.
Arrival, implementation, review, merge, verification and release are different handoffs. A merged candidate can still await independent acceptance checks. A completed implementation can still await its package release.
Start with the decision the measurement should change.
For delivery waiting, choose the events bounding each interval. For performance, choose the operation and workload. For user benefit, identify the attempts exposed to the changed version. A familiar counter cannot choose those boundaries for you.
Read the delivery timeline
Suppose the ledger work has this handoff record:
09:00 brief enters the queue
10:00 worker starts
10:45 candidate enters review
11:15 reviewer returns a finding
12:00 repaired candidate returns
12:20 required reviews complete
15:00 human merges
15:20 independent verification passes
Arrival to verified delivery is 6 hours 20 minutes. The approved-candidate wait for merge is 2 hours 40 minutes. The review interval is 1 hour 35 minutes, including the first examination, 45 minutes between finding and repair, and the revised-candidate review.
For this item, that record directs attention to the merge handoff before adding reviewer capacity. Compare more items before changing team capacity. It also prevents calling the entire review interval model execution time. The worker’s 45-minute initial interval is elapsed time too; resource usage needs a separate source.
Keep unfinished items visible alongside completed intervals. If another brief remains unverified at the end of the window, dropping it from the report can make delivery look faster while waiting grows.
Assay’s current flow reports derive time from recorded statuses. Their definitions and missing observations belong with the result so an owner can tell which interval supports the decision.
Count findings and examine their value
In the ledger case, a proposed hour-only reader can miss older refusals. The run spans several days, with its newest refusal one minute ago. Five consecutive refusals pause retries until 15 minutes after the latest, so this attempt must stop. A comparison test that makes the full reader refuse and the candidate permit supplies evidence for the finding.
If correctness and security both identify that same cutoff defect, count one underlying repair with both observations. After the worker changes the stopping rule and the comparison passes, retain the repair evidence with its disposition. That makes it possible to review the reviewer’s contribution later.
A repeated review can agree or disagree with the first assessment. Agreement is a calibration observation; the input, assertion and adjudication explain whether the finding was sound. Inspect reversals and later missed defects, rather than treating a majority of model opinions as correctness.
A review-method comparison should retain the candidate instructions and model version, use the same evaluation cases, and distinguish known defects from clean changes. Adjudicate additional findings and false alarms against the governing requirement. That makes changes to prompts, context and finding validation measurable alongside changes to the number of reviewers.
To evaluate an added performance or architecture review*, ask which additional substantiated finding changed the work and how much review, repair and human attention it required. This is a decision about useful marginal evidence.
Measure cost per accepted result
A complete cost record includes implementation attempts, reviews, test execution, repairs, tools and human decisions. Define the accepted result before dividing by it.
If an initial attempt costs 12 units and a repair adds 6, delivering one independently verified brief costs 18 units. Dividing by two candidate attempts would answer a different question and hide the cost of reaching acceptance.
Token records need the same care. If an input total contains 5,000 tokens, of which 4,000 were cached, price 1,000 at the uncached rate and 4,000 at the cached rate. Add output usage and the other recorded costs. Retain the usage source and price inputs used in the calculation.
Resource coverage determines what can be calculated. A clock derived from status changes establishes elapsed time. It cannot supply CPU time or unrecorded human attention.
Keep the benchmark attached to its workload
The 17 September verification reports a median guard time of about 905 milliseconds for the full read and 0.16 milliseconds for the bounded read, using a planted 109 MB ledger. A separate test checks that last-entry reads remain bounded as history grows.
That result supports the safeguard-performance requirement. It also preserves the important engineering question: do both readers reach the same permission decision? A fast reader that drops an old refusal has failed the work request.
Deskpost’s complete operation includes safeguards, the remaining duplicate-prevention lookup, posting and confirmation. Measure those boundaries to find what now controls the user’s waiting time. The historical guard benchmark supplies one component observation, rather than those other intervals.
Define the user-outcome population
The planned operational assessment would record the command version on each posting attempt and compare equivalent workflows. In the running example, include requested posting tasks with known command versions and comparable ledger sizes and task types. Keep retries attached to their originating tasks, and count command executions separately.
Suppose each version handles 100 comparable requested posting tasks. Complete-command medians move from 2.4 to 1.5 seconds. Four tasks on each version need one retry, producing 104 command executions per version, and no duplicate posts appear in either group. Traces locate the saving in the safeguard. The owner can retain the improvement and investigate the remaining full-history cost.
Change the input: the new version has only half as many failed tasks but also half as many requested tasks. The failure rate has not improved. Or suppose the new package has not reached the reviewers making those attempts: their results cannot establish the change’s effect.
The comparison therefore needs version exposure, actual use, an outcome denominator and preserved safeguards. If task mix or another changed stage explains the difference, gather a better matched comparison before selecting the next optimization. The operational article follows this assessment through its work-intake decision.
Use the result to choose an action
The worked handoff record suggests examining merge waiting. The historical benchmark establishes a guard improvement. The proposed exposed-use comparison would inform the next deskpost investigation. They concern the same brief, but they support different actions.
The current delivery feed uses recorded findings and reversions as failure inputs and main-branch CI recovery as its recovery interval. Those definitions help the owner interpret the report and identify the additional deployment or operational observations a different decision needs.
For an engineering lead, take one decision and trace its number back to the recorded events, workload or attempts. Then name the action the observation supports. That is more useful than adding another activity total.
*Performance and architecture reviews are planned.