How Assay measures delivery and learns from defects

An agent can produce more code while the delivery system gets slower. Reviews can catch more defects while reporting a worse first-pass rate. A release can pass every acceptance check and still leave the user's problem unsolved.

Assay is an agent-operated software delivery pipeline with human governance. AI agents implement work and separate agents review and verify it; people retain the consequential decisions. Its current delivery boundary ends with verification on the main branch. Release, deployment and customer-outcome measurement require additional integration.

We collect metrics to distinguish those situations and decide what to change. The mechanism is an evidence chain: connect the work request, delivery events, defect evidence and follow-up decision so the conclusion can be inspected.

Consider an illustrative checkout repair. A user retries a request and receives two orders. We need to know whether the repair waited for review, whether the retry requirement was clear, which change may have introduced the defect, and whether users of the repaired version stopped encountering it. Each question needs a different part of the evidence chain.

Several parts already run: delivery reporting, history mining, defect tracing and the public AssayScore calculation. Other capabilities exist as reusable analysis libraries. The broader cycle that diagnoses a problem, tests an intervention and assesses its effect remains under development. Understanding those boundaries is part of using the metrics well.

What we measure

A brief is a bounded work request with requirements and acceptance checks. It gives us a more useful unit than a commit: one brief can require several implementation attempts, reviews and repairs before it is accepted. Pull requests contribute to that work; their count alone cannot tell us whether it succeeded.

Our instruments answer different questions, with different populations and denominators:

The last row needs application-specific observation. Assay does not infer customer benefit from a completed brief, and installing the methodology does not install analytics inside an adopter's product.

AssayScore summarizes part of the delivery picture through Speed, Flow, Quality and Value. It is useful as an entry point into its component evidence. Defect lineage, operating cost and product outcomes still need their own reports.

How the evidence is collected

Workflow history

The status historian records brief transitions with timestamps. Those records feed statusgen, Assay's command-line tool for generating the work-status board and delivery reports from repository records. The board is a generated view of work and its recorded state. The reports derive completion counts, elapsed lead time, time spent in each stage and related flow measures. Historical replay can recover transitions from retained Git history, subject to what that history contains.

For the checkout repair, the history can show when its work request entered implementation, became ready for review and reached completion. Verification means a party other than the implementer checks the stated acceptance conditions and records the evidence.

The events establish when a recorded state changed. They do not establish how much CPU an agent consumed or how many minutes a person actively worked. The current AssayScore Flow component, for example, treats time in in-progress as a proxy for touch time and selected other states as waiting. A brief can remain in progress while nobody is working on it. Explicit activity spans are a separate measurement capability.

The same distinction applies to verification. A dated evidence record makes a claim inspectable and attributable; stronger execution assurance depends on the mechanism that produced and corroborated it. A status label alone does not prove that a command ran in the claimed environment. The lifecycle specification states that boundary.

Forge events and daily reports

Issues, PR bodies, review objects, commit identities and CI run conclusions supply another part of the record. Explicit links between work and its PR matter: without a resolvable link, a collector cannot safely attach a review outcome to a brief.

Our internal daily harvest captures reports for review and comparison. These include queue flow, merge waiting, review cycles, verification rework and post-merge CI failures. Different reports intentionally use different intervals. Authored-to-done lead time answers a different question from implemented-to-done time or PR-open-to-merge time.

A separate snapshot collector preserves open PR and issue counts that would otherwise be difficult to reconstruct for an earlier day. Its reducer consumes snapshots from the same run, records source coverage and produces daily roll-ups. A partial source set cannot silently publish a lower total that looks like an improvement.

Collection and reduction are distinct responsibilities. Collection records what could be read, including errors and listing limits. Reduction computes a defined measure from those inputs. That separation helps us distinguish a workflow change from an API failure or a changed reporting population.

The outcome reducer also supports merge rate, zero-developer-touch merges, review-round distributions and terminal PR categories. These need richer PR timelines that the scheduled snapshot collector does not currently supply; the scheduled report therefore leaves them unavailable. A reducer's existence does not establish an operating measurement.

Their definitions illustrate why that richer collection matters. Merge rate compares merged with closed-unmerged PRs, excluding qualified superseded duplicates. Zero-developer-touch means no non-bot commit after the first bot commit; it does not mean no person reviewed, advised or approved the work. Review rounds describe the repair loop, while terminal categories distinguish merges after churn from changes closed before or after review. A dispatch-to-merge calculation reaches further upstream to include work that never produced a PR, but its local work-assignment record and incomplete open-PR visibility limit what it can conclude.

Repository quality history

qualgen is Assay's command-line repository-quality analyzer. It reads Git history and produces local reports and data files about change patterns and potential defect origins. Its basic inputs do not require an Assay-managed repository. It can examine how changes survive, which files experience repeated churn, where knowledge is concentrated and which files repeatedly change together.

For the checkout case, statusgen answers how the repair moved through delivery; qualgen can help investigate the code history behind the duplicate-order defect.

Supplying recorded issue and PR metadata enables defect tracing. The snapshot identifies defect-labelled reports and their fixing changes, binds the data to a repository HEAD, and retains the evidence used to classify each fix. The miner runs locally; it does not implicitly collect tracker data or search an operator's sessions. The qualgen guide documents the separate collection step.

Local resource and operator evidence

Some observations exist only on the machine doing the work. Tool performance records can expose elapsed time, recorded CPU use and counted forge requests. Operator reports can describe intervention frequency or decision waiting. Neither a CI runner nor a repository clone automatically has those inputs.

We keep their units separate. Waiting for a decision is elapsed delay. An intervention count describes interruptions. Tokens describe model usage. None supplies measured human minutes or monetary cost without additional evidence.

The existing cost-per-passed-acceptance-check report illustrates the limits. It joins a point-in-time record of agent sessions and assigned work to machine-readable check results. Those session totals and current assignments do not describe the full set of attempts that produced an accepted change. The result is a bounded diagnostic with substantial sampling limits. It is not yet a reliable cost-per-accepted-brief measure.

Missing data is part of the result

A useful report must distinguish a measured zero from an unavailable observation. Zero failures among observed cases, no eligible cases, and a failed source read are different facts.

The instruments use explicit states such as could-not-check or could-not-measure. Cross-repository roll-ups also distinguish an unconfigured source from an incomplete one and withhold totals when coverage is insufficient. Comparable day-over-day totals require the same contributing sources.

These protections are not identical across every instrument. Some legacy metrics exclude unreadable or unlinked records and still emit a result for the remaining sample. Read the denominator and exclusions alongside the number. A measured result is a statement about that qualified sample, not a promise that every relevant event was observed.

Freshness matters too. A successful collector process can emit an unavailable measurement. A saved report can remain readable after its underlying source becomes stale. We retain those distinctions so an operator can investigate the instrument before acting on its output.

How we use the metrics internally

Daily reports help us choose where to investigate. Queue age and unfinished work show whether delivery is accumulating at intake, review, verification or a human decision. Completed-item latency adds a second view, but it can improve while the oldest unfinished work remains stuck.

Consider an illustrative week in which implementation time falls while approved PRs wait longer for merge. Adding workers could increase the waiting queue. The useful response is to inspect the merge handoff, its constraints and the work already awaiting it. If review instead dominates, we examine the findings, repair cycles and reviewer coverage before changing review capacity.

Rework deserves similar care. A request for changes can represent a valuable catch. A verification failure can reveal a weak implementation, an incomplete brief, an obsolete check or an unavailable environment. We retain the underlying events so a higher repair rate prompts diagnosis rather than pressure to approve more work on the first pass.

Our reporting now includes a separate verification-rework calculation using delivery transitions, verification failures and send-backs. It deduplicates repeated outcome records, identifies bulk status resets and keeps rework outside the delivery denominator visible. Older DORA-shaped reports can still carry a different or unavailable rework definition. The name alone is insufficient to compare them.

DORA-shaped delivery measures also need their stated boundaries. Brief completion is a delivery proxy; it is not a production deployment. Main-branch CI recovery is not the same interval as restoration of a customer service. An adopter with deployment and incident records can measure those operational events directly without relabelling our workflow proxies.

For code improvement, a hotspot or repeated defect trace nominates a surface for investigation. The owner still has to establish why it is brittle. Frequent changes may be appropriate in a rapidly evolving module; concentrated ownership may reflect a small team. The report narrows the search and makes the competing explanations easier to examine.

How internal and end-user bugs enter the record

The starting point is a recorded problem with enough evidence to investigate. Internally, that can be a review finding, a verification failure, a CI alert or a reproducible tool failure. For an end user, it usually starts as a support report or tracker issue describing the affected version, expected behavior, observed behavior and reproduction conditions.

Those sources do not all represent confirmed defects. Triage must distinguish a defect from a changed requirement, an environment failure, a duplicate or an unresolved report. In an adopter's project, the product team owns that judgment and the support-to-tracker integration. Assay's portable tooling can consume the resulting records; it does not supply a universal customer-support ingestion service.

The report belongs with the component that needs repair. A defect in an Assay tool belongs in that tool's issue workflow; a defect in an application built with Assay belongs with the application owner. A process failure can require a separate methodology change, linked to the concrete incident. Our internal code-repository workflow records a resolution claim and evidence in a reviewed PR linked to the issue. The temporary resolution document can later be cleaned up; the issue and merged PR remain the durable record. A customer confirmation or later operational observation is separate from that delivery closure.

For a user-reported defect, retain the report even when the inducing change is unknown. For an internally caught defect, retain where it was caught. That lets a team distinguish prevention during delivery from an escape into released use without assuming every internal failure affected a customer.

Return to the illustrative duplicate-order report. The fix links to the defect report, and a trace identifies a candidate change to retry handling. If the original requirement omitted retry safety, specification attribution deserves investigation. If the requirement included it but the brief dropped it, the brief is the likely process gap. If the brief required it and the code violated it, implementation is the likely gap. The report's end-user origin stays the same in all three cases; where a defect was discovered does not determine where it was introduced.

Tracing the change that introduced a defect

qualgen recognizes fixes from several evidence tiers: a closing reference to a defect-labelled issue, PR taxonomy, or a weaker keyword fallback. The tiers remain visible. A commit containing the word “fix” is a candidate, not an independently confirmed bug.

Its B-SZZ tracing follows the lines changed by a fix back through Git history to candidate inducing commits. This produces a useful lineage to inspect. It can be incomplete or misleading around refactoring, omitted code, incomplete history and changes whose causal relationship is not captured by line ancestry.

Reports therefore carry trace coverage and evidence tiers. A failed trace remains untraceable. Where supplied, the full-history merged-PR count provides a denominator; a partial tracker export must not masquerade as complete coverage. The inducing-commit-to-merged-PR ratio can exceed one, so it must not be presented as a probability that a PR is defective.

Attributing the process stage

Change lineage and process responsibility answer different questions. A defect may originate in a requirement that was faithfully implemented, a brief that omitted an important case, or code that violated a sufficient plan.

The existing attribution library assembles a dossier containing the relevant trace, requirement or brief context, inducing diff and review evidence. It records a stage judgment against the dossier's content hash:

The default classifier is deterministic over supplied dossier fields. An optional model can refine a traceable judgment, with a recorded rationale. Neither the classifier nor a model turns incomplete semantic evidence into causal proof. These are judgments over assembled evidence that need inspection and spot audits. The library retains amendments rather than silently replacing earlier attribution history. The stage classifier makes the rules explicit.

A separate review-escape record names the review lanes—distinct review roles, such as code review and security review—that approved the inducing candidate. That preserves both questions: where did the defect originate, and which review process let the change through? Approval is an observed event; whether a reviewer had a fair opportunity to detect that particular defect still needs assessment.

These attribution components exist, but the basic qualgen mine command does not automatically assemble a complete process dossier for every repository. Adopters must supply the provenance and review integration needed for those richer analyses.

Reporting post-merge failures

Our internal CI attribution report identifies episodes from run conclusions, then examines the changes between the last observed green run and the first red run. It distinguishes a single candidate change, a pair, a larger range and no attributable culprit.

This prevents a convenient but unreliable rule: blaming whichever PR happened to be at the tip when the failure became visible. Two individually passing changes can conflict when combined. A workflow can miss the relevant push because of its path filters. A runner can fail without a code defect. Issue open and close times also need not match the actual failure and recovery interval.

The reports retain those attribution classes alongside rates by delivery route and workstream. They inform investigation of the delivery system. They do not establish personal fault, and a CI failure count is not an end-user incident count.

How a project built with Assay can use this

Start with one decision your team needs to make. If the problem is slow acceptance, collect the relevant handoff history. If it is recurring defects, establish reliable defect reports and fix links. If it is customer task failure, define the affected task and version exposure before collecting another delivery counter.

For an Assay-managed repository, the existing statusgen interfaces expose the delivery components directly:

statusgen --root /path/to/project --throughput --json
statusgen --root /path/to/project --leadtime --by-size --json
statusgen --root /path/to/project --flow-efficiency --json
statusgen --root /path/to/project --first-pass-yield --json
statusgen --root /path/to/project --assayscore --json

Use a released version supporting those modes. The repository needs the corresponding history and work records; forge-backed measures also need correctly scoped read access and resolvable PR links. Running a command successfully does not guarantee that its inputs were sufficient to compute every component.

For repository quality, an initial local run can use:

qualgen mine --repo /path/to/project --out /path/to/quality-tracking
qualgen report --out /path/to/quality-tracking

Add a recorded --fix-source snapshot when ready to analyze defect lineage. Keep the tracking root outside the target repository unless you deliberately choose in-repository artifacts. Richer stage attribution, session forensics and learned scoring need their documented library integrations and qualified inputs.

An application team can then add a small outcome record. For a checkout repair, for example, define successful checkout attempts among users exposed to the repaired version, with duplicate orders and support burden as balancing observations. Link the repair and release to that population. A decline in complaints without an exposure denominator cannot establish that the repair worked.

Publication is a separate choice. Our website consumes an aggregate metrics snapshot prepared for that audience. Private work titles, raw transcripts and internal evidence do not belong in a public feed because a collector can read them. Adopters can retain their own records and publish only the aggregates they have authorized; using the tools does not require contributing a shared training corpus.

What machine learning already adds

There is already a learned defect-risk component in qualgen/riskscore. It is a library, not a claim that every Assay deployment is running a trained predictor or automatically changing its workflow.

The implementation fits logistic regression over change features such as how widely a change is spread, its size, prior change and defect history, and author-class or experience fields. It retains a separate heuristic decomposition based on hotspot position, traced defect density, ownership concentration and missing change-coupling partners. That decomposition is useful context and a fallback; it is not an explanation of the learned model's individual feature contributions.

Training respects when a defect became known. A bug discovered in November must not inform a simulated prediction made in September. The training set contains earlier changes and only labels available at the prediction cutoff. Evaluation compares the learned model with the heuristic on later held-out changes and carries sample size and trace coverage with the result.

The default training floor currently follows the feature count: 15 predictors times 10 gives 150 training rows, with both classes required. That is an implementation guard, not a guarantee of statistical adequacy. Rare defects, delayed discovery and incomplete traces can still make the evidence weak. With insufficient data, the component returns the heuristic and a could-not-learn reason. The learned-risk implementation exposes those choices.

A useful risk model would help direct attention: which change warrants deeper inspection, which historical defect family deserves a regression check, or which dependency needs review. A probability estimate does not approve a change or remove a required control. Ranking performance also needs to be distinguished from probability calibration and from demonstrated operational benefit.

The future feedback cycle

Assay calls its standing operating roles “desks”: intake organizes work, workers implement it, review and verification roles check it, and a coordinator manages handoffs. The planned desk-improvement capability connects measurement to a bounded experiment and an independent assessment. Its design remains a proposal with staged implementation and adoption. It defines how to assess intake, implementation, review, verification and coordination without pretending their different denominators can be added into one universal quality score.

A review-method experiment shows the intended cycle. First, identify a material cost or miss from recorded evidence. Then compare a proposed method with the current one on frozen cases, including known defects and clean changes. Preserve model and instruction versions, timeouts, retries and missing usage. An independent assessor checks useful findings, false alarms, missed defects and total review-and-repair burden. An authorized owner decides whether to adopt the change, and subsequent exposed work supplies the keep, revise, revert or hold decision.

Machine learning can contribute predictions and help prioritize investigations within that cycle. The evidence still has to establish whether acting on its advice improved the outcome. A better held-out ranking score can coexist with excessive false alarms, expensive reviews or no measurable reduction in escapes.

The same cycle should apply to products built with Assay. A subject owner identifies the beneficiary and desired condition; a bounded change records its expected effect; observations follow actual exposure. The owner can then choose another repair, a better measurement, an investigation or no change. Forecasts retain their original question and deadline so later revisions cannot turn an unsuccessful prediction into a successful one.

Our proposed adaptive-learning extension also asks when the governing rule needs reconsideration. Repeatedly fixing individual items may leave an assumption in the brief template, review policy or product target untouched. Evidence should be able to nominate that assumption for review. The owner may retain it, revise it or stop the approach; a model's recommendation supplies no new authority.

For the next stage, the important work is reliable joins between original work, attempts, defects, decisions and observed outcomes. Those joins need known coverage, permission to use the data and enough comparable cases to assess a change. We intend to use that evidence to improve both delivery and work selection, while retaining the ability to conclude that a proposed improvement did not help.

For a first application, retain one repair's report, fixing change and acceptance evidence, then define the exposed users and outcome you will check after release. That makes the next decision inspectable even before a learned model is available. The companion AssayScore explanation shows how a smaller part of this record becomes a composite delivery indicator.

Disclaimer

This account describes the inspected November 2026 implementation and proposed direction. Examples are illustrative, not measured operating results. Library availability does not imply a deployed integration, and a defect trace or stage judgment does not establish causality. Product telemetry and the proposed feedback cycle require the additional work described above.