A green check is useful when an engineering lead can explain what it inspected, which result it required and what would have made it fail.
Assay combines agent judgment with deterministic checks: programs that examine explicit rules. The reviewer decides whether the rules and evidence are adequate. CI runs the checks and consumes their actual outcomes.
Turn a wrong decision into a test
The running case is a bounded work request, Desk tools brief 24, which makes audit-ledger reads cheaper for commands including deskpost. Deskpost posts reviews after checking authority and safeguards.
One safeguard stops a write after five consecutive refusals for the same tool and target. Progress resets the run. An ignored event such as a dry run leaves it intact. The one-hour write-budget window does not expire that refusal run. The breaker pauses retries until 15 minutes after its latest refusal, then permits a fresh attempt.
Put four refusals in older history and a fifth one a minute ago, with no progress between them. The full reader must refuse the next write. An optimized reader that stops at the hour boundary sees only one refusal and can return the wrong answer.
The full reader supplies the existing behavior the optimization must preserve. The policy supplies the expected answer: the unbroken five-refusal run requires refusal. The comparison test asks both readers for the permission decision on that input. It must also test a successful progress entry that resets the run, so that always refusing cannot satisfy the requirement. Rotation tests move relevant records into yesterday’s file and require the same answer afterward.
Workflow triggers complete that coverage. A test reading a policy file protects changes to that file only if CI runs when it changes. The implementation, test inputs and workflow trigger have to agree.
Check whether the tests catch a fault
A test can execute a safeguard without asserting anything that depends on it. Muhar plants named text changes representing specific faults and runs the selected tests.
In this ledger case, one mutation replaces the valid stopping rule with the hour-only cutoff. If the old-refusal comparison is effective, that confirmed edit makes the test fail. Another mutation removes the full-read fallback; a fixture that exhausts the bounded reader before its answer is known must catch it.
Before interpreting those outcomes, muhar runs a clean baseline, checks a positive control that the suite must catch and establishes that each intended edit landed on disk.
Inspect the failure as well as its verdict: the intended behavioral assertion should fail, rather than an unrelated compilation error. A caught mutation establishes sensitivity to that selected fault.
CI must read those verdicts. A healthy harness can finish successfully while reporting a missed fault. Assay’s mutation gate requires a valid run with no missed or unplanted required mutations. A missed fault calls for investigation of the test and behavior. An unplanted edit or broken control calls for repairing and rerunning the experiment. The ledger brief’s historical verification records ten caught mutations.
Look for missing tests
I also reran muhar’s published worked example. Its small limit-checking package passed go test -cover ./... with 100% statement coverage.
Then muhar -j 1 -spec mutations.json planted the named faults. The harness was healthy, but its report ended:
Totals: 2 caught, 1 NOT CAUGHT, 0 could-not-mutate.
The rule permits a total exactly at the limit: 90 already spent plus a proposed 10 equals the allowed 100. The mutation changes spent + amount > limit into spent + amount >= limit, so that same request is wrongly refused. Existing tests check an over-limit total of 110 and an under-limit total of 60; neither asks what happens at 100. The statements execute, but the boundary decision goes unexamined.
I added the example’s boundary test and reran the same command:
Totals: 3 caught, 0 NOT CAUGHT, 0 could-not-mutate.
Both harness runs exited successfully. The report totals exposed the missing boundary assertion, then showed that the added test detected the fault. A CI gate reads those totals as well as the process exit status.
Ask five questions of every check
The mutation example exposes a general method. Identify what the check examines, the condition it asserts and the observation that proves its machinery worked.
For the ledger, the input is a specific history, the assertion is equal permission decisions and the fault is a shortened read. A missed mutation means the acceptance evidence needs attention before it can support the change. The same questions make other CI results easier to interpret.
Use quality history and ratchets
Qualgen turns repository history into signals for investigation: frequently changed files, earlier defects and files that tend to change together. Its quality report renders available churn, hotspot, ownership and instruction-reference measures. Its path checks advise authors about coverage, related files and stale references.
Suppose a budget-reader change has repeatedly needed a recovery-file repair. That history gives the author a reason to inspect recovery before declaring the consumer list complete. It selects an investigation; the actual comparison test establishes the required behavior.
A ratchet records an accepted baseline that later changes must preserve or explicitly revise. The forge-call register enforces a ceiling on permitted direct CLI launches. Retiring a permitted site lowers the ceiling. Adding one back without changing the baseline fails the check.
Other measures inform review: the weight report counts commands, flags, refusals and rule text; the regression-test ledger reports removed or renamed bug-reproducing tests and their retirement reasons. A regression floor names tests its runner must see explicitly pass. Together these keep a repair connected to the later command that is supposed to exercise it.
Check publication policy and scan controls
General secret scanners such as Gitleaks and TruffleHog detect credentials and other secrets. A publication policy also needs project-specific material: a private customer name or withheld reference can matter without containing a credential.
Assay’s leak-sweep supplies that registry, document extraction and controls around the scan. Its tool includes Gitleaks and TruffleHog integrations. Take a candidate PDF containing a private name in compressed text. Extraction makes that text searchable in a prepared scan tree: a searchable representation of the files being checked.
In the external-engine path, the harness first tests the registered rules on known positive samples. Separate canaries test the scanner’s built-in detectors. The scan tree also contains expected controls and per-file markers. A marker missing from the scanner’s findings means the harness cannot establish that the corresponding projected file was read.
Those expected control matches are distinguished from a withheld-name match in the candidate content. A live rule, observed controls and file coverage are prerequisites for interpreting an absence result. A withheld-name match fails publication; a missing control makes the examination unavailable and cannot authorize a clean publication. Restore the check and rerun it before relying on its result. The resulting certificate identifies the inspected tree by its paths and content hash, so changed bytes require a new scan.
This is the extra job around the matching engine: establish which publication rules were examined and whether the scan supplied the evidence required to call that material clean.
Check the relationship behind a claim
A claim can drift while the page remains readable. Site honesty pairs selected visible statements with executable source conditions.
One registered example is the Apache 2.0 license statement. Its check requires a license file containing the Apache license and version text. If the page still makes the claim but that source condition stops holding, the check fails. If the sentence disappears, its old anchor becomes stale and requires reconciliation.
That predicate is concrete: it checks the registered license relationship. Other statements need their own conditions. The fixture suite exercises missing pages, stale claims and broken anchors so those outcomes do not become clean results.
Publication also has checks for listed overclaim phrases, placeholders and shipped-content hygiene. Each gives the editor a specific finding to resolve; the source-linked statement and its predicate make clear which claim was examined.
Keep the baseline under examination
PR checks run when their triggers match a change. The truth-suite workflow also runs on main pushes, daily and by manual dispatch.
Its corpus leg builds and vets the discovered Go modules and runs the plugin shell suites. Its mutation jobs exercise selected guard specifications. Existing obligations therefore get another examination between relevant delivery changes.
For an engineering lead, start with one consequential safeguard. State its expected decision, provide an input that a tempting shortcut gets wrong, and test whether the suite notices that shortcut. Then make the consuming CI gate interpret the result. That establishes a useful check before expanding the inventory.
Disclaimer
The publication workflows currently use the original pattern matcher. The Gitleaks and TruffleHog integrations described above are available alternatives. In that mode, Gitleaks checks files selected for publication against Assay’s disclosure rules, and checks a separate file of planted sample secrets against its built-in rules.