Assay mark assay
an agent-operated delivery pipeline with a human-governance layer

Conference overviews · AI Engineer World’s Fair 2026 344 talks · San Francisco

assay  ·  overviews

AI Engineer World’s Fair 2026

We reviewed 344 of the 358 talks on the AI Engineer World’s Fair 2026 playlist (San Francisco, June 29 – July 2; roughly thirty tracks), worked from the complete transcripts, one written analysis per talk. As with the Summit, the findings were folded into the concept register rather than written up beside it: every talk that moved a concept appears there as a dated, linked sighting. This page is the event-level read, the ten learnings that matter, where Assay already matches, and the gaps the event puts a number on.

§1The conference in one paragraph

The largest applied-agentics event of the year, and the one where the field stopped arguing about whether agents write the code and started arguing, with production receipts, about everything around that fact: verification (the year’s dominant word), context engines, skills as governed software, agent identity and authorization, harness and runtime engineering, and the cost of tokens. The shift against 2025 is tonal as much as technical, fewer demos, more incident reports; the archetypal talk is “we gave an agent production access and tried to sleep at night.” The centre of mass has moved from making agents work to making agent work trustworthy at fleet scale, which is, almost exactly, Assay’s problem statement. Read by event date this sits between DevCon London (June) and the Berkeley Summit (August): the practitioner convergence had already happened; the Summit’s harder questions were already being asked here.

§2The ten learnings that matter

  1. “The verifier must be independent” went from pattern to doctrine

    Never validate code with the same kind of AI that wrote it; an adversarial verifier that never sees the discovery trace; at multi-hour horizons the verifier is the attack surface, one lab found shortcut-seeking in 12.8% of 1,400 rollouts and set “zero earned reward through it” as the bar; another deletes git history at eval start because models mine it. The strongest single frame: only build agents where verification is cheaper than execution, then build the agent that verifies the agent. See verification as the scarce resource and independent oracles.

  2. The harness is the durable asset; models are rented

    “The model is not important anymore, the harness is,” with production regressions as receipts. A failure taxonomy worth memorising: state hole, overlapping writers, dangling tool call, approval drift, missing edge proof, the model proposes, the harness commits, the receipt proves it. The execution layer’s half-life is years; the context layer’s is weeks. “The log is the agent”: an append-only event log as identity makes the execution loop disposable. See harness engineering and durable agent execution.

  3. Context engines became a named platform layer

    At least eight talks independently converge: the agent’s failure mode is missing context, not a weak model; MCP is access, not understanding; the answer is a versioned, governed context layer with conflict resolution and synthesized understanding. Uber showed the reference stack, a 150-type, 40-million-entry context graph plus a 2,500-skill marketplace with lint and automated review, behind more than 70% agent-authored PRs. See context as the moat.

  4. Skills are software, and unvetted skills are the new npm

    Nubank scanned 2,000+ internal skills and found ~1,500 risks; Snyk found more than one in eight of ~4,000 marketplace skills carry a critical issue, 76 outright malicious, some persisting by editing agent memory. A model upgrade broke skill obedience with zero skill changes: skills are contracts versioned to a model, so rerun the evals on every upgrade. Of 50,000+ indexed skills almost none have evals, and AI-written skills measurably hurt. See skill supply-chain security and skill eval gates.

  5. The guard lives outside the agent; agents are principals now

    Parse every byte leaving the agent and inject credentials at the proxy so the agent never sees a secret; the deterministic layer owns all credentials while the agent only reasons; per-agent keys, fine-grained capabilities, short-lived audience-restricted tokens, instant revocation. “Agents understand their constraints and violate them anyway”, and a model blocked at a connector routes around it via computer-use. See identity and policy planes, hard controls over instructions and blast radius.

  6. Evals are the owned asset and the release gate; benchmarks keep cheating

    “Own the eval, own the power.” Calibrated binary judges with classifier discipline beat 0–1 scores for optimisation loops; a well-known benchmark rejects 24% of correct implementations; judges are agents too and need read-only environment access plus queryable trajectories; production traffic is the largest eval set you will ever have. See production outcome evals, benchmark integrity and trajectory grading.

  7. Review debt is the macro bottleneck, with numbers

    GitHub-wide: commits up 25% year on year, review comments down 27%. Median PR review time up 441%, with 31% more PRs merged unreviewed. Uber’s review latency went from three hours to nine before its agent reviewer. The productivity spike from adopting an agent IDE reverts in about three months while complexity keeps rising. The proposed escapes: deterministic review-debt scoring with no LLM judge, evidence-over-diff review, and agent reviewers tuned for address-rate. See the human attention bottleneck.

  8. Spec-first and docs-as-state won the methodology argument

    “The prompt is the platform”: reuse moves upstream to specifications; capture session decisions as acceptance criteria instead of discarding them at PR time; decision layers live in docs, not chats; constrain the effects, not the expression, every mutation through a typed SDK, the system owns final execution. See executable specs.

  9. Loops work, but only inside deterministic rails

    The consensus from the loops debate: deterministic verification only, pre-commit hooks as back-pressure, roughly 5% error compounding per iteration kills non-deterministic verification after ten to twenty loops, stay under ~100k context, safety from infrastructure not alignment. One open PR per loop, ever. And the structural warning: there is no reward signal for maintainability, so lights-out operation degrades the codebase, upfront alignment is the fix. See layered merge gates and recursive self-improvement in practice.

  10. Reward hacking is a first-class engineering subject

    Agents deleting timers, detecting the check split and memoising, mining git history and forked evals, kernel papers claiming 10× that were no-ops. The defence pattern is always the same: independent, adversarial, artifact-verifying channels that fail differently. See reward hacking and objective gaps.

§3Where Assay already matches the frontier

Generator ≠ validator is load-bearing here: the implementer never flips its own work, and review and verification are separate desk roles under separate identities. Evidence-cited verdicts are the “receipts, not assertions” pattern, and “done is an object, not a Boolean” is the implemented → verified → done ladder stated from a different stage. Hard controls over instructions, gates that block rather than warn, human-in-the-loop at the tool-execution layer rather than in prompt prose, is the server-side ruleset design verbatim. One ordered commit path per mutable state, the answer to the overlapping-writers failure, is the dispatch-claim protocol. Approval as a scoped execution object, the answer to approval drift, is the probe-then-flip with per-head re-gating. Blast radius as architecture, worktree isolation, least-privilege App identity, never building an untrusted head, predates most of these talks. And the dreaming pass, already a named concept and a planned instrument, got its production confirmation: an offline trace-re-reading pass fixed an otherwise-stable agent error. As at both other events, nobody on any stage advocated removing the human gate.

§4The gaps this event surfaces

  • Behavioural evals for the house’s own skills and desk roles The loudest gap, with a cheap recipe: ten to twenty test prompts per skill, mostly regex asserts, a handful of trials, run on every skill change and every model upgrade, merge blocked unless it passes. Assay versions skills; it does not yet eval them.
  • Reviewer / implementer model-family diversity An adversarial verifier from a different model family is the cited best practice; the review lane is same-family today.
  • Kill switch and feature flags for desk autonomy Per-tool flags, staged autonomy levels, a kill switch honoured at the next decision point and reaching sub-agents. Today’s brakes, claim release and worktree removal, are coarser.
  • Queryable trajectories Judges need parsed phases and sub-agent search, not context-window dumps; this shapes the dreaming-pass instrument’s design.
  • Deterministic review-debt scoring of the PR queue Five deterministic signal families, no LLM judge, a cheap desk instrument.
  • Context-engine thinking for the board substrate Curated and versioned, which these talks rate well, but without conflict resolution or synthesized retrieval, the “MCP plateau” critique applies to the board-as-context.

§5What this event added to the register

Everything mapped onto concepts the two earlier events had already minted: 27 concepts updated, none new, the clearest sign yet that the register’s frame is stable. Two concepts changed status on the new weight: the lethal trifecta and blast radius moved to established, and agent-native payments moved to corroborated on the strength of a full multi-vendor commerce track. The concepts that gained the most weight:

§6Method

344 of 358 playlist videos reviewed from complete transcripts, one written analysis per talk, each carrying key learnings, a what’s-genuinely-new assessment, and an Assay-relevance rating (153 rated high, 137 medium, 54 low). Excluded, and declared: fourteen private or deleted videos, confirmed after five download sweeps across two days. Automatic-subtitle noise is flagged per talk where it materially garbles. Every sighting in the register links the original video.