Computer-use and web agents

An RPA script replays fixed steps in a static environment and breaks the day the button moves. A computer-use agent re-perceives the current screen and adjusts. That is the entire claim, and it is why the same vendors who once sold recorded scripts are now selling perception: the maintenance cost of encoded assumptions about someone else's UI is what made the previous generation unattractive.

The DOM-and-pixels camp parses markup where markup exists (hidden elements included) and falls back to vision. The perception-first camp argues the browser is fundamentally a renderer, its pixels are the source of truth, and hand-engineering DOM readers is the wrong bet because the long tail of the web will never grow APIs. The strongest single exhibit for the second position is a stock selector whose markup carries no quantity information at all, because availability is fetched as JSON and rendered into grayed-out pixels. Neither camp is pure, though: the perception-first system keeps an execute-JavaScript action that makes it superhuman at form filling.

Both camps report specialized smaller models beating frontier general models on this task at a fraction of the cost, against a frontier comparison of roughly ten seconds per action and $2.30 per task. And both position the enterprise value as control, traceability and humans in the loop rather than autonomy: the difference, as one put it, between a demo and a production system.

Also: their blog post, "The bitter lesson for web agents", and the Navigator model's execute-JavaScript action and "agentic API-fication" demo, answering "does this promo code work" with a mock checkout and a price diff.

Assay's desks live inside forge and CLI tooling where real APIs exist, so the central argument (drive the UI because there is no other way in) does not bite. There is no computer-use agent here and no plan for one.

What transfers is methodological: when an interface resists programmatic access, prefer the general perception channel over a hand-built scraper, because the scraper encodes assumptions about a surface you do not control. The nearest instance is browser-driven evidence gathering for interface briefs, where the choice between a screenshot path and a declared tool contract is live, but that thread is tracked under environment priming and substrate readiness, since it is about making a surface agent-ready rather than about driving one that is not. The published cost figures are also a useful calibration point for any future work that considers driving a browser at volume.

• Whether the benchmark saturation claim holds against a harder successor benchmark, or whether saturation reflects the benchmark rather than the capability.
• Whether the DOM-parsing and pixels-only camps converge, and on which side: the JSON-rendered-to-pixels example is the strongest argument yet for pixels, but the execute-JavaScript escape hatch suggests neither side is pure.
• First-hand production deployments with duration and failure rates, rather than POC timelines and benchmark placements; that is what would move this to established.