Model tiering and routing

The routing decision is made on task shape, not on convenience. Messy extraction, ambiguity and judgment get the heavy model; mechanical checks, navigation and bulk generation get a cheap or local one. What makes this survivable is the layer in between: an abstraction that keeps the application from knowing which tier answered, so a model can be swapped, blended or retired without the stack falling apart. Deterministic code is a legitimate routing destination alongside the models, the cheapest correct answer is often not a model at all.

Two guardrails recur. A capability floor: a weak model may be delegated to, but never put in the driver's seat of a loop that holds authority. You never want somebody stupid to be powerful. And budget policy as governance: spend is managed by session-aware soft checkpoints and downgrade gates rather than hard stops, so the run degrades rather than dying. Privacy joins cost and latency as a routing input once local models are in the mix: some queries are routed by where they are allowed to go, not by how hard they are.

The strongest results are not single-model at all. A measured mix beats every constituent, and a junior-generates / senior-decides ladder beats frontier-only at a fraction of the cost. Alongside it sits adaptive test-time control: how much compute a step gets, thinking tokens, retries, which workflow to run, becomes a runtime decision under an explicit budget instead of a constant frozen into a prompt. Because pricing, drift and new models move continuously, the mix is re-optimized against production traffic rather than chosen once.

Alibaba Cloud International presents Smart Fusion as one model endpoint that picks a model at every LLM call inside a task, not once per prompt, because most sub-steps of even a hard task are simple. A small, fast decision model that does not generate long outputs makes each choice, which keeps the per-call overhead affordable; the few steps the speaker calls critical go to several frontier models whose results are combined. The claimed price is half the frontier model’s list price (7:56). One unnamed customer’s slide shows an 88.17% pass rate on key issues against 87.93% for an unnamed frontier model, at about 60% estimated inference savings (11:03): a 0.24-point gap with no stated sample size or variance, and no comparison with the customer’s former hand-assigned setup. The benchmark is self-run, the customers are unnamed, the cost of fusion is not given, and prompt-cache effects are not discussed. Partial corpus: these talks do not represent the full conference.

Vercel describes choosing a stronger model for planning and cheaper models for bounded execution through a gateway. Dropbox reports automatic task routing in its internal platform and plans a layer for third-party tools. Neither account gives a controlled routing cost-versus-quality comparison. Recordings require registration; transcripts were generated locally because publisher captions were unavailable.

Presenters distinguish the always-on coordinating bot from a separate coding harness and demonstrate comparing several models for architecture work. They also decline the extra planning cost for a disposable prototype. This is task-dependent routing, without a measured quality-adjusted saving. Recordings are third-party re-uploads. Individual workshop dates within the event are not established.

Also: the Lemonade semantic router, FrugalGPT, Berkeley RouteLLM and Plano as routing prior art; Mercury and its Augment Code deployment; QLoRA, Unsloth, Qwen 4B, GLM-class models and Gemma from the fine-tuning talk.

"Can you swap models without the stack falling apart" is the same requirement Assay met by keeping desk verbs and skills harness-portable rather than binding them to one vendor. On the same evidence base Assay has declined fine-tuning as an investment: skills rot slower, port across harnesses, and review as diffs. Two smaller carries land here too, cache-aware stickiness argues against swapping models mid-session inside a desk loop, and "plug in your own eval, not the public leaderboard" restates the position that external benchmarks do not measure desk outcomes.

Which desk roles may run on a cheap model is decided ad hoc today. The frontier supplies a crisp rule worth adopting: mechanical loops, intake sweeps, board regeneration, documentation fixups, may delegate down; anything holding authority may not. Verify verdicts, review verdicts and any flip of a status row sit above the floor. That is a project-layer value rather than something a neutral role definition should carry, which is precisely why it has drifted.

Routing at every LLM call is finer than anything Assay does: desk sessions run inside a harness that owns model choice, and the cache-stickiness caution above is the first question to put to any per-call scheme. The router itself is a small decision model used for one bounded choice, which model serves the next call, a narrow scope that keeps such a model away from verdicts. The boundary runs the other way too: an endpoint that hides which model answered cannot serve an authority-bearing role, because the capability floor depends on knowing the model.

Briefs already carry effort estimates, but no desk declares a token or session budget, no soft checkpoint asks before spending the next tranche, and Assay’s own reporting ignores cost entirely: so desk economics are currently ungovernable. Cascades are closer than they look: worker → reviewer → human is the junior/associate/partner ladder, but it was designed for correctness, not cost, and has never been measured as one. The standing caution applies directly, continuous cost-optimization erodes exactly the checks that never show up in the quality metric, so cost-optimizing the guardrails away is the failure mode to pre-commit against.

• Whether anyone publishes a measured capability floor, the task classes at which a cheaper tier starts failing a specific authority-bearing role, rather than the qualitative "don't let the weak model drive" rule.
• Whether budget policy shows up as a harness primitive (soft checkpoint, downgrade gate, per-session cap) in a shipped agent framework rather than a workshop demo.
• Whether continuously re-optimized model portfolios report a guardrail regression alongside the cost win, the failure mode nobody has yet measured.