The 89% local-AI result is a ceiling for a router to approach
Best-of-local coverage selects successful models after evaluation. A deployment must show how it routes and detects misses before claiming the same coverage or savings.
An oracle can choose the model that answered correctly because it has the answer key. A production router has to make a useful decision before it knows the outcome.
That is the gap to keep beside the 88.7% figure in Intelligence per Watt, revision five. The paper reports best-of-local coverage across more than twenty models and explicitly describes the advantage of selecting among them. It is evidence of complementary capability, not a measured coverage rate for a deployed routing service.
I would treat the result as a reason to build and evaluate a router. I would not put it straight into the capacity plan as the share of requests we can safely keep local.
Selection has to happen without hindsight
A best-of-models evaluation can answer a valuable research question: does this pool contain a successful answer for this task? The operating question is harder: can our system select a suitable model at an acceptable cost, and recognise when its choice has failed?
As an illustration, suppose I build a hypothetical local-first support assistant. A local model handles an apparently easy question but omits an exception in the policy. The cloud fallback is useful only if the system identifies the miss, or if another rule sends the request there regardless.
Counting the request as recoverable after an evaluator spots the exception does not demonstrate that the live workflow would have escalated it. The missing component is an operational detection decision, with its own errors and resource requirements.
I would evaluate routing and fallback together. The test should include requests sent unnecessarily to the cloud, requests incorrectly accepted locally, and requests whose second attempt still fails. Those categories reveal different costs that an overall routing-accuracy figure can conceal.
Simulated savings are a scenario
The paper separately simulates a hybrid system with four small local models and a cloud model. Its 80%-accurate routing scenario reports 59.0% cost savings against the specified cloud baseline. Those are simulated economics under stated assumptions, rather than measurements of a router operating that traffic in production.
That distinction improves the experiment we can borrow. Change the assumed routing accuracy and inspect how the result moves. Then replace the assumptions with measurements from the intended task distribution, hardware, serving configuration and fallback policy.
For my hypothetical assistant, I would first run the proposed router in shadow mode on authorised, representative requests. Keep the answer acceptance test independent of the routing decision. Measure the resources spent on every attempted path, including retries and verification.
A second pass should examine the misses by consequence. A harmless wording issue and an omitted eligibility restriction do not deserve the same operational treatment. The point is to choose an acceptable boundary for this workflow, not to reproduce a headline percentage at any cost.
The research's main coverage result concerns single-turn chat and reasoning. It also includes an agentic extension, but that does not establish a deployed router's behaviour for every multi-turn workflow. Capability, selection and serving economics still need to be connected in the actual application.
Local models do not have to cover almost everything to be useful. A well-identified subset with reliable escalation can justify the integration. The strongest implementation claim is the subset we can recognise in advance, with the errors and fallback costs included.
Best-of-local coverage is a research ceiling; production coverage belongs to the router and its miss detector.
