Route agents by queue age, not token price

Tomasz Tunguz published a week of measurements from his own desk this morning: a local classifier on his Mac now routes his AI tasks, keeps the straightforward ones on-device, and sends only the hard ones to cloud models. Over seven days, the laptop handled 78% of the work. Throughput rose about 25%. Average task duration fell from 47 seconds to 19.

Everyone will quote the 78%. It is a great number for the local-models-are-winning narrative, and that narrative will run all year.

The number I would frame and hang on the wall is a different one: queue age fell from 73 seconds to 4.

The metric hiding in the fourth sentence

Queue age is how long a task sits waiting before anything starts working on it. Before the router, Tunguz's average task waited 73 seconds — longer than most of the tasks themselves took to run. After, it waited 4. That is not a cost improvement. That is a scheduling improvement, and it is where nearly all of the felt speed-up lives.

This is queueing theory doing what it always does. When every task — the two-line draft reply and the forty-minute research run — goes into the same line for the same expensive resource, small jobs get stuck behind big ones. Head-of-line blocking. Your work-in-progress ages on the shelf, and the system feels slow even when the workers are fast. Give the small jobs their own fast lane and the whole system's responsiveness transforms, even if the big jobs finish no faster than before.

Read the architecture again with that lens. The local model isn't valuable because it is free. It is valuable because it is idle and nearby — an always-available worker for the small stuff, which means the queue for the serious cloud work stops being contaminated by trivia. Cloud avoidance is the mechanism. Waiting time is the outcome. Tunguz's own framing gets this right: most of his AI work, it turns out, can wait a bit or run somewhere cheaper — what it must not do is stand in line.

What the router market wants you to optimize instead

There is a growing shelf of model-routing products, and almost all of them pitch the same metric: price per token. Route the easy prompts to the cheap model, save 60% on inference, here is the dashboard. The pitch works because token spend is legible — finance can see it, and nobody's finance team has ever seen a queue-age chart.

At work I sat in exactly this evaluation this quarter: a platform lead choosing between a cheaper model router and a workflow scheduler. The router demo showed cost falling. But when we pulled the traces from their existing agent fleet, the picture was Tunguz's picture: small classification and drafting tasks were queuing behind long document-analysis runs on the same worker pool, and the median task spent more time waiting than executing. A cheaper model would have made the same wait cost less. Only the scheduler made the wait go away. We chose the scheduler, and the token bill barely moved — but the backlog cleared and the team stopped babysitting the queue, which was the actual complaint.

My rule from that engagement: instrument queue age before you buy a router. If tasks wait longer than they run, you have a scheduling problem, and no per-token discount fixes a scheduling problem.

The shape of the fix

rendering diagram…

Two lanes, one classifier, and the discipline to route by latency class — what the task needs and when it is due — rather than by which model is cheapest this week. The classifier does not need to be clever. It needs to be fast and local, so classification itself never joins the queue.

The honest caveats

This is one person's desk, one week, self-reported. I could not verify the workload mix, whether quality was measured at parity between local and cloud outputs, or whether tasks that failed locally and were re-sent to the cloud got counted once or twice. A 78% local share for an investor's reading-and-writing workload will not transfer to a coding-heavy or compliance-heavy one. Treat the numbers as an existence proof, not a benchmark.

But the existence proof is enough, because the queueing argument does not depend on his exact numbers. It depends on arithmetic: any fleet where task sizes vary by two orders of magnitude and everything shares one line will spend most of its life waiting.

Steal this

Add three timestamps to every agent task in your system this week: enqueued_at, started_at, finished_at. Report two numbers per task class: p50 and p95 queue age. That is the whole instrument. If p95 queue age exceeds p50 execution time, your next purchase is a scheduler or a second lane — local model, dedicated worker pool, either works — and not a cheaper model. Re-measure after. The day queue age drops is the day the system starts feeling fast, whatever happens to the invoice.

The laptop didn't win because it was cheap — it won because the small work stopped standing in line behind the big work.