← back to the archiveCover illustration for “Model routing: sending each request to the right model”
TERM7mo ago·Published · Updated ·by Andy Padia

Model routing: sending each request to the right model

In short: Model routing sends each AI request to the most suitable model by difficulty, cost or latency, instead of using one model for everything.

Model routing is the practice of choosing, per request, which AI model or setting should handle it, based on task type, difficulty, cost, latency, data sensitivity or availability. Simple requests go to small, cheap, fast models and hard ones to larger or reasoning models. The RouteLLM paper from June 2024 reported cost reductions of over 2 times in certain cases without compromising response quality.

Model routing in plain words

Think of a hospital triage desk. Not every patient needs the senior specialist, and sending the right ones there keeps the queue moving and the bills sane.

How it works

A router examines each request and picks a destination. Routers can be:

  1. Rules: by product feature, user tier, input length or language.
  2. Classifiers: a small model predicts difficulty or category.
  3. Cascades: try a cheap model first, escalate if a check fails or confidence is low.

Anthropic's Building effective agents, published in December 2024, describes routing as classifying an input and directing it to a specialised follow-up task, with easy questions sent to smaller, cost-efficient models. Routing rules often also cover fallback when a provider is down and keeping sensitive data on approved models.

Example

As an illustration, a support assistant routes greetings and order-status lookups to a small model, policy questions to a mid-size model with retrieval, and complaints mentioning legal action to a reasoning model plus a human review queue.

Where you'll meet it

  • AI gateways and proxy layers with routing rules
  • Research and open-source routers such as RouteLLM
  • Agent workflows that classify a request before handling it
  • Cost-optimisation projects in enterprise AI platforms

What it changes for you

For architects, the model becomes a per-request decision instead of a one-time platform choice. My view: for finance, routing is one of the larger levers on AI spend. For quality owners, every route needs its own evals, because a router that misjudges difficulty sends hard cases to weak models and nobody is told.

Model routing vs model fallback

Fallback switches models only when one fails or is unavailable. Routing chooses the best model for every request, even when everything is healthy.

Common mistake

My view: routing on token price alone. A cheaper model that needs three retries or a human fix costs more per completed task. Route on cost per successful task, measured on your own traffic.

AI gateway · Cost per task · Small language model · Reasoning model · Agentic workflow

From the archive: Premium models need failure-cost routing · A model-routing price needs an expiry date

Route on cost per successful task, measured on your own traffic.

#field-guide#routing#models#cost#llm-infrastructure

Sources

  1. arxiv.org/abs/2406.18665
  2. anthropic.com/research/building-effective-agents
← older drop
Groundedness: is the AI's answer backed by its sources?
newer drop →
Benchmark contamination: when models have seen the test

related drops

explore all 557 drops →
← back to the archiveday 175