
Model routing: sending each request to the right model
In short: Model routing sends each AI request to the most suitable model by difficulty, cost or latency, instead of using one model for everything.
Model routing is the practice of choosing, per request, which AI model or setting should handle it, based on task type, difficulty, cost, latency, data sensitivity or availability. Simple requests go to small, cheap, fast models and hard ones to larger or reasoning models. The RouteLLM paper from June 2024 reported cost reductions of over 2 times in certain cases without compromising response quality.
Model routing in plain words
Think of a hospital triage desk. Not every patient needs the senior specialist, and sending the right ones there keeps the queue moving and the bills sane.
How it works
A router examines each request and picks a destination. Routers can be:
- Rules: by product feature, user tier, input length or language.
- Classifiers: a small model predicts difficulty or category.
- Cascades: try a cheap model first, escalate if a check fails or confidence is low.
Anthropic's Building effective agents, published in December 2024, describes routing as classifying an input and directing it to a specialised follow-up task, with easy questions sent to smaller, cost-efficient models. Routing rules often also cover fallback when a provider is down and keeping sensitive data on approved models.
Example
As an illustration, a support assistant routes greetings and order-status lookups to a small model, policy questions to a mid-size model with retrieval, and complaints mentioning legal action to a reasoning model plus a human review queue.
Where you'll meet it
- AI gateways and proxy layers with routing rules
- Research and open-source routers such as RouteLLM
- Agent workflows that classify a request before handling it
- Cost-optimisation projects in enterprise AI platforms
What it changes for you
For architects, the model becomes a per-request decision instead of a one-time platform choice. My view: for finance, routing is one of the larger levers on AI spend. For quality owners, every route needs its own evals, because a router that misjudges difficulty sends hard cases to weak models and nobody is told.
Model routing vs model fallback
Fallback switches models only when one fails or is unavailable. Routing chooses the best model for every request, even when everything is healthy.
Common mistake
My view: routing on token price alone. A cheaper model that needs three retries or a human fix costs more per completed task. Route on cost per successful task, measured on your own traffic.
Related terms
AI gateway · Cost per task · Small language model · Reasoning model · Agentic workflow
From the archive: Premium models need failure-cost routing · A model-routing price needs an expiry date
Route on cost per successful task, measured on your own traffic.


