
AI gateway: one control point for every model call
In short: An AI gateway sits between apps and model providers to handle credentials, routing, rate limits, caching, logs and budgets in one place.
An AI gateway is a proxy layer between your applications and AI model providers. Every model call goes through it, so you can manage credentials, route between models and providers, apply rate limits and budgets, cache, and log usage and cost in one place. Vercel's documentation, updated in September 2026, sums up the job: centralise credentials, log requests, control spend and fail over across providers.
AI gateway in plain words
It is a company switchboard for AI. Every team dials through it, so you know who called which model and what it cost, and you can reroute calls when a line goes down.
How it works
A gateway does for model calls what an API gateway does for web services. Applications call it through one consistent API. The gateway applies policies, chooses the provider and model, adds credentials, forwards the request and records what happened.
Vercel's AI Gateway accepts OpenAI and Anthropic request formats, routes across providers and fallback models, and records status, latency, token usage and cost for each request. Cloudflare's AI Gateway lists analytics, logging, caching, rate limiting and model fallback, with data loss prevention and guardrails marked beta when read in November 2026.
Example
As an illustration, a company with twelve AI features moves all calls behind a gateway. Within a week it can see that one internal summariser accounts for a large share of spend, sets a monthly budget for it, routes it to a cheaper model and keeps a fallback provider for outages.
Where you'll meet it
- Hosted gateways from Vercel and Cloudflare
- Open-source proxies such as LiteLLM
- Cloud provider model hubs
- Enterprise AI platform architectures
What it changes for you
For platform teams, the gateway is where an AI usage policy can be enforced in code. For developers, switching models becomes configuration. My view: for finance and security, it provides the per-team cost view and request logs that are hard to assemble after something has gone wrong.
AI gateway vs model router
A router decides which model handles a request. A gateway is the wider control layer, which often includes routing alongside keys, limits, logging and policy.
Common mistake
My view: adding a gateway but letting teams keep direct provider keys "just for testing". Calls made with those keys bypass every control and every cost report. Rotate the keys and make the gateway the only path.
Related terms
Model routing · Shadow AI · AI observability · Guardrails · Cost per task
From the archive: An agent gateway needs a bypass budget
A gateway controls only the calls that pass through it, so retire the direct provider keys.


