
Inference server: software that serves models at scale
In short: An inference server loads AI models onto hardware and serves them through an API, batching requests and managing GPU memory for many users.
An inference server is software that loads AI models onto hardware and serves them through an API at scale, handling request batching, GPU memory, streaming and metrics. Open-source examples for language models include vLLM, which began at UC Berkeley's Sky Computing Lab, and SGLang. Both documentation sites, read in November 2026, describe OpenAI-compatible APIs.
Inference server in plain words
Think of the kitchen system behind a restaurant: it takes orders from many tables, groups them efficiently and gets dishes out without the chefs bumping into each other.
How it works
The server loads model weights into GPU memory and keeps a queue of incoming requests. Continuous batching, which vLLM lists among its features, adds and removes requests from the running batch as they start and finish, so the GPU stays busy.
Memory management matters as much. The PagedAttention paper behind vLLM, submitted in September 2023, manages the KV cache in pages and reported 2 to 4 times higher throughput at the same latency than the systems it was compared with. SGLang's documentation highlights RadixAttention prefix caching for the same reason. The server also handles tokenisation, sampling and streaming outputs.
Example
vllm serve Qwen/Qwen3-8B --max-model-len 32768
curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Hello"}]}'
The --max-model-len flag caps prompt plus output length, and vLLM listens on port 8000 by default.
Where you'll meet it
- Self-hosted and private-cloud AI deployments
- Managed services built on top of open-source servers
- Other servers you will hear named, such as Text Generation Inference, TensorRT-LLM, Triton, llama.cpp and Ollama
- Platform engineering roadmaps for open-weights models
What it changes for you
For platform teams, choosing and tuning an inference server becomes core infrastructure work, alongside databases and queues. For developers, an OpenAI-compatible endpoint means application code can switch between hosted and self-hosted models with little change. For finance, utilisation, meaning how busy the GPUs actually are, decides whether self-hosting saves money.
Inference server vs model API
A model API is a provider running inference for you, billed per token. An inference server is software you run yourself, paying for hardware and operations.
Common mistake
My view: benchmarking a server with one request at a time. Inference servers earn their keep under concurrency. Load-test at realistic traffic with realistic prompt lengths before choosing.
Related terms
Inference · KV cache · Open-weights model · Quantization · AI gateway
From the archive: A 128K context window needs a concurrency budget
Load-test an inference server at realistic concurrency and prompt lengths before you choose it.


