← back to the archiveCover illustration for “Inference server: software that serves models at scale”
TERM8mo ago·Published · Updated ·by Andy Padia

Inference server: software that serves models at scale

In short: An inference server loads AI models onto hardware and serves them through an API, batching requests and managing GPU memory for many users.

An inference server is software that loads AI models onto hardware and serves them through an API at scale, handling request batching, GPU memory, streaming and metrics. Open-source examples for language models include vLLM, which began at UC Berkeley's Sky Computing Lab, and SGLang. Both documentation sites, read in November 2026, describe OpenAI-compatible APIs.

Inference server in plain words

Think of the kitchen system behind a restaurant: it takes orders from many tables, groups them efficiently and gets dishes out without the chefs bumping into each other.

How it works

The server loads model weights into GPU memory and keeps a queue of incoming requests. Continuous batching, which vLLM lists among its features, adds and removes requests from the running batch as they start and finish, so the GPU stays busy.

Memory management matters as much. The PagedAttention paper behind vLLM, submitted in September 2023, manages the KV cache in pages and reported 2 to 4 times higher throughput at the same latency than the systems it was compared with. SGLang's documentation highlights RadixAttention prefix caching for the same reason. The server also handles tokenisation, sampling and streaming outputs.

Example

vllm serve Qwen/Qwen3-8B --max-model-len 32768
curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Hello"}]}'

The --max-model-len flag caps prompt plus output length, and vLLM listens on port 8000 by default.

Where you'll meet it

  • Self-hosted and private-cloud AI deployments
  • Managed services built on top of open-source servers
  • Other servers you will hear named, such as Text Generation Inference, TensorRT-LLM, Triton, llama.cpp and Ollama
  • Platform engineering roadmaps for open-weights models

What it changes for you

For platform teams, choosing and tuning an inference server becomes core infrastructure work, alongside databases and queues. For developers, an OpenAI-compatible endpoint means application code can switch between hosted and self-hosted models with little change. For finance, utilisation, meaning how busy the GPUs actually are, decides whether self-hosting saves money.

Inference server vs model API

A model API is a provider running inference for you, billed per token. An inference server is software you run yourself, paying for hardware and operations.

Common mistake

My view: benchmarking a server with one request at a time. Inference servers earn their keep under concurrency. Load-test at realistic traffic with realistic prompt lengths before choosing.

Inference · KV cache · Open-weights model · Quantization · AI gateway

From the archive: A 128K context window needs a concurrency budget

Load-test an inference server at realistic concurrency and prompt lengths before you choose it.

#field-guide#inference#infrastructure#open-weights#llm-infrastructure

Sources

  1. docs.vllm.ai
  2. docs.sglang.io
  3. arxiv.org/abs/2309.06180
← older drop
Time to first token (TTFT): how fast an AI starts replying
newer drop →
Reranking: a second, sharper pass over search results

related drops

explore all 557 drops →
← back to the archiveday 175