Local model benchmarks are not capacity plans
Ollama 0.30 is up to 20% faster on NVIDIA — measured on one model, one GPU, one quantization. A single-GPU speedup proves an optimization exists. It does not tell you whether local inference can carry your workload.
Ollama shipped 0.30 this week: broader GGUF compatibility through llama.cpp, Vulkan on by default so AMD and Intel GPUs work out of the box, tool-calling preserved where the model exposes it, and a headline number — up to 20% faster on NVIDIA hardware.
Read the fine print on that number. The disclosed test is Gemma 4 26B, Q4_K_M quantization, on a single RTX 5090. One model, one quant, one GPU. Ollama published no broader benchmark matrix, so performance anywhere outside that exact configuration is extrapolation — mine and yours.
None of this is a criticism of the release. It is a good release. The criticism is reserved for what happens next in a dozen planning meetings: someone pastes the 20% into a slide and a local-first deployment gets approved on the strength of a number that answers a different question.
A speedup is not a capacity plan
A single-GPU benchmark tells you an optimization exists. A capacity plan answers questions the benchmark never asked:
How many concurrent sessions before tokens-per-second per user collapses? Vendor numbers are almost always single-stream; production is never single-stream. What happens at your real prompt lengths — the 40-page policy document, not the benchmark's tidy prompt? Where does tail latency land when the KV cache fills and requests start queuing? What does memory pressure do when two long contexts collide on a 32GB card? How much quality did Q4_K_M cost on your tasks — not on MMLU, on your extraction schema? And when the process falls over at 2 a.m., what is the recovery behavior — who reloads the model, how long does the warm-up take, and where do in-flight requests go?
Six budgets: concurrency, prompt length, tail latency, memory, quality, recovery. A vendor benchmark funds none of them.
The meeting where this bites
At work this spring, an engineering lead showed me a local-inference proposal justified by exactly this genre of number — a vendor's single-GPU tokens-per-second, multiplied by GPU count, divided by expected users. On paper it cleared the workload with headroom. We ran a load test with the client's actual prompt distribution instead: p95 latency crossed their SLA at roughly a third of the concurrency the slide promised, because their real prompts were four times longer than anything the vendor measured and the cache thrashed. The fix wasn't a faster runtime. It was buying to the measured number and keeping a cloud overflow lane for the spikes.
My rule since: a vendor benchmark is admissible as evidence that the software improved, never as evidence that your deployment will hold. The only benchmark that counts as a capacity plan is the one that ran your prompts, your quant, your concurrency, on the hardware you will actually buy.
Steal this for the next local-model proposal that crosses your desk: require one table before approval — target concurrent sessions, real p50/p95 prompt lengths, measured p95 latency at that concurrency, memory headroom at peak, task-level quality delta versus the cloud baseline, and time-to-recover from a process kill. If any cell says "from vendor blog", the table is not done. The test costs a day on rented hardware. The slide-number deployment costs you the quarter in which you discover the difference.
A benchmark proves the engine got faster — it says nothing about the traffic you are about to drive into it.
