← back to the archiveCover illustration for “Speculative decoding: a draft model speeding up a big one”
TERMday 173·2d ago·Published ·by Andy Padia

Speculative decoding: a draft model speeding up a big one

In short: Speculative decoding lets a small draft model guess several tokens that the large model verifies in one pass, speeding generation without changing output.

Speculative decoding is an inference technique where a small, fast "draft" model proposes several upcoming tokens and the large target model checks them in a single forward pass, accepting the ones it agrees with. The original paper, submitted in November 2022, reported a 2X to 3X speed-up on T5-XXL with identical outputs, and needed no retraining or architecture change.

Speculative decoding in plain words

A junior drafts the next few words of a letter; the senior glances over them and keeps what is right. The senior signs off faster than if they wrote every word themselves.

How it works

  1. The draft model generates a few candidate tokens quickly.
  2. The target model scores all of them in parallel.
  3. Tokens are accepted up to the first disagreement; the target model supplies the correct token there.
  4. Repeat from the new position.

Speed-up depends on how often the draft agrees with the target, which tends to be higher for predictable text such as code and structured output. The vLLM documentation lists variants beyond a separate draft model, including n-gram lookup against the prompt, EAGLE and multi-token prediction. It describes the method as lossless in theory, up to hardware numerical precision.

Example

As an illustration, when a coding agent rewrites a file that is mostly unchanged, the draft model correctly predicts long runs of existing code, and the large model accepts them in big steps. Speed rises noticeably for that kind of edit.

Where you'll meet it

  • Inference servers such as vLLM
  • Self-hosted deployments tuning latency
  • Research on cheaper ways to produce the draft

What it changes for you

For platform teams, it is a speed-up that leaves the output distribution unchanged in theory, so quality evals should carry over. vLLM's documentation flags the exceptions: log probabilities are not guaranteed stable, and one relaxed verification mode gives up distribution preservation. For developers using hosted APIs, it is mostly invisible. Gains vary by workload, so benchmark on your real prompts.

Speculative decoding vs quantization

Quantization changes the model's numbers to run faster, with some quality risk. Speculative decoding keeps the model unchanged and speeds up how tokens are produced.

Common mistake

My view: expecting the same speed-up on every task. Creative or highly varied text gives lower acceptance rates and smaller gains. Measure acceptance rate per workload before committing hardware plans to it.

Tokens per second · Inference server · Inference · Quantization

From the archive: Speculative decoding needs a load-dependent off switch

Measure acceptance rate on your own workload before you plan hardware around speculative decoding.

#field-guide#inference#performance#optimization

Sources

  1. arxiv.org/abs/2211.17192
  2. docs.vllm.ai/en/latest/features/speculative_decoding
← older drop
AI adoption needs a default-path test
newer drop →
Google Antigravity: an agent-first coding environment

related drops

explore all 557 drops →
← back to the archiveday 175