Open Models Almanac
Methodsinference & servingdecoding strategy

general family · filed under inference & serving

Speculative decoding

Speculative decoding is an inference technique that decouples token generation from verification to speed up LLM decoding.1 A lightweight drafter predicts several future tokens at once, and the heavy target model verifies them in parallel, accepting the whole sequence in a single forward pass if it agrees.2,3

Trade-off It delivers large speedups without degrading output quality, since the target model still verifies every token.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Speculative decoding decouples token generation from verification — Accelerating Gemma 4: faster inference with multi-token prediction drafters
  2. we can utilize idle compute to “predict” several future tokens at once with the drafter in less time than it takes for the target model to process just one token. The target model then verifies all of these suggested tokens in parallel — Accelerating Gemma 4: faster inference with multi-token prediction drafters
  3. If the target model agrees with the draft, it accepts the entire sequence in a single forward pass — Accelerating Gemma 4: faster inference with multi-token prediction drafters
  4. these drafters deliver up to a 3x speedup without any degradation in output quality or reasoning logic — Accelerating Gemma 4: faster inference with multi-token prediction drafters

Also called speculative decoding methods, speculative decoding module.

sources
15
models
10
labs adopt it
4
strongest
core

Documented in

Evidence

15 quotes from 15 sources, one card per model, each opening with its strongest.

How sources treat it
mentioned 3optional 2used 5default 1core 4

The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision

coreinference servingin Nemotron 3.5 Super VLNVIDIA
1 more from this model

Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned

coreinference servingin Nemotron 3.5NVIDIA

This is a process whereby the model, or draft model, will draft some number of tokens which are efficiently reviewed.

coreinference servingin Nemotron 3.5 LightningNVIDIA
1 more from this model

The Lightning 3.5 model is released alongside a number of speculative decoding methods for faster text generation.

mentionedinference servingin Nemotron 3.5 LightningNVIDIA

Step-5-Preview

StepFunadopted2 sources

Use a framework build that includes Step 5 Preview support, including the reasoning, tool-call, and speculative decoding features used in the serving commands.

usedinference servingin Step 5 PreviewStepFun
1 more from this model

including the reasoning, tool-call, and speculative decoding features used in the serving commands

optionalinference servingin Step 5 PreviewStepFun

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑