general family · filed under inference & serving
Speculative decoding
Speculative decoding is an inference technique that decouples token generation from verification to speed up LLM decoding.1 A lightweight drafter predicts several future tokens at once, and the heavy target model verifies them in parallel, accepting the whole sequence in a single forward pass if it agrees.2,3
Trade-off It delivers large speedups without degrading output quality, since the target model still verifies every token.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Speculative decoding decouples token generation from verification
— Accelerating Gemma 4: faster inference with multi-token prediction drafterswe can utilize idle compute to “predict” several future tokens at once with the drafter in less time than it takes for the target model to process just one token. The target model then verifies all of these suggested tokens in parallel
— Accelerating Gemma 4: faster inference with multi-token prediction draftersIf the target model agrees with the draft, it accepts the entire sequence in a single forward pass
— Accelerating Gemma 4: faster inference with multi-token prediction draftersthese drafters deliver up to a 3x speedup without any degradation in output quality or reasoning logic
— Accelerating Gemma 4: faster inference with multi-token prediction drafters
Also called speculative decoding methods, speculative decoding module.