Speculative decoding

This is a process whereby the model, or draft model, will draft some number of tokens which are efficiently reviewed.

MTP mitigates this inefficiency through speculative decoding, a technique introduced by Google researchers in Fast Inference from Transformers via Speculative Decoding.

for speculative decoding

enables faster inference via native speculative decoding

A 3-layer Multi-Token Prediction (MTP) head is included for speculative decoding.

With one MTP layer for speculative decoding, the acceptance rate is about 90%.

We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%

is built in for speculative decoding

Both vLLM and SGLang enable it through speculative decoding.

Multi Token Prediction (MTP) leads to faster inference with speculative decoding

Head-only KL distillation aligns MTP drafts with backbone logits for faster speculative decoding.

Includes MTP layers for faster inference through native speculative decoding.

use DSpark (Cheng et al., 2026a) for speculative decoding.

accelerate inference via speculative decoding

Multi Token Prediction (MTP) leads to faster inference with speculative decoding

we use speculative decoding with Multi-Token Prediction (MTP).

Multi-Token Prediction for native speculative decoding

MTP layer for speculative decoding

accelerate inference via speculative decoding

by repurposing MTP as a draft model for speculative decoding

1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding

supports vLLM speculative decoding with: --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

--speculative-algorithm EAGLE

The Lightning 3.5 model is released alongside a number of speculative decoding methods for faster text generation.

it comes with a speculative decoding module attached