Multi-Token Prediction

8 chained MTP heads (single-layer full-attention transformers with a dense MLP, all BF16) enable up to 9 tokens per forward step for speculative decoding.

We’re releasing Multi-Token Prediction (MTP) drafters for the Gemma 4 family. By using a specialized speculative decoding architecture, these drafters deliver up to a 3x speedup without any degradation in output quality or reasoning logic.

We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.

We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%

1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding.

--speculative-config.method mtp

Gemma 4 12B comes equipped with Multi-Token Prediction (MTP) drafters to reduce latency.

In order to accelerate rollout generation, we use speculative decoding with Multi-Token Prediction (MTP).

We train a small autoregressive MTP drafter head with our models, used for speculative decoding.

ships native FP8 weights plus one MTP draft layer

Includes MTP layers for faster inference through native speculative decoding.

--speculative-config '{"method":"mtp","num_speculative_tokens":5}'

A native MTP layer adds 10B total parameters (0.7B activated) for speculative decoding.

the system can speculate about: token t+1 token t+2 token t+3 ... Those candidates can then be verified efficiently.

These auxiliary predictions also serve naturally as draft tokens for speculative decoding

--speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

MTP (multi-token prediction) is supported out of the box for low-latency decoding.

--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

the built-in Multi-Token Prediction module supports speculative decoding.

MTP: A modeling technique that trains the network to predict several future tokens at each position instead of only the next one.