Open Models Almanac
Methodsinference & servingdecoding strategy

general family · filed under inference & serving

Multi-Token Prediction

As a drafting method, an MTP module proposes the next few tokens and the backbone checks them all in one forward pass. Each further token counts only if every earlier one was accepted, so its value falls with distance.

Written for the explainer Multi-token prediction: a training signal and a built-in drafter.

Also called MTP-based speculative decoding, Multi-Token Prediction (MTP), MTP layer, MTP layer for speculative decoding, MTP speculative decoding, MTP.

sources
19
models
13
labs adopt it
8
strongest
core

Documented in

Evidence

19 quotes from 19 sources, one card per model, each opening with its strongest.

How sources treat it
optional 6used 11core 2

Gemma 4 31B

Google DeepMindadopted3 sources

We’re releasing Multi-Token Prediction (MTP) drafters for the Gemma 4 family. By using a specialized speculative decoding architecture, these drafters deliver up to a 3x speedup without any degradation in output quality or reasoning logic.

coreinference servingin Gemma 4 MTP draftersGoogle
2 more from this model

We train a small autoregressive MTP drafter head with our models, used for speculative decoding.

usedinference servingin Gemma 4Google DeepMind

Gemma 4 12B comes equipped with Multi-Token Prediction (MTP) drafters to reduce latency.

usedinference servingin Gemma 4 12BGoogle

MTP: A modeling technique that trains the network to predict several future tokens at each position instead of only the next one.

optionalinference servingin Nemotron 3.5 LightningNVIDIA
2 more from this model

the system can speculate about: token t+1 token t+2 token t+3 ... Those candidates can then be verified efficiently.

usedunclearin Nemotron 3.5 LightningNVIDIA

These auxiliary predictions also serve naturally as draft tokens for speculative decoding

usedinference servingin Nemotron 3 Super and UltraNVIDIA
↑