Open Models Almanac
Methodsmodel architectureprediction head

general family · filed under model architecture

Multi-Token Prediction

Multi-token prediction trains extra layers to predict tokens further ahead than the next one. Labs use such a module as an extra training signal, as a drafter for speculative decoding, or both.

Written for the explainer Multi-token prediction: a training signal and a built-in drafter.

Also called MTP, Multi Token Prediction (MTP), Multi-Token Prediction (MTP) head, Multi-Token Prediction (MTP) layer, Multi-Token Prediction (MTP) layers, Multi-Token Prediction (MTP) strategy.

sources
33
models
14
labs adopt it
7
strongest
core

In practice

Checked as “Multi-token prediction layers” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
DeepSeek-V4.1-Flash-1017in its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares MTP layers in the config, and the code loads/builds mtp.{0,1,2}.* draft weights/layers (e.g. in the DSpark/MTP draft model loader).

text_config.num_nextn_predict_layers = 3
self.num_dspark_layers = (
            getattr(config, "n_mtp_layers", None)
            or getattr(config, "num_nextn_predict_layers", None)
            or 3
        )
dspark.py · L90–94
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codeadopted
num_nextn_predict_layers=1

The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra vision norm specifically for checkpoints whose language model carries MTP layers, confirming MTP modules are active for this config.

llm_config.num_nextn_predict_layers = 1
            nn.LayerNorm(config.vision_hidden_size, eps=config.vision_config.layer_norm_eps)
            if config.text_config.num_nextn_predict_layers > 0
            else None
modeling_nemotron_h_omni.py · L157–159
Step-5-Previewin its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares three MTP modules, built as Step5MTP and referenced by modules_to_not_convert entries for layers 92-94.

text_config.num_nextn_predict_layers = 3
MiMo-V2.6-Flashin its codeadopted
num_nextn_predict_layers=3

config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts.

num_nextn_predict_layers = 3
r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
DeepSeek-V4.1-Flashin its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares MTP modules, and the DSpark/MTP draft model code loads mtp.{0,1,2}.* weights from the target checkpoint.

text_config.num_nextn_predict_layers = 3
        self.num_dspark_layers = (
            getattr(config, "n_mtp_layers", None)
            or getattr(config, "num_nextn_predict_layers", None)
            or 3
        )
dspark.py · L90–94
Hy4-previewin its codenot stated
num_nextn_predict_layers=1 · mtp_loss_factor=0.1

num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint.

num_nextn_predict_layers = 1mtp_loss_factor = 0.1
_keys_to_ignore_on_load_unexpected = [r"model\.mtp_layers\..*"]
modeling_hy_v4.py · L750
GLM-5.3-Flashin its codenot stated
num_nextn_predict_layers=1

num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern.

text_config.num_nextn_predict_layers = 1
    _keys_to_ignore_on_load_unexpected = [r"layers\.45\.", r"layers\.\d+\.shared_head\."]
modeling_glm5_next.py · L1379
Qwen3.8-Flash-Nextin its codeadopted
mtp_num_hidden_layers=1 · num_hidden_layers=1

The config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are declared in config.

text_config.mtp_num_hidden_layers = 1text_config.mtp.num_hidden_layers = 1
DeepSeek-V4-Flash-Vision-Expin its codenot stated
num_nextn_predict_layers=3

The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly.

num_nextn_predict_layers = 3
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
GLM-5.3in its codenot stated
num_nextn_predict_layers=1

Config declares num_nextn_predict_layers=1, and the quantization modules_to_not_convert list references layer 78 MTP-specific weights (eh_proj, shared_head.norm, enorm, hnorm) consistent with an extra MTP module.

num_nextn_predict_layers = 1
DeepSeek-V4-Pro-0813in its codenot stated
num_nextn_predict_layers=1

config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
DeepSeek-V4-Flash-0731in its codenot stated
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
Inklingin its codenot stated
num_nextn_predict_layers=8

The config declares 8 next-next-token prediction (MTP) depth layers, and the model ships an InklingMTP module that builds and runs these draft transformer blocks.

mtp_config.num_nextn_predict_layers = 8
n_predict = config.num_nextn_predict_layers
        num_spec = vllm_config.speculative_config.num_speculative_tokens
        self.num_mtp_layers = _select_mtp_depth_count(n_predict, num_spec)
mtp.py · L127–129
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codeadopted
num_nextn_predict_layers=1 · mtp_layers_block_type=["attention", "moe"]

The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on load, indicating MTP modules are declared by the checkpoint.

num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"]
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
MiniMax-M3in its codenot stated
num_nextn_predict_layers=1 · num_mtp_modules=7

The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint.

text_config.num_mtp_modules = 7text_config.num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_minimax_m3_vl.py · L686
Step-3.7-Flashin its codeadopted
num_nextn_predict_layers=3

num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint.

text_config.num_nextn_predict_layers = 3
DeepSeek-V4-Flashin its codeadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
DeepSeek-V4-Proin its codeadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys confirms the checkpoint carries these extra modules even though they are not built in this file.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
Qwen3.5-397B-A17Bin its codeadopted
mtp_num_hidden_layers=1

The text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights are ignored on load per _keys_to_ignore_on_load_unexpected = [r"^mtp.*"]).

text_config.mtp_num_hidden_layers = 1
_keys_to_ignore_on_load_unexpected = [r"^mtp.*"]
modeling_qwen3_5_moe.py · L994
Mistral-Large-4not in its codenot stated
verdict and lines

The code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers count declaring MTP modules to build.

_keys_to_ignore_on_load_unexpected = [r"mtp_dspark\..*"]
modeling_mistral_large4.py · L684
LFM3-1.7Bnot in its codenot stated
verdict and lines

No extra next-next-token prediction modules are declared in config or built in the modeling code.

MiniMax-M3.1-Flashcode unclearnot stated
verdict and lines

The code references and skips loading MTP weights ("mtp.*") via a key-ignore pattern, but no mtp layer count config key (e.g. num_nextn_predict_layers) appears in the given config to confirm whether MTP modules are actually declared/built.

_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_minimax_m3_vl.py · L686
Kolibri-1not in its codenot stated
verdict and lines

No multi-token-prediction modules or layer count config (num_nextn_predict_layers, mtp_num_layers) appear in the config or code.

MiMo-V2.6-Procode unclearadopted
verdict and lines

The code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codeadopted
num_nextn_predict_layers=1

num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No num_nextn_predict_layers or MTP module declarations exist in the config or code; the model only outputs a single next-token/action prediction head.

Laguna-S-2.1not in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp config key or MTP module exists in the config or code.

Kimi K3not in its codenot stated
num_nextn_predict_layers=0

num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists.

text_config.num_nextn_predict_layers = 0
Hy3not in its codeadopted
num_nextn_predict_layers=1

Config declares num_nextn_predict_layers=1, but the model explicitly ignores/skips those weights and builds no MTP module.

num_nextn_predict_layers = 1
# Not supporting multi-token prediction (MTP) atm
    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.80.*"]
modeling_hy_v3.py · L427–428
GLM-5.2code unclearadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the given modeling file to confirm how it is built.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"model\.layers\.78.*"]
modeling_glm_moe_dsa.py · L637
MolmoAct 2not in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction.

MiMo-V2.5code unclearadopted
verdict and lines

There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
MiMo-V2.5-Procode unclearadopted
verdict and lines

The code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint.

r"model\.mtp\..*",
modeling_mimo_v2.py · L623
Qwen3.6-35B-A3Bcode unclearnot stated
mtp_num_hidden_layers=1 · mtp_use_dedicated_embeddings=false

The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built.

text_config.mtp_num_hidden_layers = 1text_config.mtp_use_dedicated_embeddings = false
Gemma 4 31Bnot in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the config or code.

DeepSeek-V3.2code unclearnot stated
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not settled by these files.

num_nextn_predict_layers = 1
X-VLAnot in its codenot stated
verdict and lines

No config key or code declares extra next-next-token prediction modules; the model predicts a single action chunk via diffusion-style denoising, not multi-token text prediction.

gpt-oss-120bnot in its codenot stated
verdict and lines

No multi-token-prediction modules or related config keys (e.g. num_nextn_predict_layers) appear in this config or code

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No num_nextn_predict_layers, mtp_num_layers, or similar config key or extra prediction-module code is present in the given files.

Documented in

Evidence

40 quotes from 33 sources, one card per model, each opening with its strongest.

How sources treat it
optional 3used 18default 1core 18

Multi-token prediction (MTP) gives Nemotron 3.5 Super VL an internal drafter that proposes several tokens for the main model to verify in one forward pass.

coreinference servingin Nemotron 3.5 Super VLNVIDIA
5 more from this model

The MTP head reaches a higher average acceptance length than both Qwen models under probabilistic draft sampling

usedinference servingin Nemotron 3.5 SuperNVIDIA

Multi-Token Prediction (MTP) enables native speculative decoding.

coremodel architecturein Nemotron 3.5 Super VLNVIDIA

Multi-Token Prediction (MTP) enables native speculative decoding.

coremodel architecturein Nemotron 3.5 Super VLNVIDIA

Multi-Token Prediction (MTP) heads accelerate generation

coremodel architecturein NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 LightningNVIDIA

with multi-token prediction

coretraining objectivein Nemotron 3.5 Super VLNVIDIA

The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.

coretraining objectivein Nemotron 3.5 LightningNVIDIA
4 more from this model

Nemotron 3.5 Lightning underwent a dedicated pretraining stage to bake multi-token prediction (MTP) into the model, as with Nemotron 3 Super and Ultra.

usedtraining objectivein Nemotron 3.5 LightningNVIDIA

The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.

usedtraining objectivein Nemotron 3.5 LightningNVIDIA

The MTP head reaches a higher average acceptance length than both Qwen models under probabilistic draft sampling

usedinference servingin Nemotron 3.5 SuperNVIDIA

Nemotron 3.5 Lightning includes Multi-Token Prediction, or MTP, layers that learn to predict multiple future tokens during training.

coreunclearin Nemotron 3.5 LightningNVIDIA

Hy3

Tencent (Hunyuan)adopted5 sources

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters

coremodel architecturein Hy3Tencent Hy Team
6 more from this model

also characterized by having an 'MTP layer' with 3.8 billion parameters

coremodel architecturein Hy3Tencent

The model also uses a Multi-Token Prediction (MTP) layer. MTP predicts several tokens at once for faster decoding.

usedmodel architecturein Hy3Tencent

MTP Layer Parameters | 3.8B

coremodel architecturein Hy3Tencent Hy Team

--speculative-config.method mtp

usedinference servingin Hy3Tencent Hy Team

plus a 3.8B MTP layer for speculative decoding

coremodel architecturein Hy3Tencent

MTP (recommended for lower latency)

optionalinference servingin vLLMvLLM

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted4 sources

the Ultra model incorporates Multi-Token Prediction (MTP) layers for faster text generation and improved quality

coremodel architecturein Nemotron 3 UltraNVIDIA
5 more from this model

Nemotron 3 Ultra ships with native speculative-decoding support via a Multi-Token Prediction (MTP) head

usedinference servingin Nemotron 3 UltraNVIDIA

Multi Token Prediction (MTP) for inference acceleration with two heads during pre-training

usedtraining objectivein Nemotron 3 UltraNVIDIA

improves training signal quality, enables faster inference via native speculative decoding

usedtraining objectivein Nemotron 3 UltraNVIDIA

MTP (5 speculative tokens) enabled

defaultinference servingin Nemotron 3 UltraNVIDIA

Multi-Token Prediction for native speculative decoding

usedinference servingin Nemotron 3 UltraNVIDIA

MiMo-V2.5

Xiaomiadopted3 sources

We analyze the relationship between the model’s predictive uncertainty measured by next token cross-entropy and the efficiency of the Multi-Token Prediction (MTP) module.

coremodel architecturein MiMo-V2-FlashXiaomi
3 more from this model

Multi-Token Prediction (MTP) to enhance training performance and accelerate inference decoding

coretraining objectivein MiMo-V2-FlashXiaomi

Multi-Token Prediction (MTP): Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.

usedtraining objectivein MiMo-V2.5Xiaomi

Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.

usedtraining objectivein MiMo-V2.5Xiaomi

GLM-5.2

Z.ai (Zhipu AI)adopted2 sources

Multi-token prediction (MTP) [13;25] increases the performance of base models and acts as draft models for speculative decoding [20].

usedtraining objectivein GLM-5Z.ai
2 more from this model

MTP—particularly for multi-turn RL workloads

usedinference servingin GLM-5Z.ai

GLM-5.2 also ships an upgraded Multi-Token Prediction (MTP) layer for speculative decoding

usedinference servingin GLM-5.2Z.ai

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑