general family · filed under model architecture
Multi-Token Prediction
A prediction method that predicts multiple future tokens, with the evidence also describing its use as a source of richer training signals and for speculative decoding.
Also called 3.8B MTP layer, MTP, MTP layer parameters, Multi Token Prediction (MTP), Multi-Token Prediction (MTP) head, Multi-Token Prediction (MTP) layer.
- sources
- 30
- models
- 14
- labs adopt it
- 7
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Better & Faster Large Language Models via Multi-token Prediction (Gloeckle et al., 2024) paper arxiv.org
- Multi-Token Prediction (MTP) - LLM Architecture Gallery explainer sebastianraschka.com
In the code
Checked as “Multi-token prediction layers” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | num_nextn_predict_layers=1 declares an MTP module, and the quantization_config lists model.layers.45.* weights (eh_proj, enorm, hnorm, shared_head) corresponding to the extra next-token-prediction layer beyond the 45 main layers. |
| DeepSeek-V4.1-Flash | in its code | not stated | The config declares num_nextn_predict_layers=3 and the checkpoint's mtp.{0,1,2}.* weights are loaded as extra next-token-prediction decoder layers (DSpark draft model reuses this count too). dspark.py#L87 |
| Hy4-preview | code unclear | core | The config declares num_nextn_predict_layers=1 and mtp_loss_factor, and the modeling file references model.mtp_layers.* weights to ignore on load, but the MTP module itself is not defined/built in the given files. modeling_hy_v4.py#L761 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | Config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected MTP weight keys on load, indicating the checkpoint carries an MTP module the code skips. modeling_deepseek_v4.py#L1259 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | core | The config declares num_nextn_predict_layers=1 with mtp_layers_block_type set, and the modeling code has weight-skip handling for MTP module keys. modeling_nemotron_h.py#L981 |
| MiMo-V2.6-Flash | in its code | core | config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts. modeling_mimo_v2.py#L1694 |
| DeepSeek-V4-Flash | in its code | core | num_nextn_predict_layers is 1 in the config, and the modeling file explicitly declares an ignore pattern for mtp weights on load, indicating declared but skipped MTP modules. modeling_deepseek_v4.py#L1259 |
| MiMo-V2.5 | code unclear | core | There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files. modeling_mimo_v2.py#L1694 |
| GLM-5.3 | in its code | not stated | num_nextn_predict_layers is 1 and the quantization config's modules_to_not_convert list explicitly references an extra layer 78 (enorm/hnorm/eh_proj/shared_head) beyond the 78 main layers (0-77), indicating a declared MTP module. |
| Hy3 | not in its code | core | The config declares num_nextn_predict_layers: 1 but the code explicitly states MTP is not supported and skips loading those weights modeling_hy_v3.py#L438 |
| GLM-5.2 | code unclear | used | num_nextn_predict_layers=1 is set in config and there's a reference to skipping model.layers.78* on load, but no MTP module class or building logic is present in the given modeling file. modeling_glm_moe_dsa.py#L648 |
| MiniMax-M3 | in its code | not stated | The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, indicating declared but skipped MTP modules. modeling_minimax_m3_vl.py#L703 |
| DeepSeek-V3.2 | in its code | not stated | The config declares num_nextn_predict_layers: 1, and the model's weight-loading ignores those extra MTP layer weights, indicating the checkpoint carries them. modeling_deepseek_v32.py#L676 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | The config declares num_nextn_predict_layers=3 and the model explicitly ignores MTP weight keys on load, indicating the checkpoint carries MTP modules even though this file does not instantiate them. modeling_deepseek_v4.py#L1259 |
| DeepSeek-V4-Pro | in its code | core | num_nextn_predict_layers is 1 in this config, and the modeling code declares an ignore pattern for mtp weights, indicating declared MTP modules in the checkpoint modeling_deepseek_v4.py#L1259 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | num_nextn_predict_layers=1 in config declares an MTP module count, and the modeling file explicitly ignores mtp.* weights on load, indicating the checkpoint carries MTP layers even though they are not instantiated here. modeling_deepseek_v4.py#L1259 |
| Gemma 4 31B | not in its code | not stated | No MTP module declarations or config keys (num_nextn_predict_layers, mtp_num_layers, etc.) exist in this config or code. |
| Inkling | in its code | not stated | The config declares 8 MTP next-token prediction depth layers, and a dedicated InklingMTP module builds and loads them as separate transformer blocks. mtp.py#L118 |
| Kimi K3 | not in its code | not stated | num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists. |
| Laguna-S-2.1 | not in its code | not stated | No config key declares extra next-next-token prediction modules and no MTP module exists in the modeling code. |
| MiMo-V2.5-Pro | code unclear | used | The code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint. modeling_mimo_v2.py#L623 |
| MiMo-V2.6-Pro | code unclear | core | The code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json. modeling_mimo_v2.py#L1694 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | core | config declares num_nextn_predict_layers=1 and mtp_layers_block_type ['attention','moe'], enabling multi-token prediction module construction. |
| Qwen3.5-397B-A17B | code unclear | used | The config declares mtp_num_hidden_layers=1 but no MTP module implementation appears in the given modeling file besides ignoring mtp.* weights on load. modeling_qwen3_5_moe.py#L1014 |
| Qwen3.6-35B-A3B | code unclear | not stated | The config declares mtp_num_hidden_layers=1, but no MTP module class or weight declaration for it appears in the given modeling file, so the mechanism cannot be confirmed from these files. |
| Qwen3.8-Flash-Next | in its code | core | The config declares an mtp block with num_hidden_layers=1 (and mtp_num_hidden_layers=1), a nonzero count of extra next-token-prediction layers, even though the given modeling file skips loading mtp weights. modeling_qwen4_exp.py#L1322 |
| Step-3.7-Flash | in its code | optional | num_nextn_predict_layers=3 declares three trailing MTP layers, and the text model filters their weight keys as unexpected on plain load, confirming they are declared/skipped per config. modeling_step3p7.py#L843 |
| gpt-oss-120b | not in its code | not stated | No multi-token-prediction modules or config fields are declared in this model. |
Evidence
40 spans quoted from the sources, strongest treatment first.
The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.
MTP Layer Parameters | 3.8B
Multi-Token Prediction (MTP): 5-layer speculative decoder
1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding
Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters
MTP: 1 layer, trained with multi-steps
the Ultra model incorporates Multi-Token Prediction (MTP) layers for faster text generation and improved quality
DeepSeek-V4 series also set MTP modules and objectives
Nemotron 3.5 Lightning includes Multi-Token Prediction, or MTP, layers that learn to predict multiple future tokens during training.
also characterized by having an 'MTP layer' with 3.8 billion parameters
Multi Token Prediction (MTP) leads to faster inference with speculative decoding
Multi-Token Prediction for native speculative decoding
Multi-Token Prediction (MTP): 5-layer speculative decoder
Multi-Token Prediction (MTP) to enhance training performance and accelerate inference decoding
We analyze the relationship between the model’s predictive uncertainty measured by next token cross-entropy and the efficiency of the Multi-Token Prediction (MTP) module.
plus a 3.8B MTP layer for speculative decoding
MTP (5 speculative tokens) enabled
Nemotron 3.5 Lightning underwent a dedicated pretraining stage to bake multi-token prediction (MTP) into the model, as with Nemotron 3 Super and Ultra.
A 3-layer Multi-Token Prediction (MTP) head is included for speculative decoding.
retains the DeepSeekMoE framework and Multi-Token Prediction (MTP) strategy from DeepSeek-V3
The model also uses a Multi-Token Prediction (MTP) layer. MTP predicts several tokens at once for faster decoding.
MTP: trained with multi-steps
--speculative-config.method mtp
Multi Token Prediction (MTP) for inference acceleration with two heads during pre-training
Multi-Token Prediction for native speculative decoding
Multi-Token Prediction (MTP): Equipped with three lightweight MTP modules using dense FFNs.
This triples output speed during inference
The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.
GLM-5.2 also ships an upgraded Multi-Token Prediction (MTP) layer for speculative decoding
improves training signal quality, enables faster inference via native speculative decoding
Multi-token prediction (MTP) [13;25] increases the performance of base models and acts as draft models for speculative decoding [20].
MTP—particularly for multi-turn RL workloads
Multi-Token Prediction (MTP): Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.
Nemotron 3 Ultra ships with native speculative-decoding support via a Multi-Token Prediction (MTP) head
improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%
Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.
by repurposing MTP as a draft model for speculative decoding
The latest NVFP4 checkpoint includes MTP draft layers and supports vLLM speculative decoding
--speculative_config '{"method": "mtp", "num_speculative_tokens": 3}'
MTP (recommended for lower latency)
Filed alongside
Other methods under model architecture :: prediction head.