Open Models Almanac
Methodsmodel architectureprediction head

methods · model architecture

prediction head

27 methods filed at this node or below it, from the sources of 25 models.

In practice

Checked as “Multi-token prediction layers” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
DeepSeek-V4.1-Flash-1017in its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares MTP layers in the config, and the code loads/builds mtp.{0,1,2}.* draft weights/layers (e.g. in the DSpark/MTP draft model loader).

text_config.num_nextn_predict_layers = 3
self.num_dspark_layers = (
            getattr(config, "n_mtp_layers", None)
            or getattr(config, "num_nextn_predict_layers", None)
            or 3
        )
dspark.py · L90–94
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codeadopted
num_nextn_predict_layers=1

The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra vision norm specifically for checkpoints whose language model carries MTP layers, confirming MTP modules are active for this config.

llm_config.num_nextn_predict_layers = 1
            nn.LayerNorm(config.vision_hidden_size, eps=config.vision_config.layer_norm_eps)
            if config.text_config.num_nextn_predict_layers > 0
            else None
modeling_nemotron_h_omni.py · L157–159
Step-5-Previewin its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares three MTP modules, built as Step5MTP and referenced by modules_to_not_convert entries for layers 92-94.

text_config.num_nextn_predict_layers = 3
MiMo-V2.6-Flashin its codeadopted
num_nextn_predict_layers=3

config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts.

num_nextn_predict_layers = 3
r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
DeepSeek-V4.1-Flashin its codenot stated
num_nextn_predict_layers=3

num_nextn_predict_layers=3 declares MTP modules, and the DSpark/MTP draft model code loads mtp.{0,1,2}.* weights from the target checkpoint.

text_config.num_nextn_predict_layers = 3
        self.num_dspark_layers = (
            getattr(config, "n_mtp_layers", None)
            or getattr(config, "num_nextn_predict_layers", None)
            or 3
        )
dspark.py · L90–94
Hy4-previewin its codenot stated
num_nextn_predict_layers=1 · mtp_loss_factor=0.1

num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint.

num_nextn_predict_layers = 1mtp_loss_factor = 0.1
_keys_to_ignore_on_load_unexpected = [r"model\.mtp_layers\..*"]
modeling_hy_v4.py · L750
GLM-5.3-Flashin its codenot stated
num_nextn_predict_layers=1

num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern.

text_config.num_nextn_predict_layers = 1
    _keys_to_ignore_on_load_unexpected = [r"layers\.45\.", r"layers\.\d+\.shared_head\."]
modeling_glm5_next.py · L1379
Qwen3.8-Flash-Nextin its codeadopted
mtp_num_hidden_layers=1 · num_hidden_layers=1

The config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are declared in config.

text_config.mtp_num_hidden_layers = 1text_config.mtp.num_hidden_layers = 1
DeepSeek-V4-Flash-Vision-Expin its codenot stated
num_nextn_predict_layers=3

The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly.

num_nextn_predict_layers = 3
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
GLM-5.3in its codenot stated
num_nextn_predict_layers=1

Config declares num_nextn_predict_layers=1, and the quantization modules_to_not_convert list references layer 78 MTP-specific weights (eh_proj, shared_head.norm, enorm, hnorm) consistent with an extra MTP module.

num_nextn_predict_layers = 1
DeepSeek-V4-Pro-0813in its codenot stated
num_nextn_predict_layers=1

config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
DeepSeek-V4-Flash-0731in its codenot stated
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
Inklingin its codenot stated
num_nextn_predict_layers=8

The config declares 8 next-next-token prediction (MTP) depth layers, and the model ships an InklingMTP module that builds and runs these draft transformer blocks.

mtp_config.num_nextn_predict_layers = 8
n_predict = config.num_nextn_predict_layers
        num_spec = vllm_config.speculative_config.num_speculative_tokens
        self.num_mtp_layers = _select_mtp_depth_count(n_predict, num_spec)
mtp.py · L127–129
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codeadopted
num_nextn_predict_layers=1 · mtp_layers_block_type=["attention", "moe"]

The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on load, indicating MTP modules are declared by the checkpoint.

num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"]
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
MiniMax-M3in its codenot stated
num_nextn_predict_layers=1 · num_mtp_modules=7

The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint.

text_config.num_mtp_modules = 7text_config.num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_minimax_m3_vl.py · L686
Step-3.7-Flashin its codeadopted
num_nextn_predict_layers=3

num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint.

text_config.num_nextn_predict_layers = 3
DeepSeek-V4-Flashin its codeadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
DeepSeek-V4-Proin its codeadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys confirms the checkpoint carries these extra modules even though they are not built in this file.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
Qwen3.5-397B-A17Bin its codeadopted
mtp_num_hidden_layers=1

The text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights are ignored on load per _keys_to_ignore_on_load_unexpected = [r"^mtp.*"]).

text_config.mtp_num_hidden_layers = 1
_keys_to_ignore_on_load_unexpected = [r"^mtp.*"]
modeling_qwen3_5_moe.py · L994
Mistral-Large-4not in its codenot stated
verdict and lines

The code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers count declaring MTP modules to build.

_keys_to_ignore_on_load_unexpected = [r"mtp_dspark\..*"]
modeling_mistral_large4.py · L684
LFM3-1.7Bnot in its codenot stated
verdict and lines

No extra next-next-token prediction modules are declared in config or built in the modeling code.

MiniMax-M3.1-Flashcode unclearnot stated
verdict and lines

The code references and skips loading MTP weights ("mtp.*") via a key-ignore pattern, but no mtp layer count config key (e.g. num_nextn_predict_layers) appears in the given config to confirm whether MTP modules are actually declared/built.

_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_minimax_m3_vl.py · L686
Kolibri-1not in its codenot stated
verdict and lines

No multi-token-prediction modules or layer count config (num_nextn_predict_layers, mtp_num_layers) appear in the config or code.

MiMo-V2.6-Procode unclearadopted
verdict and lines

The code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codeadopted
num_nextn_predict_layers=1

num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No num_nextn_predict_layers or MTP module declarations exist in the config or code; the model only outputs a single next-token/action prediction head.

Laguna-S-2.1not in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp config key or MTP module exists in the config or code.

Kimi K3not in its codenot stated
num_nextn_predict_layers=0

num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists.

text_config.num_nextn_predict_layers = 0
Hy3not in its codeadopted
num_nextn_predict_layers=1

Config declares num_nextn_predict_layers=1, but the model explicitly ignores/skips those weights and builds no MTP module.

num_nextn_predict_layers = 1
# Not supporting multi-token prediction (MTP) atm
    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.80.*"]
modeling_hy_v3.py · L427–428
GLM-5.2code unclearadopted
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the given modeling file to confirm how it is built.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"model\.layers\.78.*"]
modeling_glm_moe_dsa.py · L637
MolmoAct 2not in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction.

MiMo-V2.5code unclearadopted
verdict and lines

There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files.

r"model\.mtp\..*",
modeling_mimo_v2.py · L1694
MiMo-V2.5-Procode unclearadopted
verdict and lines

The code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint.

r"model\.mtp\..*",
modeling_mimo_v2.py · L623
Qwen3.6-35B-A3Bcode unclearnot stated
mtp_num_hidden_layers=1 · mtp_use_dedicated_embeddings=false

The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built.

text_config.mtp_num_hidden_layers = 1text_config.mtp_use_dedicated_embeddings = false
Gemma 4 31Bnot in its codenot stated
verdict and lines

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the config or code.

DeepSeek-V3.2code unclearnot stated
num_nextn_predict_layers=1

The config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not settled by these files.

num_nextn_predict_layers = 1
X-VLAnot in its codenot stated
verdict and lines

No config key or code declares extra next-next-token prediction modules; the model predicts a single action chunk via diffusion-style denoising, not multi-token text prediction.

gpt-oss-120bnot in its codenot stated
verdict and lines

No multi-token-prediction modules or related config keys (e.g. num_nextn_predict_layers) appear in this config or code

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No num_nextn_predict_layers, mtp_num_layers, or similar config key or extra prediction-module code is present in the given files.

In this branch 27

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 27

Multi-Token Prediction core · 33 sources · 40 quotes
MTP speculative decoding core · 11 sources · 23 quotes
Flow-matching action head core · 3 sources · 3 quotes
Frozen-backbone MTP-head-only fine-tuning core · 3 sources · 5 quotes
Shared Multi-Token Prediction head core · 3 sources · 3 quotes
Diffusion-policy action head used · 2 sources · 2 quotes
20 single mentions
Diffusion-based trajectory decoder core · 1 source · 1 quote
Discrete autoregressive robot policy core · 1 source · 1 quote
Parallel action decoding core · 1 source · 2 quotes
Diffusion-Transformer-style action expert used · 1 source · 1 quote
MTP-head warm-up with a frozen backbone used · 1 source · 1 quote
MTP-only training used · 1 source · 1 quote
Nemotron Hybrid Multi-Token Prediction optional · 1 source · 1 quote
Autoregressive discrete action-tokenization head mentioned · 1 source · 1 quote
Regression-head action prediction mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
Hy4-preview coreMTP speculative decoding core—
Hy3 coreMulti-Token Prediction core—
MiMo-V2.6-Flash coreMulti-Token Prediction core—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreMTP speculative decoding coreMulti-Token Prediction coreHidden-state mixing during recursive MTP training usedShared Multi-Token Prediction head usedNemotron Hybrid Multi-Token Prediction optional—
DeepSeek-V4-Flash coreMulti-Token Prediction core—
Alpamayo 1.5 / 2 Super coreDiffusion-based trajectory decoder core—
DeepSeek-V4-Pro coreMulti-Token Prediction core—
GLM-5.2 usedMTP speculative decoding usedMulti-Token Prediction used—
GR00T N1.7 usedDiffusion Transformer IDM using SigLIP-2 vision embeddings used—
Gemma 4 31B coreMTP speculative decoding core—
LFM3-1.7B coreResidual-MLP action head for single-pass prediction of eight-step action chunks core—
LingBot-VLA 2.0 usedDiffusion-policy action head used—
MiMo-V2.5 coreMulti-Token Prediction coreMTP speculative decoding used—
MiMo-V2.5-Pro usedMTP speculative decoding usedMulti-Token Prediction used—
MiMo-V2.6-Pro coreMulti-Token Prediction core—
MolmoAct 2 coreDiscrete autoregressive robot policy corePredict discrete depth tokens before action generation corePredicting end-effector trajectory traces as intermediate visual plans coreSequential depth, trajectory, and action token generation coreDiffusion-Transformer-style action expert used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreFrozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreFrozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreShared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used—
OpenVLA / OpenVLA-OFT coreEmpty action embeddings with distinct positional encodings coreParallel action decoding core—
Qwen3.5-397B-A17B usedMulti-Token Prediction usedMulti-Token Prediction module for residual codebooks used—
Qwen3.8-Flash-Next coreMulti-Token Prediction core—
Step-3.7-Flash optionalMTP speculative decoding optionalMulti-Token Prediction optional—
VLA Foundry (Foundry-VLA-1.7B) coreFlow-matching action head coreDiffusion-policy action head optionalAutoregressive discrete action-tokenization head mentioned—
X-VLA evaluatedSeparate embodiment-specific action projection heads evaluated—
π0.5 defaultFlow-matching action head defaultDiffusion decoding for vision-language-action policies evaluatedRegression-head action prediction mentioned—
↑