methods · model architecture
prediction head
27 methods filed at this node or below it, from the sources of 25 models.
In practice
Checked as “Multi-token prediction layers” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | num_nextn_predict_layers=3num_nextn_predict_layers=3 declares MTP layers in the config, and the code loads/builds mtp.{0,1,2}.* draft weights/layers (e.g. in the DSpark/MTP draft model loader). text_config.num_nextn_predict_layers = 3 | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | adopted | num_nextn_predict_layers=1The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra vision norm specifically for checkpoints whose language model carries MTP layers, confirming MTP modules are active for this config. llm_config.num_nextn_predict_layers = 1 | code → |
| Step-5-Preview | in its code | not stated | num_nextn_predict_layers=3num_nextn_predict_layers=3 declares three MTP modules, built as Step5MTP and referenced by modules_to_not_convert entries for layers 92-94. text_config.num_nextn_predict_layers = 3 | code → |
| MiMo-V2.6-Flash | in its code | adopted | num_nextn_predict_layers=3config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts. num_nextn_predict_layers = 3 | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | num_nextn_predict_layers=3num_nextn_predict_layers=3 declares MTP modules, and the DSpark/MTP draft model code loads mtp.{0,1,2}.* weights from the target checkpoint. text_config.num_nextn_predict_layers = 3 | code → |
| Hy4-preview | in its code | not stated | num_nextn_predict_layers=1 · mtp_loss_factor=0.1num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint. num_nextn_predict_layers = 1mtp_loss_factor = 0.1 | code → |
| GLM-5.3-Flash | in its code | not stated | num_nextn_predict_layers=1num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern. text_config.num_nextn_predict_layers = 1 | code → |
| Qwen3.8-Flash-Next | in its code | adopted | mtp_num_hidden_layers=1 · num_hidden_layers=1The config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are declared in config. text_config.mtp_num_hidden_layers = 1text_config.mtp.num_hidden_layers = 1 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | num_nextn_predict_layers=3The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly. num_nextn_predict_layers = 3 | code → |
| GLM-5.3 | in its code | not stated | num_nextn_predict_layers=1Config declares num_nextn_predict_layers=1, and the quantization modules_to_not_convert list references layer 78 MTP-specific weights (eh_proj, shared_head.norm, enorm, hnorm) consistent with an extra MTP module. num_nextn_predict_layers = 1 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | num_nextn_predict_layers=1config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of num_nextn_predict_layers = 1 | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | num_nextn_predict_layers=1The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern. num_nextn_predict_layers = 1 | code → |
| Inkling | in its code | not stated | num_nextn_predict_layers=8The config declares 8 next-next-token prediction (MTP) depth layers, and the model ships an InklingMTP module that builds and runs these draft transformer blocks. mtp_config.num_nextn_predict_layers = 8 | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | adopted | num_nextn_predict_layers=1 · mtp_layers_block_type=["attention", "moe"]The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on load, indicating MTP modules are declared by the checkpoint. num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"] | code → |
| MiniMax-M3 | in its code | not stated | num_nextn_predict_layers=1 · num_mtp_modules=7The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint. text_config.num_mtp_modules = 7text_config.num_nextn_predict_layers = 1 | code → |
| Step-3.7-Flash | in its code | adopted | num_nextn_predict_layers=3num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint. text_config.num_nextn_predict_layers = 3 | code → |
| DeepSeek-V4-Flash | in its code | adopted | num_nextn_predict_layers=1The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time. num_nextn_predict_layers = 1 | code → |
| DeepSeek-V4-Pro | in its code | adopted | num_nextn_predict_layers=1The config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys confirms the checkpoint carries these extra modules even though they are not built in this file. num_nextn_predict_layers = 1 | code → |
| Qwen3.5-397B-A17B | in its code | adopted | mtp_num_hidden_layers=1The text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights are ignored on load per _keys_to_ignore_on_load_unexpected = [r"^mtp.*"]). text_config.mtp_num_hidden_layers = 1 | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThe code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers count declaring MTP modules to build. | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo extra next-next-token prediction modules are declared in config or built in the modeling code. | code → |
| MiniMax-M3.1-Flash | code unclear | not stated | verdict and linesThe code references and skips loading MTP weights ("mtp.*") via a key-ignore pattern, but no mtp layer count config key (e.g. num_nextn_predict_layers) appears in the given config to confirm whether MTP modules are actually declared/built. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesNo multi-token-prediction modules or layer count config (num_nextn_predict_layers, mtp_num_layers) appear in the config or code. | code → |
| MiMo-V2.6-Pro | code unclear | adopted | verdict and linesThe code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json. | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | adopted | num_nextn_predict_layers=1num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build. num_nextn_predict_layers = 1 | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo num_nextn_predict_layers or MTP module declarations exist in the config or code; the model only outputs a single next-token/action prediction head. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesNo num_nextn_predict_layers/mtp config key or MTP module exists in the config or code. | code → |
| Kimi K3 | not in its code | not stated | num_nextn_predict_layers=0num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists. text_config.num_nextn_predict_layers = 0 | code → |
| Hy3 | not in its code | adopted | num_nextn_predict_layers=1Config declares num_nextn_predict_layers=1, but the model explicitly ignores/skips those weights and builds no MTP module. num_nextn_predict_layers = 1 | code → |
| GLM-5.2 | code unclear | adopted | num_nextn_predict_layers=1The config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the given modeling file to confirm how it is built. num_nextn_predict_layers = 1 | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction. | code → |
| MiMo-V2.5 | code unclear | adopted | verdict and linesThere is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files. | code → |
| MiMo-V2.5-Pro | code unclear | adopted | verdict and linesThe code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint. | code → |
| Qwen3.6-35B-A3B | code unclear | not stated | mtp_num_hidden_layers=1 · mtp_use_dedicated_embeddings=falseThe config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built. text_config.mtp_num_hidden_layers = 1text_config.mtp_use_dedicated_embeddings = false | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the config or code. | code → |
| DeepSeek-V3.2 | code unclear | not stated | num_nextn_predict_layers=1The config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not settled by these files. num_nextn_predict_layers = 1 | code → |
| X-VLA | not in its code | not stated | verdict and linesNo config key or code declares extra next-next-token prediction modules; the model predicts a single action chunk via diffusion-style denoising, not multi-token text prediction. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo multi-token-prediction modules or related config keys (e.g. num_nextn_predict_layers) appear in this config or code | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo num_nextn_predict_layers, mtp_num_layers, or similar config key or extra prediction-module code is present in the given files. | code → |
In this branch 27
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 27
20 single mentions
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.