Multi-token prediction trains extra layers to predict tokens further ahead than the next one. Labs use such a module as an extra training signal, as a drafter for speculative decoding, or both.
num_nextn_predict_layers=3 declares MTP layers in the config, and the code loads/builds mtp.{0,1,2}.* draft weights/layers (e.g. in the DSpark/MTP draft model loader).
The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra vision norm specifically for checkpoints whose language model carries MTP layers, confirming MTP modules are active for this config.
config declares num_nextn_predict_layers=3 for MTP modules, even though the given modeling file does not build them (weights are ignored via the mtp key pattern), which counts per the check's rule for config-declared nonzero MTP counts.
num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint.
num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern.
The config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are declared in config.
The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly.
Config declares num_nextn_predict_layers=1, and the quantization modules_to_not_convert list references layer 78 MTP-specific weights (eh_proj, shared_head.norm, enorm, hnorm) consistent with an extra MTP module.
config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of
The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern.
The config declares 8 next-next-token prediction (MTP) depth layers, and the model ships an InklingMTP module that builds and runs these draft transformer blocks.
The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on load, indicating MTP modules are declared by the checkpoint.
The config declares num_mtp_modules=7 and num_nextn_predict_layers=1, and the model explicitly ignores mtp.* weights on load, confirming MTP modules are part of the checkpoint.
num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint.
The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time.
The config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys confirms the checkpoint carries these extra modules even though they are not built in this file.
The text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights are ignored on load per _keys_to_ignore_on_load_unexpected = [r"^mtp.*"]).
The code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers count declaring MTP modules to build.
The code references and skips loading MTP weights ("mtp.*") via a key-ignore pattern, but no mtp layer count config key (e.g. num_nextn_predict_layers) appears in the given config to confirm whether MTP modules are actually declared/built.
The code has a regex to ignore loading model.mtp.* weights implying an MTP module may exist, but no MTP module is defined in the given file and no mtp layer count key is present in config.json.
num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build.
The config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the given modeling file to confirm how it is built.
No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction.
There is a regex to ignore loading model.mtp.* weights, implying an MTP module may exist, but no num_nextn_predict_layers or similar config key or module implementation is present in the given files.
The code ignores model.mtp.* weights on load and the config has no nextn/mtp layer count key, so it cannot be determined whether MTP modules are declared for this checkpoint.
The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built.
The config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not settled by these files.
No config key or code declares extra next-next-token prediction modules; the model predicts a single action chunk via diffusion-style denoising, not multi-token text prediction.
Multi-token prediction (MTP) gives Nemotron 3.5 Super VL an internal drafter that proposes several tokens for the main model to verify in one forward pass.
coreinference servingin Nemotron 3.5 Super VLNVIDIA
Nemotron 3.5 Lightning underwent a dedicated pretraining stage to bake multi-token prediction (MTP) into the model, as with Nemotron 3 Super and Ultra.
We analyze the relationship between the model’s predictive uncertainty measured by next token cross-entropy and the efficiency of the Multi-Token Prediction (MTP) module.
Multi-Token Prediction (MTP): Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.