Qwen3.5-397B-A17B
Alibaba (Qwen)MTP: trained with multi-steps
1 more from this model
MTP: trained with multi-steps
general family · filed under training objective
As a training objective, multi-token prediction adds a loss on tokens further ahead than the next; the three labs that state a weight use 0.1 to 0.3. The two published ablations report modest gains, on test models.
Written for the explainer Multi-token prediction: a training signal and a built-in drafter.
Also called Multi-Token Prediction (MTP), MTP: trained with multi-steps, Multi-token prediction training, MTP, Multiple-Token Prediction.
7 quotes from 5 sources, one card per model, each opening with its strongest.
MTP: trained with multi-steps
MTP: trained with multi-steps
MTP: 1 layer, trained with multi-steps
predicting multiple future tokens provides richer training signals and encourages models to plan several steps ahead
We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.
The MTP layers of MiMo-7B is tuned during pretraining and SFT and freezed during RL.
We incorporate Multiple-Token Prediction as an additional training objective