specific method · filed under training objective
Multi-Token Prediction
An auxiliary training objective that predicts multiple future tokens to provide richer training signals.
Also called Multi-Token Prediction (MTP), Multiple-Token Prediction, MTP, Multi-token prediction training.
- sources
- 3
- models
- 4
- labs adopt it
- 3
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
5 spans quoted from the sources, strongest treatment first.
We incorporate Multiple-Token Prediction as an additional training objective
The MTP layers of MiMo-7B is tuned during pretraining and SFT and freezed during RL.
We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.
MTP: 1 layer, trained with multi-steps
predicting multiple future tokens provides richer training signals and encourages models to plan several steps ahead
Filed alongside
Other methods under training objective :: multi-token prediction objective.