Open Models Almanac
Methodstraining objectivemulti-token prediction objective

general family · filed under training objective

Multi-Token Prediction

As a training objective, multi-token prediction adds a loss on tokens further ahead than the next; the three labs that state a weight use 0.1 to 0.3. The two published ablations report modest gains, on test models.

Written for the explainer Multi-token prediction: a training signal and a built-in drafter.

Also called Multi-Token Prediction (MTP), MTP: trained with multi-steps, Multi-token prediction training, MTP, Multiple-Token Prediction.

sources
5
models
5
labs adopt it
3
strongest
used

Documented in

Evidence

7 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
used 7

MiMo-V2.5 · MiMo-V2.5-Pro

Xiaomiadopted1 source

We support MTP in vLLM and enhance the robustness of the inference engine in the RL system.

usedmodel architecturein MiMo-7B seriesXiaomi
2 more from these models

The MTP layers of MiMo-7B is tuned during pretraining and SFT and freezed during RL.

usedtraining objectivein MiMo-7B-BaseXiaomi

We incorporate Multiple-Token Prediction as an additional training objective

usedtraining objectivein MiMo-7B-BaseXiaomi
↑