Open Models Almanac
Methodstraining objectivemulti-token prediction objective

specific method · filed under training objective

MTP Boosting

MTP Boosting is a dedicated post-training phase that updates only the multi-token-prediction head's parameters while the backbone stays frozen.1 It trains the MTP head with a knowledge-distillation loss against the frozen backbone and continues hidden-state mixing, reducing train-inference mismatch and improving draft acceptance.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the backbone remains frozen and only the MTP head parameters are updated — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. these runs use a knowledge- distillation loss with the frozen backbone as the teacher — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  3. reducing the train–inference mismatch and improving acceptance at longer draft lengths — NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called MTP-boosting training phase, MTP-boosting phase, dedicated MTP-boosting phase, Multi-Token Prediction Boosting, post-training MTP boosting.

sources
6
models
3
lab adopt it
1
strongest
used

Documented in

Evidence

7 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 7

A dedicated MTP-boosting training phase further aligned the drafter with the final model.

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA
2 more from this model

Post-training MTP boosting improves average speculative-decoding acceptance length

usedpost trainingin Nemotron 3.5 SuperNVIDIA

dedicated post-training that improves acceptance at deeper draft positions

usedpost trainingin NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 LightningNVIDIA

Post-training MTP boosting improves average speculative-decoding acceptance length

usedpost trainingin Nemotron 3.5 SuperNVIDIA
1 more from this model

After training, a dedicated MTP-boosting phase further improved MTP accuracy.

usedpost trainingin Nemotron 3.5 LightningNVIDIA

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

The goal of MTP Boosting is to make the MTP head match the backbone’s next-token distribution under the input conditions or noise it encounters at inference.

usedunclearin Nemotron 3 UltraNVIDIA
2 more from this model

The goal of MTP Boosting is to make the MTP head match the backbone’s next-token distribution under the input conditions or noise it encounters at inference.

usedpost trainingin Nemotron 3 UltraNVIDIA

iterative MOPD optimization and MTP Boosting procedures

usedinference servingin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑