Open Models Almanac
Methodsmodel architectureprediction head

implementation detail · filed under model architecture

Frozen-backbone MTP-head-only fine-tuning

Trains MTP-head parameters while keeping the backbone frozen, including a post-trained backbone in some evidence.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See prediction head for the methods filed beside it.

Also called the backbone remains frozen.

sources
3
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

5 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 4core 1

the backbone remains frozen and only the MTP head parameters are updated.

coreunclearin Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA
4 more from this model

The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.

usedpost trainingin Nemotron 3.5 Super VLNVIDIA

A shared MTP head is trained for 100B tokens while the backbone remains frozen

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA

A shared MTP head is trained for 100B tokens while the backbone remains frozen

usedtraining objectivein Nemotron 3.5 Super VLNVIDIA
↑