Open Models Almanac
Methodsmodel architectureprediction head

specific method · filed under model architecture

Hidden-state mixing during recursive MTP training

Conditions an MTP step on hidden states sampled from earlier MTP steps rather than only the immediately preceding step.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See prediction head for the methods filed beside it.

Also called hidden-state-mixing strategy.

sources
2
models
3
lab adopt it
1
strongest
used

Documented in

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

input hidden states are sampled from the sequence of states produced by earlier MTP steps 1,...,k − 1, rather than relying exclusively on step k − 1

usedtraining objectivein Nemotron 3.5 SuperNVIDIA
1 more from these models

we continue applying the hidden-state-mixing strategy described in Section 2.1.

usedunclearin Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA
↑