the backbone remains frozen and only the MTP head parameters are updated.
4 more from this model
The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.
The MTP head is further trained against the frozen post-trained backbone to improve speculative-decoding acceptance.
A shared MTP head is trained for 100B tokens while the backbone remains frozen
A shared MTP head is trained for 100B tokens while the backbone remains frozen