implementation detail · filed under inference & serving
Recursive shared MTP-head drafting
Applies an MTP head recursively to propose multiple tokens, which the base model verifies in a single forward pass.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 2
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
we use a shared MTP-head formulation applied recursively for several MTP steps
coreunclearin Nemotron 3 UltraNVIDIA
At each decoding iteration, the MTP head is applied recurrently to propose 𝑘 candidate tokens, which the base model verifies in a single forward pass.
coreinference servingin Nemotron 3 UltraNVIDIA
Filed alongside
Other methods under inference & serving :: decoding strategy.
Speculative decodingMulti-Token PredictionDSparkEAGLEDFlashPresence PenaltyBest-of-N scaffoldingNEXTN speculative decodingMulti-layer EAGLESpeculative samplingStandardized sampling configurationMulti-stage candidate filteringTask-specific sampling parametersChat Prefix CompletionConcurrency-aware draft-length tuningDistribution-matched draft-model fine-tuningEAGLE-3-style draft-model fine-tuningFused recurrent replay kernelGrammar-constrained decodingKV-cache sharing between drafter and targetLongest-trace selectionMTP-1 speculative decodingSame-checkpoint target and draft weightsThroughput-aware dynamic verification-length scheduling