Open Models Almanac

Prediction head · P1

Multi-token prediction: a training signal and a built-in drafter

An MTP module takes one or two percent of a checkpoint's parameters and has two uses: a training signal whose measured gains are modest, and a drafter that loses value at each further token and can fall behind the backbone as training goes on

Illustration of a line of round stepping stones crossing still water, from a large solid ink-blue stone on the left through two teal stones, one holding a small gold ball, to three coral stones that grow smaller and paler with distance, ending in open water on the right.

Tencent’s Hy3 has 80 decoder layers, according to its config (num_hidden_layers: 80). Its checkpoint has 81. The last one, model.layers.80, holds 3.75 billion parameters, 1.26% of the checkpoint (computed from the tensor headers), and the Hugging Face Transformers class for Hy3 lists it under the keys to ignore on load, so Transformers never runs that layer.

Hy3 is not alone. Of the 39 models in this map whose configs and checkpoints could be read (35 text models and 4 robotics models), 21 ship a sequential module of this kind and six more ship a separate drafter stack under a similar name. Where a Transformers class was read for such a checkpoint, 22 of 23 ignore the extra tensors on load, and a serving engine such as vLLM runs them instead. These layers are a multi-token prediction (MTP) module: they were trained to predict tokens further ahead than the next one.

Labs train them for two reasons, and the two call for different properties. One is a denser training signal: each position in the training text gets more than one target, and the extra loss reaches the backbone. The other is drafting: at serving time the same layers cheaply guess the next few tokens, and the backbone checks all the guesses in one forward pass. For the training signal the loss has to reach the backbone; for drafting the module has to agree with the finished model.

The two uses, in the terms of this series

Some names first. The backbone is the main network. The module is the extra MTP layers, and a drafter is anything that proposes tokens. Gloeckle et al. and several labs say “head” for the same slot, and an output head is the matrix that turns a hidden state into a next-token distribution. The gold token is the true next token from the training text.

A decoder trained on next-token prediction gets one target per position, and generation runs one forward pass per token. In decode, that pass reads the weights and the KV cache (each attention layer’s stored keys and values for earlier tokens) once for a single token, which is what makes decode memory-bound (00).

An MTP module adds targets further ahead. We count extra tokens predicted, as DeepSeek and Medusa do: a module predicting one token beyond the next has depth 1. Gloeckle et al. count the next-token head in their nn, so their n=4n=4 is our depth 3.

Drafting relies on speculative decoding (02). A drafter proposes γ\gamma tokens, the draft length. The backbone then runs one forward pass over them. It outputs a next-token distribution at every position it reads, so that one pass gives a prediction to check each draft against. Decode time goes mostly on reading weights and cache; the cache and the dense weights are read once per pass, so at low load checking several tokens costs about as much as generating one. In an MoE each drafted token also reads its own experts, so the gain is smaller (02).

Figure 1.One speculative step with a draft length of three. The module drafts, one backbone pass checks all the drafts, and the step ends at the first rejection with one token from the backbone itself.

Drafts are accepted in order, up to the first rejection, and the step ends with one token from the backbone. With sampling, a draft xx drawn from the drafter’s distribution qq is accepted with probability min⁡(1,p(x)/q(x))\min(1, p(x)/q(x)), where pp is the backbone’s distribution; after a rejection the backbone samples its own token from what is left of pp. That rule makes the output distribution exactly the backbone’s (Leviathan et al., 2022; Chen et al., 2023), and the probability of acceptance is β=∑xmin⁡(p(x),q(x))\beta = \sum_x \min(p(x), q(x)), which is 1 when q=pq = p. With greedy decoding the test is simply whether the draft equals the backbone’s top token.

Write αk\alpha_k for the probability that draft kk is accepted, given that the earlier ones were. A step then yields on average

1+α1+α1α2+⋯+α1⋯αγ1 + \alpha_1 + \alpha_1\alpha_2 + \dots + \alpha_1 \cdots \alpha_\gamma

tokens, one of which is the backbone’s own token. This is the chain rule, so it needs no assumption of independence; the closed form in the primer, stated there for independent acceptance, needs only that αk\alpha_k is the same at every position. Each further draft counts only if all earlier ones were accepted. Reports quote the mean as “accepted length”. Nemotron 3.5’s report, in its evaluation of parallel drafters, counts the backbone’s own token, and other reports often do not say.

Parallel heads

To Gloeckle et al.’s knowledge, Stern, Shazeer and Uszkoreit (2018) were the first to suggest a speculative scheme of this kind, for translation and image super-resolution, predicting several future positions at once. Gloeckle et al. (2024) replaced the output layers with one transformer layer each and trained language models this way: a shared trunk fsf_s, nn independent heads fhkf_{h_k}, and a shared unembedding fuf_u:

Pθ(xi+k∣x1:i)=softmax(fu(fhk(fs(x1:i)))),k=1,…,n.P_\theta(x_{i+k} \mid x_{1:i}) = \mathrm{softmax}\bigl(f_u(f_{h_k}(f_s(x_{1:i})))\bigr), \qquad k = 1,\dots,n .

The loss is the unweighted sum of the nn cross-entropies. Head 1 is the ordinary next-token head. At inference the other heads can be discarded, or used as drafts.

Figure 2.Parallel heads all read the trunk's states up to position i. Sequential modules form a chain in which each link also takes the embedding of the true next token.

Gloeckle et al. hold compute constant: “when we add n−1 layers in future prediction heads, we remove n−1 layers from the shared model trunk”, and report “no train time or memory overhead”. Their design costs trunk depth, inside the same parameter budget, and nothing is appended. The gains depend on model size, data and task:

  • At 13B parameters the abstract reports 12% more problems solved on HumanEval and 17% more on MBPP (two coding benchmarks), relative to next-token models. At 7B parameters and 200B tokens of training, MBPP pass@1 (the fraction of problems solved in one attempt) goes from 30.0 to 33.8 points and HumanEval from 22.8 to 24.0, with the best temperature per benchmark chosen on test scores.
  • Small models get worse (“worse than the baseline for small model sizes, but outperform the baseline at scale”). On six natural-language benchmarks at 7B parameters and 200B tokens, 4-token prediction regresses a bit where 2-token prediction is on par.
  • With more data, 1T tokens over four epochs, “the improvements diminish”: MBPP pass@1 goes from 40.7 to 43.1, and HumanEval pass@1 from 31.7 to 31.6.

Gloeckle et al. give an illustrative account of the gain: predicting two tokens counts the mutual information between adjacent tokens twice, and the “choice point” tokens that decide what comes next get more weight than ones that follow from them. With unlimited capacity and data, the next-token head’s best output does not depend on the other heads, so any gain on the next-token model is about what a finite trunk learns to represent, and it should shrink with more data and capacity, which the four-epoch result fits. The idea has older relatives: predicting extra targets from a shared representation is multi-task learning, contrastive predictive coding (van den Oord et al., 2018) has one predictor per future offset, and Inception’s auxiliary classifiers (Szegedy et al., 2015) were weighted by 0.3 and discarded at inference, as an MTP module is when a server does not draft.

As a drafter, four parallel heads give 3.5 tokens per forward pass on code, greedy, on a 7B model, and a 3.05× speedup; the paper reports the speedup as constant across batch sizes up to 42 for that model. Each added head adds less: 0.94, 0.84, then 0.72 tokens on code, and 0.88, 0.69, then 0.55 on Wikipedia.

What a parallel head can’t represent

Cross-entropy is separable across heads. Each head’s best possible output is the distribution of its own future token given the prefix, with the tokens in between summed out: a marginal. Because the trunk is deterministic, no sample is drawn inside the forward pass, so what the heads jointly represent is the product of those marginals. They keep all dependence on the prefix and lose dependence among the future tokens given the prefix.

Say two continuations are equally likely, “New York” and “San Diego”. Both heads can be exact: the first gives New or San at 0.5 each, the second York or Diego at 0.5 each. Sampling them independently draws a pair that never occurs half the time. Under the acceptance rule, after New the backbone gives York probability 1 and the head offers it half the time, so β=0.5\beta = 0.5 and a step yields 1.5 tokens at γ=1\gamma = 1. This toy is the extreme case, in which the first token fixes the second completely. In general, for an exact marginal head and a first token sampled from the backbone, β\beta for the second token is 1 minus the expected total-variation distance between its marginal and its conditional distribution, so the loss grows with the dependence between the two tokens. A module that matched the backbone’s next-token conditional exactly would score 2.0, but no real module does.

Toy example. Each table cell is the probability of that pair; the two heads are exact and sampled independently.

True jointYorkDiegoNew0.500.00San0.000.50Product of themarginalsYorkDiegoNew0.250.25San0.250.25
Sampled second token acceptedTokens per step at γ=1
Two independent heads0.501.50
A module that knows the conditional, if it matched the backbone exactly1.002.00
  • a pair the true joint gives probability 0
Figure 3.Toy example with sampled drafts. Both independent heads are exact, yet at 50/50 with the first token fixing the second, half their pairs never occur. Lower the second slider to weaken that dependence. The last row assumes a module that matches the backbone exactly.

In this toy greedy decoding hides the problem: at 0.6 and 0.4, both heads pick “New” and “York”, the pair is real and greedy verification accepts it. Greedy can fail too: with pairs (A,C) at 0.4, (A,D) at 0.2 and (B,D) at 0.4, head 2’s top token is D, while the backbone’s top next token after A is C, so head 2 is rejected every time. Gloeckle et al.’s and Stern et al.’s speedups are greedy, so they say little about sampled drafts.

The argument is not new. EAGLE’s abstract says that feeding a token sequence “advanced by one time step” “effectively resolves the uncertainty”. The Nemotron 3.5 report says of parallel drafters that “tokens selected independently at each slot can be individually plausible yet jointly incoherent”. Hydra (Ankner et al., 2024) observes that “all existing draft heads have been sequentially independent” and, against Medusa’s independent heads (Cai et al., 2024), reports throughput up to 1.31 times higher for its tuned variant. Medusa itself submits the Cartesian product of each head’s top candidates and verifies them as a tree, so a real pair among the candidates can still be accepted.

Sequential modules: feeding the next token in

In DeepSeek-V3 (2024) DeepSeek kept the training-signal idea and changed the wiring, to “keep the complete causal chain at each prediction depth”. Its module jj has one transformer block TRMj\mathrm{TRM}_j, run over the sequence, and a projection Mj∈Rd×2dM_j \in \mathbb{R}^{d \times 2d}, and shares the backbone’s embedding and output head. In our notation:

si′ j=Mj[RMSNorm(sij−1); RMSNorm(Emb(xi+j))],sj=TRMj(s′ j),Pi+j+1j=OutHead(sij).s'^{\,j}_i = M_j\bigl[\mathrm{RMSNorm}(s^{j-1}_i);\ \mathrm{RMSNorm}(\mathrm{Emb}(x_{i+j}))\bigr], \qquad s^{j} = \mathrm{TRM}_j(s'^{\,j}), \qquad P^j_{i+j+1} = \mathrm{OutHead}(s^j_i).

Here si0s^0_i is the backbone’s hidden state at position ii, from which the backbone’s own head predicts xi+1x_{i+1}; [ ; ][\,;\,] is concatenation; RMSNorm is LayerNorm without the mean; and Emb(xi+j)\mathrm{Emb}(x_{i+j}) is the embedding of the gold token. Module jj predicts offset j+1j+1 from position ii.

Module jj can use the trunk’s state at ii and the next jj gold tokens. If those were enough, and one block could compute the function, the best it could do is the next-token conditional, the distribution the backbone estimates. In practice it gets a coarser estimate: its state at ii is jj tokens stale, the newest token passes through only one block, and anything that needs several layers of processing of that token is out of reach. Assuming the block sees only earlier positions, which the paper does not state, the dependence between draft tokens that parallel heads lose is available here through one token embedding.

The training signal is different too. The module’s gradient reaches the backbone, so the state at ii is pushed to carry what xi+2x_{i+2} needs as a function of an xi+1x_{i+1} it has not yet seen. DeepSeek-V3 gives both of the hypotheses in play, a denser signal and representations that “pre-plan”.

Feeding the gold token is teacher forcing: training conditions on the true token, while generation has only the model’s own. Every language model trained this way has that gap. With depth 1 the token at drafting is the backbone’s own verified next token, so the module faces the same gap as the backbone and no new one. Scheduled sampling (Bengio et al., 2015) feeds generated tokens during training to close the gap, and Gloeckle et al. reject it as “inapplicable to language modelling due to the discrete nature of text”.

The total loss adds λ\lambda times the mean of the module losses, with DD the number of modules (L\mathcal{L} is a loss here, since LL is layers in this series):

LMTP=λD∑j=1DLMTPj.\mathcal{L}_{\mathrm{MTP}} = \frac{\lambda}{D}\sum_{j=1}^{D} \mathcal{L}^{j}_{\mathrm{MTP}} .

Three labs state a weight in a document: DeepSeek, Xiaomi and NVIDIA. DeepSeek-V3 uses 0.3 for the first 10T tokens and 0.1 for the remaining 4.8T. DeepSeek-V4 uses 0.3 “for most of the training, and to 0.1 upon the start of learning rate decay”. Xiaomi’s MiMo-V2-Flash uses 0.3 in its first pre-training stage and 0.1 afterwards. NVIDIA’s Nemotron 3 Super uses 0.3 and Nemotron 3 Ultra 0.1, which is 0.05 for each of its two steps, the λ/D\lambda/D form. Hy4’s config has mtp_loss_factor: 0.1, and no document states it.

The first cost is that the module is appended: DeepSeek-V3’s ablation adds it on top of the same backbone, and its parameters and training compute are extra. Sharing the embedding and output head keeps the increase small, at a second cost: the shared layers accumulate small MTP gradients together with much larger backbone gradients. In Nemotron 3 Ultra’s training, moving gradient accumulation for the shared output layer from FP32 to the 16-bit BF16 format “essentially lost” the MTP steps’ contribution, in the report’s account; the second step’s loss spiked first and training diverged at about 8T tokens (S1).

After training the modules can be discarded. DeepSeek-V3’s report treats that as the default, and adds that “we can also repurpose these MTP modules for speculative decoding”. That a depth-1 module makes a natural drafter is our reading of the design.

What the training signal is worth

Two labs publish ablations of the training signal, and both use test models. DeepSeek-V3 compares models with and without an appended depth-1 module on a small mixture-of-experts model (MoE: each token uses only a few of many feed-forward experts; 15.7B parameters, 1.33T tokens) and a large one (228.7B parameters, 540B tokens). NVIDIA’s Nemotron 3 whitepaper uses an MoE model with 8B active parameters (the ones a token actually uses), trained on 1T tokens. The chart shows every benchmark row they print except Pile-test.

Figure 4.Score change from adding an MTP objective, in points, for every benchmark in the two published ablations (DeepSeek-V3 Table 4; Nemotron 3 Table 2). Test models only, base models with no post-training, no error bars in the sources. Pile-test, measured in bits per byte, is left out.

In DeepSeek-V3’s table eight of nine benchmarks rise on each model and one falls (NaturalQuestions by 0.4 on the small model, MMLU by 0.9 on the large). HumanEval rises most, by 9.2 points on the large model, on a small benchmark, and neither table gives an uncertainty. Nemotron 3’s mean is 1.58 points (computed), which is the whitepaper’s “roughly 2.4% on average” in relative terms, and the paper calls these “modest improvements in model quality”. All of these are base-model scores, with no post-training.

No released model has an ablation behind it. In the documents we read no lab reports that the training signal did nothing, and labs publish ablations of what they go on to ship, so these gains are likely to overstate the typical effect.

Re-applying one module over several draft steps

The number of modules does not set how far a model drafts. vLLM picks a module with spec_step_idx % self.num_mtp_layers, so one module can be applied again for the next draft step. GLM-5.3’s config has num_nextn_predict_layers: 1, and its vLLM recipe runs that layer for 5 speculative tokens.

From the second step on, the module reads its own earlier output, and only drafts that were accepted matter, since a rejection discards everything after it. The gap is in how the input is produced. NVIDIA’s Nemotron 3 Super report (a source outside this site’s corpus) describes one form of it: “Reusing a fixed-offset head introduces a training–inference mismatch: the head is trained under ground-truth hidden states but, at inference, conditions on its own generated states.” (Ground-truth states are those the backbone computes from gold tokens.) The Nemotron 3 Ultra report describes a second, for a head trained unrolled, that is, applied to its own output several times as an RNN is. In training, the second step’s inputs “all … originate from the previous MTP step”, while at inference the inputs are the backbone’s states for the prefix and the module’s own state only for the new position.

  • token from the backbone
  • token from the module
  • not trained at this step
first draft token fedstep 1s0xi+1M1step 2s1d1M2step 3s2d2M3step 4s3d3M1step 5s4d4M2top row: state read; middle row: token fed.M1, M2, M3: three separate sets of weights.first draft token fedstate readtoken fedmodulestep 1s0xi+1module 1step 2s1d1module 2step 3s2d2module 3step 4s3d3module 1step 5s4d4module 2

Three sets of weights, one per depth. Steps 4 and 5 wrap around to modules 1 and 2. From step 2 each module reads a state and a token that came from the drafting module.

The wrap-around is vLLM's rule: spec_step_idx % self.num_mtp_layers.

Figure 5.How a checkpoint's modules map to draft steps in vLLM. A single module can be re-applied for as many steps as the server asks, including steps it was never trained on.

Labs narrow the gap in three ways. The first is to train the module unrolled, each step reading the previous step’s states, as in Nemotron 3.5 Lightning’s warm-up, which the report calls “naive” because the mismatch above remains. GLM-5 shares the parameters of three MTP layers during training, which keeps the memory of the draft model equal to one layer. The report’s comparison, 2.76 mean accepted length at four steps against 2.55 for DeepSeek-V3.2 on private prompts, is between two different models, so no ablation isolates the sharing. The second is to mix the input: Nemotron 3 Ultra’s boosting stage and Nemotron 3.5 Super’s warm-up sample a step’s input states from earlier MTP steps. The Ultra report’s reason is that this “exposes the head at training time to a similar noise it encounters at inference”. The third is to feed the drafter its own outputs from earlier steps in training (EAGLE-3’s training-time test), which Kimi K3 does. The token input in the first two is the next token of the training sequence, as far as the reports state; in Boosting that sequence is a rollout the model generated.

Other labs keep distinct modules. Step-3.7 and Step-5 ship three, with a separate output head for each, and Inkling ships eight. Distinct modules add parameters instead of training steps, and at serving a module further along the chain reads a drafted token where training gave it the gold one. Xiaomi’s MiMo models ship three too, but the Flash report says a single head is attached in pre-training and “replicated K times” afterwards, and vLLM uses only the first.

Training the module against the backbone

After pre-training on documents, a model gets supervised fine-tuning (SFT) on demonstrations, then reinforcement learning (RL) in which it generates its own samples (rollouts) to be scored. The backbone is still being trained through these stages. Nemotron 3.5 Super’s own head has a mean accepted length of 4.05 at draft length 7 and temperature 1 at the SFT checkpoint, and 3.79 at the final backbone; the two backbones differ, so the numbers do not isolate how much of the fall is drift. The labs’ responses differ, stage by stage:

ModelPre-trainingPost-training (SFT, RL)Head-only stage, backbone frozen
DeepSeek-V3 / V4module trained jointly, λ\lambda 0.3 then 0.1not statednone stated
DeepSeek-V4.1MTP omittedDSpark trained alongside, no gradient into backboneDSpark, dedicated stage after pre-training
MiMo-V2-Flashone head, λ\lambda 0.3 then 0.1replicated K times, jointly trained
MiMo-V2.6an SWA and dense-FFN MTP“MTP-3” inherited from SFT, RL rollouts use a DFlash drafter
Nemotron 3 Superλ\lambda 0.3SFT 0.3“MTP Healing” after RL
Nemotron 3 Ultraλ\lambda 0.1trained throughoutMTP Boosting, 7 steps, KL to backbone
Nemotron 3.5 Superdistillation λ\lambda 0.1SFT 0.3, jointly trained100B-token warm-up before distillation; 4,000-step warm start at SFT; Boosting with LK loss
Kimi K3pre-trained with one MTP layerfine-tuned into an EAGLE-3 draft

Across these stages the module’s training target changes in four steps: gold tokens in pre-training, then responses generated for the RL prompts (the report does not say by which checkpoint), then the backbone’s distribution, then acceptance. Nemotron 3 Super’s Healing uses negative log-likelihood on generated responses. Nemotron 3 Ultra’s Boosting replaces the sampled target with the whole distribution, a “temperature-scaled forward-KL loss against the backbone’s logits” with “the standard cross-entropy term against the gold token” disabled, temperature τ=2\tau=2 (written TT in the report), and seven unrolled steps. This is distillation (Hinton et al., 2015), with the frozen backbone as teacher, and Medusa-1 had already frozen the backbone for a head-only stage. Kimi K3 uses the LK loss, described in its report as “the negative logarithm of the acceptance rate itself”, which in this article’s terms is −log⁡β-\log \beta; Nemotron 3.5 names “the LK loss (Samarin et al., 2026)” without saying which of that paper’s two variants.

A loss against gold tokens rewards predicting the text. A loss against the backbone rewards predicting the model the drafts will be checked against, on the prefixes the model itself produces. The accept-or-reject coin is discrete and passes no gradient, so training needs a differentiable objective. Under sampling β\beta is one: it integrates out both the draw and the coin, and it depends on the module through qq. Greedy agreement has zero gradient almost everywhere, and KL or cross-entropy stand in for it. In Nemotron 3.5 Super, mean accepted length at draft length 7 and temperature 1 on SPEED-Bench’s qualitative set is 3.79 for the native head at the final backbone, 4.44 for a head with 4,000 head-only steps at the SFT checkpoint, and 4.61 after 25,000 more steps against the final backbone; no row of the table varies the loss.

With the backbone frozen “there is no risk of regressing backbone quality”, and the training signal is off for that stage. In the recipes above that also train the module jointly, the two uses share the same weights in the other stages.

Some labs replace the drafter, and one drops MTP from pre-training:

  • A drafter here can be a small head reading the backbone’s features (EAGLE-3), or a block of tokens proposed at once from backbone features (DFlash, DSpark). I5 compares them.
  • DeepSeek-V4.1 omits MTP from pre-training. Then, in a stage with the backbone frozen, it trains DSpark, whose report describes “three Transformer blocks with a sliding attention window of 128 tokens” that compute five draft positions in parallel, with “a lightweight Markov head” modelling dependencies among the draft tokens. Training then continues “alongside the backbone, without propagating gradients from the DSpark objective into the backbone”. The report does not measure what giving up the training signal costs.
  • Kimi K3’s config has num_nextn_predict_layers: 0 and its checkpoint has no MTP tensors, but the report says it was pre-trained with an MTP layer, which was fine-tuned into an EAGLE-3 draft unrolled for seven steps.
  • MiMo-V2.6’s report uses “MTP” for two things: a sequential SWA and dense-FFN module in pre-training, and a five-layer drafter “following the block diffusion design of DFlash” for serving, which predicts seven tokens in one pass. Its RL rollouts default to a block-6 DFlash, “replacing the multi-token prediction (MTP-3) configuration inherited from SFT”, at 31.3% higher accepted length.
  • MiniMax’s M3.1-Flash card says DSpark “replaces the earlier EAGLE-like MTP approach”, without saying what was used in training.

Measured speedups from drafting

Reports disagree about how the gain depends on load, and they measure different things. At large batch every drafted position is verified for every sequence, so verification starts to cost compute (02). Nemotron 3 Ultra’s report says that at small batches “high draft length wins for latency”, with a peak of 2.89× at draft length 6 at batch 1 on GB200, and that at large batches “lower draft lengths, or disabling MTP, often win”. vLLM’s Qwen3.5 recipe says MTP-1 “degrades throughput under high concurrency because speculative tokens consume KV cache capacity”. Step-3.7’s table fixes three speculative tokens and raises concurrency, and the gain from NVFP4 with MTP over NVFP4 alone is 1.13× at 8 concurrent, 1.26× at 32 and 1.45× at 64 (GB200, four GPUs in tensor parallelism). MiMo-V2-Flash’s table at batch 32 to 128 is not monotone, with 1.82× to 1.99× at accepted length 2.8, and the report attributes the variation to “the corresponding computation and I/O demands as well as kernel efficiency”. vLLM’s Qwen3.8 recipe for one H100 FP8 path records MTP as worse at both concurrencies it tried (8 and 64), at about 36% acceptance. Nemotron 3 Ultra reports 1.46× faster mean rollout generation at five drafted tokens in RL, with the benefit concentrated in the slowest generations, and GLM-5 reports the same long-tail benefit without a number.

The module inside a hybrid

Hybrids mix attention layers with recurrent layers (Mamba, Gated DeltaNet, KDA) that keep a fixed-size state and no growing KV cache. In the hybrids that ship MTP, the module itself is an attention layer with no recurrent layer. Nemotron’s module is ["attention", "moe"], Qwen3.5 and Qwen3.6 use a full-attention layer, and GLM-5.3-Flash has sparse attention with an indexer (the part that picks which entries each query reads). On Qwen3.8 the sources conflict: the config says mtp.layer_types: ["full_attention"], the report says the module’s attention is replaced by QSA, a sparse attention, and the checkpoint has an indexer. Outside recurrent hybrids Step-3.7 and Step-5 use sliding-window attention with a dense feed-forward network, Inkling’s config and vLLM code mix sliding and full across its eight depths, and Xiaomi keeps “a small dense FFN rather than MoE” and sliding-window attention “to prevent it from becoming a new inference bottleneck”.

The module costs something in every phase. In storage it adds its parameters plus, if it has attention, a KV cache of its own: an extra attention layer would add about 7% to Qwen3.5-397B’s attention-layer cache and about 8% to Nemotron 3 Ultra’s (computed from 15 and 12 attention layers, assuming the same width), which is larger than its parameter share. In prefill every token of a long prompt passes through the module to fill that cache, at a FLOP cost near its share of active parameters. Gemma 4’s MTP head avoids this: its small block cross-attends to the main model’s KV cache, “eliminating the need for MTP prefill”. In decode each draft step reads the module’s active weights, the output head (the d×d \times vocabulary matrix, shared or not) and the module’s KV cache.

Two further costs come from running alongside the backbone. Rejected drafts must be undone, which a KV cache does by dropping entries and a recurrent state cannot, so engines keep copies or replay inputs (T2). Nemotron 3.5’s report says the draft module’s KV cache cannot use a different precision from the backbone’s. Sparse attention can also save work by reusing its top-k entries across draft steps: GLM-5.2, 5.3 and 5.3-Flash set index_share_for_mtp_iteration: true (T3). In Qwen3.5-Omni’s audio Talker, “MTP module” names a predictor of the current frame’s residual codebooks (X2).

The designs side by side

DesignDependence between draftsToken fed inExtra parametersGradient to backboneDraft
Parallel heads (Gloeckle)none; product of marginalsnonenone in Gloeckle (layers moved from trunk)yesone pass, all offsets
Sequential, distinct modules (DeepSeek-V3 style, Step, Inkling)through one fed token per depthgold token in training, accepted draft laterone layer per depth, appendedyes, weighted by λ\lambdaone step per token
One module, unrolled (Nemotron, GLM-5)same, through re-applicationas above; some train on mixed hidden statesone module’s worth, used for several stepsyes, then head-only stagesone step per token, any length
Separate drafter (DSpark, DFlash-style)none, or a first-order Markov termbackbone featuresa small stack, 0.2% to 6.5% of the checkpointnoneone pass per block

Where the models in the map sit

Automated checks of the model code disagreed with the checkpoints on this method (Hy3 came back absent although its checkpoint has the layer), so the counts come from configs and checkpoint tensors, for the 39 models with data. 21 ship a sequential MTP module, from 8 of the 19 labs with data (DeepSeek, NVIDIA, Qwen, StepFun, Tencent, Thinking Machines, Xiaomi, Z.ai), and 6 more ship a DSpark-type stack (five DeepSeek checkpoints and Mistral Large 4.0). A nonzero MTP field is declared by 24; MiniMax-M3 declares one and ships no such tensors in its public repository, and three MiMo checkpoints ship modules with no field.

ModelConfig fieldIn the checkpointAttentionFeed-forwardShare of paramsλ\lambda stated
DeepSeek-V3.2num_nextn_predict_layers: 11 modulesparseMoE1.70%
DeepSeek-V4-Flash / Pro11 module (adds rather than concatenates, in vLLM)backbone’sMoE2.27% / 1.62%0.3 then 0.1
DeepSeek-V4-Flash-0731 / V4-Pro-08131, dspark_block_size: 53 DSpark layersMoE6.53% / 4.70%not stated
DeepSeek-V4.1-Flash3, dspark_block_size: 53 DSpark layerssliding windowMoE1.53%*MTP omitted
GLM-5.2, GLM-5.311 modulesparseMoE1.32%
GLM-5.3-Flash11 modulesparseMoE2.31%
Hy311 modulefullMoE1.26%
Hy4-preview1, mtp_loss_factor: 0.11 modulesparseMoE1.29%config only
Qwen3.5-397Bmtp_num_hidden_layers: 11 modulefullMoE1.64%
Qwen3.6-35B11 modulefullMoE2.35%
Qwen3.8-Flash11 modulesources conflictMoE1.45%*
Nemotron 3 Ultra11 module of an attention and an MoE layerattentionMoE2.00%0.1
Nemotron 3.5 Super VL / Lightning11 module, as aboveattentionMoE2.36% / 4.06%0.1, Super SFT 0.3
MiMo-V2.5 / V2.5-Pro / V2.6-Flash / V2.6-Pro3 on V2.6-Flash only3 modules, copies of one headsliding windowdense0.19% to 0.62%0.3 then 0.1 (V2-Flash)
Step-3.7-Flash / Step-5-Preview33 modules, own output headssliding windowdense1.21% / 0.42%
Inkling88 modulessliding and fulldense0.55%
Mistral-Large-4none3 drafter layersdense0.20%
Kimi K30none (report: pre-trained with one)
MiniMax-M3num_mtp_modules: 7 and num_nextn_predict_layers: 1none in the public repository

The share is of all parameters in the checkpoint, computed from tensor headers. The two starred rows include a large lookup table (196.6B Engram on V4.1, 51.2B n-gram on Qwen3.8); without it the shares are 2.58% and 2.03%. Gemma 4’s MTP head is not in the checkpoint read.

Figure 6.One draft step's active parameters as a percentage of a full forward pass, computed from checkpoint tensors. Active parameters are a proxy for the weight bytes a step reads. The output head and the module's KV reads are left out of the step's cost, so its true cost is higher.

A draft step reads a small part of a full pass’s weights: for one Hy3 step, 0.28B of the 20.68B active parameters a full pass uses, or 1.35% (computed). That is why drafting can pay. Each step also reads the output head and the module’s KV cache. In Nemotron 3.5’s simulated latency at draft length 5, with the module and head unquantized, drafting is 23.5% of the total and about half of drafting is the head.

Easy to get wrong

  • num_nextn_predict_layers often names something else. DeepSeek-V4-Flash-0731’s says 1 and the checkpoint has three DSpark layers. V4.1-Flash’s says 3 and pre-training had no MTP module. Three MiMo checkpoints have three modules and no field. MiniMax-M3’s config has two different numbers.
  • A Transformers class that ignores mtp.* does not mean the model lacks MTP. It means that class doesn’t run it.
  • Count conventions differ. DeepSeek’s depth 1 is one extra token; Gloeckle’s n=4n=4 includes the next-token head, so depth 3. Both differ from draft length, which is set at serving time.
  • A relative gain and a point gain differ: at 7B Gloeckle’s +3.8 on MBPP pass@1 is 12.7% relative, and the 17% in the abstract is a different model.
  • Module, head and layer are not the same count in Nemotron’s table: one module of an attention layer and an MoE layer, trained as two unrolled steps, each a loss term weighted 0.05.
  • “MTP speedup shrinks at large batch” holds in some sources and not in others, and the setups differ. A single table does not settle it.

Check yourself

In the New York / San Diego toy, let P(New York) be 0.7, with the first token fixing the second. How much of the mass from two independent heads falls on pairs that never occur, and how many tokens per step does the acceptance rule give at γ=1\gamma=1?

Pairs that never occur: 0.7⋅0.3+0.3⋅0.7=0.420.7\cdot 0.3 + 0.3\cdot 0.7 = 0.42. After New the backbone gives York probability 1 and the head offers it with 0.7, so β=0.7\beta=0.7; after San, β=0.3\beta=0.3. The mean is 0.7⋅0.7+0.3⋅0.3=0.580.7\cdot 0.7 + 0.3\cdot 0.3 = 0.58, and a step yields 1.58 tokens.

A depth-1 module trained only on a single step is re-applied for a second draft step. What does it read as its input, compared with what it read during training?

Its hidden-state input is its own output from the previous step, not a state the backbone computed, and its token input is the previous draft, accepted if we got this far. In training it read a backbone state computed from gold tokens and the gold next token. The difference is in how the input was produced; a rejected draft ends the step, so it cannot feed a later one.

Why does a head-only stage with a loss against the backbone’s logits make sense for drafting, and what does it give up?

Drafts are verified against the backbone, so what matters is what the backbone would say on the prefixes the model produces; gold tokens are only a proxy for that. The stage gives up any effect on the backbone: with the backbone frozen, no gradient reaches it, so the stage adds nothing to the training signal.

In the map

Methods

Taxonomy nodes

Further reading

  • Stern, Shazeer, Uszkoreit, Blockwise Parallel Decoding for Deep Autoregressive Models (2018), arXiv:1811.03115: predicting several future positions and verifying them.
  • Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction (2024), arXiv:2404.19737: parallel heads, the choice-point argument, the appendix on alternative head wirings.
  • DeepSeek-AI, DeepSeek-V3 (2024), arXiv:2412.19437, §2.2 and §4.5.1: the sequential module equations and the ablation table.
  • Cai et al., Medusa (2024), arXiv:2401.10774: parallel heads fine-tuned after pre-training, with tree-structured candidates.
  • Ankner et al., Hydra (2024), arXiv:2402.05109: sequentially dependent draft heads against Medusa’s independent ones.
  • Nemotron 3 Ultra report, arXiv:2606.15007, §2.1, §3.4 and §5.1: shared-weight heads, MTP Boosting and the operating-point discussion.
  • GLM-5 report, arXiv:2602.15763, §2.1: parameter sharing across MTP layers.
  • DeepSeek-V4.1-Flash report, §2.1 and §2.4.3: dropping MTP from pre-training and training DSpark separately.
  • Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022), arXiv:2211.17192: the acceptance rule and the tokens-per-step formula.
  • Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (2015), arXiv:1503.02531: the temperature-scaled loss used in MTP Boosting.
↑