Kimi K3 decodes about 113 tokens per second for a single user on SGLang, after LMSYS tuned its kernels. SGLang and vLLM are the two open-source serving engines that the recipes for these models target. With a drafter attached that guesses several tokens ahead, the same deployment reaches about 423 tokens per second (LMSYS, July 2026; the post does not say which prompts). On chat traffic the same kind of drafter has about 2.7 tokens accepted per check of eight positions, so most of what it checks is thrown away. At 256 concurrent sequences, a version that trims the checks per request delivers 68% more throughput than one that checks all eight (LMSYS).
Nearly every serving recipe for the models in this map turns speculative decoding on, with drafters as different as a built-in multi-token prediction (MTP) layer run five times, a one-layer EAGLE-3 head, a block drafter that proposes seven tokens in one pass, and a lookup in earlier text. Published speedups run from below 1× to above 5×. What separates them is how many tokens the model accepts per check, what each check costs at the load it runs under, and what the drafter was trained on. Most of these models are mixtures of experts (MoE), where each token runs through only a few of many expert feed-forward blocks, and that changes what a check costs.
What you need from 02 and P1
In decode a model produces one token per forward pass, and at low load the pass is limited by reading weights and the KV cache (the stored keys and values of earlier tokens), not by arithmetic (00). Speculative decoding uses that slack. A drafter proposes tokens; recipes call num_speculative_tokens. The large model, called the target here, runs one pass over the last accepted token plus the drafts, so it verifies positions and gets a next-token distribution at each. It keeps the drafts up to the first one it rejects, then adds one token of its own: a correction at the rejected position, or a bonus token after the last draft if all were accepted.
- last accepted token
- draft
- rejected draft
- kept draft
- the target's own token
Placeholder words. Each position's distribution judges the draft one place to its right; the distribution after the last draft gives the bonus token when every draft is kept.
The acceptance length (AL) is the number of tokens a verification step emits, that last token included. It runs from 1 (every draft rejected) to . vLLM, NVIDIA, DFlash and DSpark all count it this way; a few papers count accepted drafts only, which is one less. A step costs one draft phase plus one verification pass, so the speedup over plain decoding is
the form MagicDec (Sadhukhan et al., 2024) and DFlash use. Since a verify step costs at least a plain one, the speedup can never exceed AL. 02 introduced why grows with load, and P1 covers how MTP modules are built and trained. Below, is the number of sequences decoding at once.
What acceptance length measures
One token at a time
Write for the target’s next-token distribution and for the drafter’s. Speculative sampling (Leviathan et al., 2023; Chen et al., 2023) accepts a drafted token with probability and, on rejection, samples a replacement from renormalized. The output is then distributed exactly as , so the method is lossless (up to floating-point numerics, as vLLM’s docs add). The probability that a draft is accepted is
one minus the total-variation distance between the two distributions (Leviathan et al., Theorem 3.5; the paper calls the distance ). In probability terms the rule is a maximal coupling of and , so no lossless test of a single draft can accept more often: how much probability mass the drafter shares with the target is the whole of its quality, one position at a time.
Greedy decoding is the case where both distributions put all their mass on their top token: is 1 if the drafter’s top token is the target’s and 0 otherwise. With sampling, also counts mass the two share on other tokens. A higher temperature can move either way, since the sign depends on the two distributions; in the measurements it mostly lowers it. In Leviathan’s T5 experiments every drafter and task loses 0.11 to 0.18 of acceptance from temperature 0 to 1 (Table 2), while the GPT-like and LaMDA drafters in Table 3 barely move. For Nemotron 3.5 Super’s built-in MTP head at draft length 7, AL falls from 4.288 greedy to 3.787 at temperature 1 (report, Table 9).
How the drafter samples matters as well. If it proposes its own top token, its is a point mass on that token, and the rule above accepts it with probability , the target’s probability of that one token. If it samples from its full distribution, the acceptance is , which is larger when the target’s own distribution is flat, as at a high temperature. vLLM’s recipe for Mistral Large 4 (three drafts) measures the difference at temperature 1.5: AL 2.04 when the drafter proposes its argmax, 2.36 to 2.53 when it samples.
Along the draft
With the token-by-token rule, the -th draft is kept only if all before it were, so
where is the probability that draft is accepted given that drafts 1 to were. If every equals one , the sum is the capped geometric series that 02 gives. Several recipes now use block verification instead ("rejection_sample_method": "block" in vLLM, --speculative-use-block-verification in SGLang), which checks every prefix of the draft and keeps the longest one accepted, with a matching correction distribution so the output stays lossless; with sampling it keeps at least as many tokens on average. Greedy requests verify as before.
Measured per-position rates are not constant. On Qwen3-4B chat prompts, the DSpark paper measures an EAGLE-3 head’s conditional acceptance rising from 0.53 at the first position to 0.74 at the seventh, and DFlash’s falling from 0.72 to 0.63. These rates pool every request that reached position , and only the easier ones get that far, so a pooled rate can rise along the draft even if no single request’s rate does. The fall for DFlash is the stronger signal: a drafter that guesses all positions at once guesses the later ones with less information.
It still helps to turn a reported AL into the constant that would produce it. Kimi K3’s chat AL of 2.7 at seven drafts corresponds to (computed). At that rate three drafts would already give 2.31 tokens, so drafts four to seven add 0.39 tokens per step for four more verified positions per sequence. Nemotron 3.5 Super’s AL values above give greedy and 0.77 at temperature 1. The constant is an average over requests that differ, and averaging hides how uneven they are. AL is convex in , so a population of easy and hard requests has a higher mean AL than its mean would give, and a single cut-off for everyone loses more of it than the constant suggests: in two-group mixtures with the same mean AL of 2.7 (for example half the requests at 0.8, half at 0.19), cutting every request to three drafts loses 0.6 to 0.8 tokens instead of 0.39 (computed). That gap is the case for choosing the length per request, which the section after next takes up.
- kept (solid) or rejected (outline)
- what that position adds to the accepted length
7 drafted: accepted length 2.70 tokens per step (the bonus token included), from 8 verified positions per sequence: 0.34 accepted tokens per verified position.
Drafts 5 to 7 add 0.22 tokens between them, for 3 more verified positions per sequence.
Limit with endless drafts: 1 / (1 − α) = 2.78.
Accepted length counts the token the target adds itself. Each preset's α is the constant rate that reproduces a reported accepted length at seven drafts (Kimi K3 chat 2.7, LMSYS; Nemotron 3.5 Super native MTP 4.288 greedy and 3.787 at temperature 1, its report's Tables 8 and 9). The rising and falling profiles are illustrative shapes (α − 0.10 to α + 0.10, and α + 0.05 to α − 0.05, over positions 1 to 7, then held; every rate clamped to 0.01–0.99), after the per-position curves in the DSpark paper; they are not fitted.
What a verification step costs
A model to reason with
Costs here come from a roofline model: each operation takes as long as the larger of its bytes read divided by memory bandwidth and its arithmetic divided by peak FLOPs. The ratio of the two peaks is the ridge point, 412 FLOP per byte for an H200 in FP8 (1,979 TFLOPS over 4.8 TB/s); an operation that does fewer FLOPs per byte than that waits on memory. The worked example is GLM-5.3 on eight H200s in FP8, as in vLLM’s recipe, which drafts five tokens with its MTP layer. Per GPU and step, its non-expert weights are 3.67 GB, each of its 75 MoE layers holds 256 experts of 4.72 MB per GPU, and 8 experts serve each token. The numbers below are computed from that model with ideal kernels (compute_h.py); they show directions, not measured times.
At batch 1
At batch 1 the extra verified positions add almost no bytes for the dense parts, which are read once for all of them, and for attention, which reads the cache once. The experts are another matter. Each verified position picks its own 8 experts, and the expected number of distinct experts that tokens touch per layer is , with experts and per token. For six positions that is about 44 of 256, against 8 for one token (computed; independent uniform routing assumed, which overstates the spread, since neighbouring tokens often share experts). So the GLM-5.3 verify step reads 19.5 GB per GPU against 6.6 GB for a plain step, and on the roofline takes 4.06 ms against 1.38 ms.
Real batch-1 steps take longer than any roofline: LMSYS shows a 7.3 ms verify step in an example profile, beside DeepSeek-V4-Pro (a different model) on eight B300s, since kernel launches and communication between GPUs add a fixed time to every pass. That fixed time is why drafting pays at batch 1 even for an MoE: the saving is in steps. Leviathan’s model makes the trade explicit when verification costs one plain step and a draft step a fraction of one:
with an optimal that grows with and shrinks with : at and it is 8, for 3.09× (computed).
As the batch grows
A batch of sequences verifies positions per step, and the three parts of the step respond differently.
The dense matrix multiplications turn compute-bound early. An FP8 weight does 2 FLOPs per token, so with 412 FLOP per byte to fill, a dense operation becomes compute-bound at about 206 tokens per step: 34 sequences verifying six positions each. Past that point every verified position costs arithmetic, which is 02’s argument, and at the limit a verify step costs plain steps.
The experts behave the other way. A plain step at batch 64 already touches about 222 of GLM-5.3’s 256 experts per layer, and verification adds little: 256 against 222. Each expert only becomes compute-bound when it sees about 206 tokens, which takes tokens per step. So in an MoE the drafts’ extra expert reads cost most at small batch, and fade as the batch grows.
Attention depends on the layout. Under tensor parallelism (TP), every weight matrix is split across the eight GPUs, but GLM-5.3’s compressed attention cache has a single head, so each GPU keeps a full copy and the node holds only 73 sequences of 8K tokens (I3). With attention run data-parallel (each GPU serving its own sequences with all heads) and experts split across GPUs, the cache is no longer copied, and after each GPU stores all the attention weights (10.2 GB) the node holds about 405. Attention with all 64 heads on one GPU does about 212 FLOPs per byte of cache for one query, near the 206 of BF16 arithmetic, and about 1,270 for six, so the verify step’s attention becomes compute-bound too. For grouped-query attention with few FLOPs per cached byte, the cache read stays the bound and verification stays cheap at long contexts; MagicDec (Sadhukhan et al., 2024) gets speedups at batch 32 to 256 that way, with a drafter whose own cache is kept small. Most shipped block drafters limit their own cache with a sliding window. Sparse attention adds one cost: if each verified position selects its own top-k entries, as DeepSeek’s sparse attention (DSA) does, a verify step can read up to times as many.
Put together, the best draft length moves with load in both directions (computed, ideal kernels, AL 3.19 at five drafts). Under TP at batch 1 a single draft does best (1.19×; five drafts only break even), because the drafts’ experts dominate. Around 64 sequences seven drafts do best (2.67×). With data-parallel attention and 405 sequences the verify step is 1.38 times a plain step at five drafts; three and five drafts tie at 2.1× and seven falls to 2.0×. Apart from long drafts under TP at batch 8 or less, the model shows no loss at this acceptance within one node; real kernels run well below peak, so measured losses start earlier, and large deployments that pool many nodes’ sequences per expert push the experts toward their compute limit too.
- γ = 1
- γ = 3
- γ = 5
- γ = 7
- fixed latency
- weights other than routed experts
- routed experts
- attention and KV cache
- MTP draft passes
B = 64, γ = 5, α = 0.737 (accepted length 3.19): modelled speedup 2.54×. Verify step 22.9 ms = 1.17 plain steps (19.5 ms), plus 5 draft passes 1.6 ms.
Verify step: non-expert weights 2.9 ms (bytes), routed experts 18.9 ms (bytes), attention and KV 1.1 ms (FLOPs). Experts read per MoE layer: 222 of 256 in a plain step, 256 in the verify step. Best of γ = 1, 3, 5, 7 here: 7.
A per-operation roofline, not a measurement: on each GPU, the non-expert weights, the routed experts and attention each take the longer of their bytes at 4.8 TB/s and their arithmetic at the H200's dense peak (FP8 1,979, attention in BF16 989 TFLOPS), with ideal kernels; FP8 weights and KV cache, 8K context. Each segment of a bar is labelled by the limit it hit. Experts touched assume every token picks 8 of 256 experts independently, so the small-batch penalty is an upper bound. In TP8 every GPU keeps a copy of the compressed cache, so the node holds 73 sequences; with attention data-parallel each GPU keeps only its own sequences' cache but holds all the attention weights (10.2 GB more per GPU), so the node holds about 405. Accepted length is (1 − α^(γ+1)) / (1 − α); 0.737 reproduces GLM-5.3's 3.19 at five drafts (vLLM recipe). The fixed latency is illustrative (6 ms per target forward, 0.2 ms per draft pass): measured batch-1 steps run several milliseconds, e.g. 7.3 ms for a DeepSeek-V4-Pro verify on eight B300s (LMSYS).
What is measured
Measured curves come from different setups, but their directions match. On dense models the gain falls with load. EAGLE-3’s paper reports 1.81× at batch 2 and 1.38× at batch 64 (LLaMA-3.1-8B, one H100, SGLang, a chain of three drafts). On the same coding task in DFlash’s paper (HumanEval, one B200, SGLang), a dense Qwen3-8B goes from 4.2× at one request to 2.4× at 32, while the MoE Qwen3-Coder-30B-A3B goes from 3.5× to 3.1×. Step-3.7-Flash’s NVFP4 checkpoint with three MTP drafts gains 1.13× over itself without MTP at 8 concurrent requests, 1.26× at 32 and 1.45× at 64 (vLLM on four GB200s, about 250-token prompts and 1.6K-token answers, StepFun’s card). Short contexts and an MoE with 288 experts fit the expert-read explanation, which is ours; StepFun gives no cause.
At high concurrency the loss shows up. vLLM’s recipe for Qwen3.8-2.4T, an MoE with 95B active parameters served over GB300 trays, says MTP with one draft “was measured at 64.8% acceptance and is not worth it: +3.4% at concurrency 1 and a loss above it (−9% at 128, −23% at 256), because the draft pass displaces real work once the batch is compute-bound”, while three drafts give about 2.3× per user at concurrency 1. That model is larger and its layout differs from the example above, so the arithmetic here does not reproduce the number; the direction is the same.
The layouts that reach the compute limit are also where engines support the least: SGLang’s DeepSeek-V4 presets with data-parallel attention run without DSpark (“incompatible with on current releases”), and vLLM’s per-batch-size draft lengths are “Not compatible with data parallelism”.
- dense model
- mixture-of-experts model
- 1× (no gain)
Speedup is throughput with speculation over the same engine without it, as each source prints it: (a) EAGLE-3 paper, Table 3; (b) DFlash paper, Table 3, the dense 8B on two tasks and the 30B MoE on the same coding task; (c) Step-3.7-Flash repository, aggregate decode tokens per second with and without MTP on the same NVFP4 checkpoint (1,309/1,155, 4,391/3,480, 8,229/5,667); (d) the vLLM recipe for Qwen3.8-2.4T-A95B ("+3.4% at concurrency 1", "−9% at 128, −23% at 256"). Each panel has its own axes; the setups differ in model, GPU, engine, drafter and task.
Memory, graphs and split serving
Drafting also takes memory, and memory caps the batch. Every drafted token needs a slot in the target’s KV cache until it is rejected, an EAGLE or MTP head keeps a cache of its own, and a hybrid model keeps intermediate states for each draft position. Kimi K3 is such a hybrid: its KDA layers (Kimi Delta Attention, a linear-attention variant) keep a fixed-size recurrent state per request instead of a growing cache, and the memory reserved for those states caps how many requests run. SGLang’s K3 cookbook lists the cap: 101 requests without speculation and 68 with DSpark when the experts are in MXFP4, 91 and 60 in NVFP4. LMSYS’s GLM-5.3-Flash post shows the other side: adding five MTP drafts cut the running-request limit from 96 to 56 on four H200s, yet output throughput rose from 3,882 to 5,625 tokens per second.
Engines replay decode steps as CUDA graphs (recorded kernel sequences, one per batch shape), and with speculation a shape is tokens for requests. Under vLLM’s default ceiling of 512 tokens (1,024 on data-centre Blackwell), seven drafts leave graphs for at most 64 requests (128); larger batches run without them unless the ceiling is raised, as vLLM’s DeepSeek-V4.1-Flash recipe does (“128 sequences with five drafts need 128 * (1 + 5) = 768 tokens”).
When prefill and decode run on separate GPUs (I4), the drafter runs on both sides: vLLM’s DeepSeek-V4-Pro recipe says to “Use the same K on both PD roles”, and its Kimi K3 recipe sizes the decode graphs as times the concurrency.
The drafters that ship
Lookup in earlier text
The cheapest drafter is no model at all. Prompt-lookup (“n-gram”) drafting copies the tokens that followed the last few generated tokens wherever they occurred in the prompt. SuffixDecoding (Oliaro et al., 2025) keeps suffix trees over the prompt and over earlier responses, and proposes a longer draft when the counts in the tree say a continuation is likely. A draft costs microseconds on the CPU, so in Leviathan’s formula, and the only cost is verification. Acceptance is high only where the output repeats text the drafter has seen: in the EAGLE-2 paper’s batch-1 table (Vicuna 7B, greedy), prompt lookup gives 2.53× on CNN/DailyMail summaries, which copy from the article, and 1.23× on Natural Questions. SuffixDecoding reports 1.8× to 4.5× for an OpenHands agent on SWE-Bench in vLLM at eight concurrent tasks.
Heads that read the target’s features
A trained drafter can read the target’s hidden states, which already encode much of what comes next. MTP modules do this and ship inside 21 of the 39 checkpoints in the map whose configs and tensors could be read (P1); each draft step is a pass through the module, so drafting time grows with .
EAGLE (Li et al., 2024) trains a separate head of one decoder layer on the target’s second-to-top features. EAGLE-3 (Li et al., 2025) makes two changes. It fuses low-, mid- and high-level features of the target (concatenated, then projected back to the model width), and it drops the loss that made the head predict the target’s next feature. Without that loss the head’s own outputs at step two look unlike anything it saw in training, so EAGLE-3 runs the head on its own outputs during training as well (“training-time test”, the remedy that scheduled sampling and DAgger give for the same mismatch). On LLaMA-3.1-8B they raise the accepted length on MT-Bench from EAGLE-2’s 4.05 to 6.13 (paper, Table 2, greedy).
Kimi K3’s report fine-tunes its own MTP layer, which has the same shape as an EAGLE-3 head, into an EAGLE-3 draft with the target frozen, initializing the fusion matrix as so that at the start the head sees exactly the high-level feature it was pre-trained on. The released K3 config has no MTP layer, and the serving engines use DSpark drafters trained by Inferact, Red Hat and RadixArk; the EAGLE-3 drafter for K3 that we found on Hugging Face is a third party’s and appears in no recipe.
Trees, and why production uses chains
A drafter can propose several candidates per position and verify a tree of them in one pass, with an attention mask that lets each node see only its ancestors (Medusa; SpecInfer; EAGLE). EAGLE-2 grows the tree from the drafter’s own confidence and keeps 60 nodes for an 8B target. The tree adds 0.6 to 0.8 accepted tokens over a chain in the original EAGLE paper, but at most one node per depth is accepted, so a tree of depth 6 with 60 nodes keeps at most 7 tokens of 61 verified positions; under load that is multiplied by . DFlash’s paper has the cleanest comparison (LLaMA-3.1-8B, GSM8K, one B200, SGLang): EAGLE-3 with a 60-token tree reaches 1.9× at one request and falls to 0.6× at 32, while the same head with a 10-token budget goes from 1.6× to 1.0×. SGLang’s overlapped scheduler supports only chains (--speculative-eagle-topk 1) and every shipped recipe for the models here uses one.
Drafting a whole block in one pass
A sequential head makes passes to draft tokens. A block drafter makes one. DFlash (Chen et al., 2026) is a small diffusion-style model: given the last accepted token and a block of mask positions, it fills the whole block in a single forward pass. Its five layers read the target’s hidden states from five layers spread through the model, fused and injected as extra keys and values into every draft layer, so depth does not dilute them. The paper defaults to a block of 16 (the accepted token plus 15 drafts); shipped DFlash drafters for Qwen3.5-397B, Kimi K3 and Laguna-S-2.1 have six layers. Drafting cost no longer grows with , so a long block costs little more to draft than a short one: on Qwen3-8B the paper’s average AL is 6.49 greedy and 5.48 at temperature 1, against 3.40 and 3.26 for EAGLE-3 with a 60-node tree (Transformers, one H200, seven tasks).
A parallel drafter learns each position’s distribution separately, so it can propose tokens that are each plausible and together incoherent (“of problem” where “of course” was meant). This is the multimodality problem of non-autoregressive translation (Gu et al., 2018), and its fixes have the classical shape of a chain model. DSpark (Cheng et al., 2026, from Peking University and DeepSeek) keeps the parallel body and adds a Markov head: the previous drafted token’s row of a rank-256 matrix product gives a vocabulary-sized bias that is added to position ‘s logits, and the block is sampled left to right with that bias. Each position’s is still a proper distribution given the tokens before it, which is all the acceptance test needs. On Qwen3-8B at temperature 1 this lifts AL over DFlash by 18% on average over nine benchmarks, 14% to 23% each (paper, Table 1). DSpark also adds a confidence head, trained to predict each position’s acceptance from the drafter’s own state and then calibrated per position. NVIDIA’s LiLiCorr, shipped as a drafter for Nemotron 3.5 Super, keeps the eight most likely tokens at each of seven positions, scores the whole lattice with a two-layer transformer that gives a compatibility score between neighbouring candidates, and walks it left to right, greedily or by sampling at the request’s temperature. The walk is sequential over 8 candidates rather than the vocabulary.
Block drafters ship widely. DeepSeek-V4.1-Flash’s drafter is three transformer blocks with a 128-token sliding window that draft five positions in parallel, with a Markov head and a confidence head (report §2.4.3); DeepSeek dropped the MTP module from pre-training to use it. Mistral Large 4 ships a DSpark head of three dense layers with a block of three. MiMo-V2.6 ships a five-layer DFlash-style drafter that predicts seven tokens. MiniMax-M3.1-Flash ships a DSpark head that “replaces the earlier EAGLE-like MTP approach”. Of the 12 labs whose own documents describe a drafter, six describe block drafters.
- last committed token
- draft token
- one draft-model pass
- the target's hidden states
- positions the target verifies in one pass
Seven drafted tokens; the target verifies them together with the last committed token, whose position yields the bonus token, so a chain or block verifies eight positions per sequence. The tree is drawn as a schematic; EAGLE-2's default for 7B and 8B models has 60 nodes at depth 6 (its Appendix A), so the target verifies 61 positions to accept at most 7. DFlash feeds the target's features into every draft layer as keys and values; DSpark builds on a DFlash backbone and corrects it with a low-rank Markov bias from the previous draft token, plus a confidence head that predicts which drafts will survive; LiLiCorr keeps the 8 likeliest tokens at each slot and walks that lattice left to right, scored by a small head run once (block 8 in Nemotron 3.5 Super).
Choosing how much to verify
A fixed draft length suits one load. vLLM’s adaptive-verification page puts it in two sentences: “At batch size 1 that is a good trade: the GPU is memory-bound with spare compute, so the extra draft tokens are close to free. At batch size 256 it is a much more delicate one.” Engines size the check with a table by load, with a running average, or per request from predicted acceptance.
The table is the simplest. vLLM’s num_speculative_tokens_per_batch_size maps batch-size ranges to a draft length; its DeepSeek-V4.1-Flash command for one GB300 workstation drafts five tokens up to four requests and one from 5 to 24. Shipped recipes do the same by hand: vLLM’s Kimi K3 recipe on MI355X drafts seven at concurrency 8 and four at 24 to 48, and SGLang’s DeepSeek-V4-Pro-0813 table on MI355X uses a DSpark block of six at concurrency 16 and three from 48 up. SGLang’s adaptive mode moves the number of steps with a running average of the accepted length per batch size.
DSpark’s scheduler decides per request and per position, every step. Its confidence head predicts , the probability that position of sequence survives given that the positions before it did, so the probability that the position is accepted at all is the running product . If sequence verifies drafts, the step carries tokens and is expected to emit . The engine measures once, at start-up, how many steps per second it runs at each step size, , and the scheduler admits positions in order of while the expected throughput still rises (paper §3.2.2). A confident sequence’s fifth position can win over a doubtful one’s first.
Two properties make this more than a heuristic. Since each , never rises along a sequence, so the best positions by are always a prefix of each sequence’s block, and for a fixed step size taking them in that order maximizes ; if the step time is convex in , has a single peak, and stopping at the first fall finds it. And the output stays the target’s because whether position is verified depends only on the drafts up to , never on drafts after it: “admission decisions must not depend on future candidate tokens”. In production DeepSeek runs a lagged form, setting the step’s capacity from the confidence predictions of two steps earlier (paper §5.2), with a throughput table that ignores context length. vLLM’s adaptive verification applies the same ranking to other drafters, with a small logistic model on the draft logits, recalibrated during serving, in place of a trained confidence head.
Toy, numbers invented. Other load: 128 sequences verifying one token each.
Scheduled: N = 147 tokens per step, τ = 145.30 expected tokens, Θ = 8,171 tokens/s. Windows ℓ per sequence: 5, 3, 2, 1, 0, 0, 0, 0.
Static K = 7: N = 192 tokens per step, τ = 160.89, Θ = 7,313 tokens/s.
A toy with invented numbers, after the DSpark paper's scheduler (§3.2.2). Each of eight sequences has a confidence-head estimate κr,j that its draft at position j survives if every earlier one did (constant per sequence here), so position j survives with ar,j = κr,1 ⋯ κr,j. Every sequence verifies its own next token (“own”). A step carries N = Σr (1 + ℓr) tokens over every sequence in the batch, the other load included (ℓ = 0 for those), and τ = Σr (1 + ar,1 + … + ar,ℓr) tokens are expected back. The scheduler admits positions in order of ar,j and stops when the expected throughput Θ = τ · SPS(N) stops rising. The engine profile is invented: a step takes t(N) = 4 ms + max(6 ms, 6 ms × N / 64), flat up to 64 tokens and growing beyond.
The DSpark paper reports DeepSeek’s switch on live traffic. Before it, DeepSeek served its V4 previews with one MTP draft, because “deploying a static multi-token drafter (e.g., MTP-3/5) strictly degrades aggregate throughput under high concurrency due to excessive verification overhead.” With DSpark drafting up to five, V4-Flash’s throughput at a per-user speed of 80 tokens per second rose 51%, and at matched throughput per-user speed rose 60% to 85% (V4-Pro: 52% at 35 tokens per second per user, and 57% to 78%). DSpark verified roughly four to six tokens per sequence, against MTP-1’s static two, and the average “decreases smoothly with load”. In SGLang the window follows difficulty: 5.24 tokens on GSM8K, 3.78 on Arena-Hard and 2.91 on poetry (LMSYS, block of six). On Kimi K3 the same trimming breaks even with verifying everything up to batch 8, and at batch 256 it is 68% (chat) and 24% (math) ahead.
Training a drafter for the traffic it will see
The drafter is trained against a frozen target in every account here except MTP during pre-training and DeepSeek’s DSpark during post-training, which trains alongside the backbone without passing gradient into it.
The loss decides where a small drafter ends up. Cross-entropy on text the target generated is the forward KL divergence to the target up to a constant, and both it and the acceptance rate are best at ; a drafter too small to reach settles at different points under each. The LK loss (Samarin et al., 2026) trains on , averaged over contexts. Its gradient in each context is the total-variation gradient scaled by , so contexts where acceptance is low weigh more. Kimi K3 trains its EAGLE-3 draft on it at temperature 1 with no cross-entropy term, and Nemotron 3.5 uses an LK loss for MTP boosting, where Nemotron 3 Ultra had used KL. DSpark trains on the total-variation term itself, which targets mean acceptance, with a small cross-entropy weight; DFlash, DSpark and LiLiCorr also weight later positions down, since they count only if earlier ones survive.
Drafters are trained on what the target generates, not on human-written answers. EAGLE-3 regenerates its training responses with the target, DFlash and DSpark do the same, and LiLiCorr’s second stage regenerates 18.2M samples with the released post-RL Nemotron 3.5 Super “so that the drafter adapts to the hidden states and output distribution of the model it serves”. MTP boosting in Nemotron 3.5 trains only the MTP head, after all backbone post-training, on rollouts sampled at temperature 1. At draft length 7 and temperature 1 the final Super’s native head gets 3.787; starting from a head trained at the SFT stage gives 4.436, and 25k more steps against the final backbone give 4.612 (report, Table 8). DeepSeek keeps training DSpark through V4.1’s post-training to keep it “aligned with the evolving policy”. MiMo-V2.6 fine-tunes its DFlash drafter on early RL rollout logs resampled to the RL training distribution, and in RL its accepted length is 31.3% above the MTP-3 configuration it replaced. No lab reports training a drafter on logged production traffic.
Long prompts need long training windows: DFlash’s Qwen3.5-27B drafter trained at 4K gets AL 2.09 on 32K-token reports and 3.56 after fine-tuning on long contexts (paper, Table 4).
A drafter’s weights are read on every draft pass, so at low load quantizing them makes drafting faster: LiLiCorr with NVFP4 weights runs 11% faster on a DGX Spark, and quantizing its LM head 5% more (Nemotron 3.5, §4.8). Kimi K3 fine-tunes its draft in the target’s quantization-aware format (MXFP4 expert weights). MiMo-V2.6 runs its draft compute in FP8. Others keep the drafter wide beside a narrow target: Mistral’s DSpark layers stay BF16 while the routed experts go to NVFP4. Quantizing a checkpoint can also move the acceptance of the MTP head inside it: vLLM measured Qwen3.8-27B’s at 0.771 in FP8 and 0.788 to 0.897 in two NVFP4 builds.
Rolling back in hybrid and sparse models
When the target rejects drafts, an attention layer drops their cache entries. A recurrent layer (Mamba, Gated DeltaNet, KDA) cannot: its state is overwritten every token, and by the end of verification it has advanced past the last accepted one (T2). The simple fix is a snapshot after each draft position. Nemotron 3 Ultra does this. For Kimi K3, LMSYS counts the cost: 64 KB per request, layer and head, times eight positions, 512 KB, across 69 KDA layers, reserved per running request and competing with the state pool that caps concurrency.
The fix that shipped stores inputs instead of states. The state after any accepted prefix follows from the state before the step and the projected inputs of the accepted tokens, which are far smaller. Moonshot’s K3 report keeps only those, rebuilds the accepted states on-chip and writes back the final one, in one fused kernel (projected-input caching). The same design, called ReplaySSM, was developed concurrently in the Dao lab; SGLang’s version cuts K3’s draft window from 512 KB to 16 KB. NVIDIA’s version for Mamba-2 in Nemotron 3.5 keeps a checkpointed state and a buffer of the inputs since, so a rejection is a decremented counter, and a step writes the state about once per buffer length instead of once per verified token. On one GB200 in vLLM it raised output throughput 4.7% to 6.4% with speculation on (report, Table 24).
Prefix caching adds a second case: after a hit, the drafter needs the target’s hidden state at the last cached token, which vLLM recovers by replaying from the nearest recurrent checkpoint, up to a whole Mamba cache block earlier. Nemotron 3.5 keeps an extra checkpoint near that boundary, so with a 16-token prefix-match unit a hit replays 16 tokens.
Sparse attention has its own shortcut. Within one draft iteration the query moves by a few positions, so the indexer’s top-k selection barely changes. GLM-5.3’s config sets index_share_for_mtp_iteration: true, and Qwen3.8-Flash-Next’s MTP reuses the selection across steps; in SGLang that cuts the draft’s indexer work per iteration “from N invocations to one”. Qwen’s report finds no change in mean accepted length (4.06 with full attention in the MTP module, 4.07 with its sparse attention and reuse), though that pair does not separate the reuse from the sparse attention.
Measuring a drafter
All five labs that report a speedup for their own drafter state a batch size for at least one number, four of them at more than one user (Xiaomi 32 to 128, StepFun 8 to 64, Mistral 8, Google 4 to 8), and none states a temperature. Acceptance depends on the text, and engine notes warn about prompts. vLLM’s GLM-5.3 recipe: “pure throughput benchmarks tend to under-report real speed, because MTP’s acceptance rate is usually low in synthetic runs.” Its DeepSeek-V4-Pro recipe: benchmarking with raw completion prompts “silently depresses the acceptance rate”, since MTP was trained on chat-formatted input.
Some published numbers do not measure acceptance at all. SGLang’s Kimi K3 benchmark cells set SGLANG_SIMULATE_ACC_LEN=4.5, which pins the accepted length, and its DeepSeek-V4-Pro-0813 low-latency cells pin 4.00 where the shipped recipe earns 4.678. vLLM’s agentic DeepSeek-V4-Pro lane on MI355X uses a synthetic accepted length of 3.77 for throughput. Pinned numbers isolate the cost side and say nothing about the drafter. A drafter can also fail silently: SGLang warns that serving a DSpark checkpoint with EAGLE “starts and serves without any error, but the draft head it binds accepts nothing”. vLLM’s Qwen3.8 recipe measures acceptance from the engine’s counters “since throughput alone cannot distinguish a working drafter from one that loaded and was ignored.”
A useful report gives the accepted length on the traffic served, the temperature and sampling mode, the concurrency and the layout, against the same deployment without speculation.
The designs side by side
| Design | Draft passes for tokens | Verified per sequence | Accepted length reported (setup) | What grows with load | Shipped by (labs’ own documents) |
|---|---|---|---|---|---|
| n-gram, suffix | 0 (CPU lookup) | ; suffix adapts | 1.4 to 2.7 for prompt lookup (Vicuna 7B, greedy) | verification only | engines only |
| MTP module | 3.8 to 4.6 at 7 drafts (Nemotron 3.5 Super, temp 1) | verification, draft cache | DeepSeek V4, NVIDIA, Z.ai, Xiaomi, Qwen, StepFun, Tencent | ||
| EAGLE-3 head | 6.13 (LLaMA-3.1-8B, MT-Bench, greedy, tree) | verification, draft cache | Moonshot (report) | ||
| Tree (EAGLE-2 style) | depth | up to ~61 | at most depth + 1 | verification × nodes | engines; not in SGLang’s overlap path |
| DFlash | 1 | block (8 or 16) | 6.49 / 5.48 (Qwen3-8B, greedy / temp 1) | verification | Xiaomi, NVIDIA, Poolside |
| DSpark | 1 + light sequential head | per sequence, by confidence | 3.2 to 6.2 (Qwen3-8B, temp 1, nine benchmarks) | verification, trimmed by the scheduler | DeepSeek, Mistral, MiniMax, NVIDIA |
| LiLiCorr | 1 + lattice walk | block | 4.77 / 4.37 (Nemotron 3.5 Super, 64 concurrent) | verification | NVIDIA |
Where the models in the map sit
The table gives the drafter each model is served with and the draft length , from the vLLM recipes of 2026-11-11 unless noted. Checkpoints state block sizes in two conventions: DFlash and LiLiCorr count the accepted token (a block of 8 is 7 drafts), DSpark checkpoints count drafted tokens (Red Hat’s block of 8 is 8 drafts, DeepSeek’s dspark_block_size: 5 is 5), so read the recipe’s num_speculative_tokens. The multi-token-prediction code check marks 18 of the 25 models in this table’s scope as present, 3 unclear and 4 absent. Three of the absent are models served without an MTP module (Kimi K3, whose config sets num_nextn_predict_layers: 0; Mistral Large 4; Laguna); the fourth, Nemotron 3.5 Lightning, has one, and the checker misread it.
| Model | Served drafter | Note | |
|---|---|---|---|
| GLM-5.3 | built-in MTP, one layer | 5 | SGLang balanced preset: 1 |
| DeepSeek-V4-Pro-0813 | bundled DSpark | 7 | head trained for 5; SGLang on MI355X: 6 at c16, 3 at c48+ |
| DeepSeek-V4.1-Flash | bundled DSpark (3 blocks) | 5 | adaptive verification on some NVIDIA targets; no MTP |
| Mistral Large 4 | bundled DSpark (3 layers, BF16) | 3 | fixed by the checkpoint |
| Kimi K3 | DSpark from Red Hat (5 layers, 4.7B) | 8 | SGLang: RadixArk’s, 7 |
| Qwen3.5-397B | built-in MTP | 3 | earlier guide: 1 |
| Qwen3.8-Flash-Next | built-in MTP, sparse index reused | 3 | |
| MiMo-V2.6 | in-checkpoint DFlash (5 layers, block 8) | 7 | block 6 in RL |
| Laguna-S-2.1 | DFlash (6 layers, block 16) | 15 | card and llama.cpp: 7 |
| Nemotron 3 Ultra | built-in MTP | 5 | |
| Nemotron 3.5 Super | built-in MTP | 3 or 5 | LiLiCorr, 7, on DGX Spark |
| Nemotron 3.5 Lightning | DSpark, NVFP4, no confidence head | 3 | none for maximum throughput on one H100 |
| MiniMax-M3 | EAGLE-3 head from Inferact | 3 | a GQA variant has a 16× smaller draft cache |
| MiniMax-M3.1-Flash | in-checkpoint DSpark | 7 | 3 on MI355X |
| Step-3.7-Flash | built-in MTP, three modules | 3 |
Kimi K3 and MiniMax-M3 are served by default with drafters trained outside the lab.
Easy to get wrong
- Acceptance length usually counts the target’s own token. “AL 2.7” at seven drafts means 1.7 accepted drafts per step. Comparing a paper that counts drafts only with one that counts the extra token is off by one.
- A recipe’s
num_speculative_tokenscan exceed what the head was trained for. V4-Pro-0813’s DSpark head was trained for five positions and vLLM runs seven; vLLM’s blog finds the seventh survives less than 10% of the time. - A speedup at batch 1 says little about a server. Kimi K3’s 3.7× is a batch-1 number; at 256 sequences LMSYS compares trimmed against full verification, not against plain decoding.
- A speedup that rises with concurrency (Step-3.7) does not contradict one that falls. In an MoE at small batch the drafts’ extra experts are a cost (our reading), and it fades as the batch grows; the fall comes once the step turns compute-bound or memory caps the batch.
Check yourself
A drafter proposes seven tokens and the reported AL (the target’s token included) is 3.1. Under constant acceptance, roughly what is the per-draft acceptance rate, and how many tokens would three drafts give?
Solve : (AL is 3.14 at 0.70). At three drafts, . Four extra verified positions per sequence buy about 0.6 tokens per step, and if requests differ in difficulty a common cut-off loses more than that.
Using the GLM-5.3 numbers from “A model to reason with”, estimate the bytes per GPU of a plain step and of a six-position verify step at batch 1, ignoring the cache. Why is the ratio far smaller at batch 64?
Plain: GB. Verify: the six positions touch about 44 experts per layer, so GB, about three times as much. At batch 64 the plain step already touches about 222 of 256 experts per layer and the verify step 256, so the expert bytes differ by about 15%. (Independent uniform routing assumed.)
Sequence A’s confidence head predicts 0.95 at every position, sequence B’s 0.5. Which does a DSpark-style scheduler admit first: A’s fifth position or B’s first?
A’s fifth: its survival probability is , against for B’s first. Admission follows survival, not position, so a confident sequence keeps a long window while a doubtful one may verify only the token the target must emit.
In the map
Methods
- Speculative decoding 15 sources
- Speculative sampling 3 sources
- EAGLE 10 sources
- Multi-layer EAGLE 5 sources
- DFlash 6 sources
- DSpark 14 sources
- DSpark draft head 1 source
- LiLiCorr 2 sources
- Standalone parallel speculative drafters 2 sources
- Concurrency-aware draft-length tuning 1 source
- FP8 low-precision speculative draft computation 1 source
- MTP Boosting 6 sources
- LK loss 1 source
- Distribution-matched speculative-draft fine-tuning 1 source
- Throughput-based draft-block sizing 1 source
- Mamba checkpointing for speculative-decoding replay 2 sources
- Projected-input caching for speculative KDA rollback 1 source
- Reuse QSA index selection across speculative decoding steps 2 sources
Taxonomy nodes
- decoding strategy 52 methods
Further reading
-
Leviathan, Kalman and Matias, “Fast Inference from Transformers via Speculative Decoding” (ICML 2023): the acceptance rule, , and the walltime model with its optimal .
-
Li et al., EAGLE, EAGLE-2 and EAGLE-3 (arXiv 2401.15077, 2406.16858, 2503.01840): feature-level drafting, confidence-built trees, training-time test, and SGLang batch results.
-
Sadhukhan et al., “MagicDec” (arXiv 2408.11049): why long contexts keep verification cheap at large batch, given a drafter with a small cache.
-
Chen et al., “DFlash” (arXiv 2602.06036): block-diffusion drafting, with SGLang and vLLM tables by concurrency.
-
Cheng et al., “DSpark” (arXiv 2607.05147): the Markov and confidence heads, the throughput-maximizing scheduler, and DeepSeek’s live-traffic switch from MTP-1.
-
Rusanovsky et al., LiLiCorr (arXiv 2608.20530), and Nemotron 3.5 report §5: lattice scoring, drafter quantization, ReplaySSM.
-
Samarin et al., “LK Losses” (arXiv 2602.23881): training a drafter on the acceptance rate directly.
-
Oliaro et al., “SuffixDecoding” (arXiv 2411.04975): model-free drafting for agents.
-
vLLM speculative-decoding docs (dynamic speculative decoding, adaptive verification) and SGLang’s speculative-decoding docs: the flags, and what each engine supports.
