Requests in SemiAnalysis’s AgentX benchmark, which replays 393 real coding-agent sessions, carry a median of 142,000 tokens of context and get back a median of 444 tokens. On OpenRouter in late October 2026, providers selling DeepSeek-V4.1-Flash collected about $0.013 per million input tokens and $0.57 per million output tokens, according to Yingjun Wu’s analysis. Wu asked whether renting GPUs to serve that model is profitable. By his numbers, an 8×H200 node never breaks even, even when busy all the time; an 8×B300 node breaks even once it is busy about a third of the time.
With the model and price fixed, the difference is how many tokens per second each GPU produces while still giving every user the promised speed. That depends on the GPU and on a stack of choices: the number format of the weights and of the KV cache, how the model is split across GPUs, how prefill and decode share the hardware, whether earlier prompts are kept, and whether the model drafts several tokens per pass. vLLM and SGLang are the two main open-source serving engines, and a command line for either, for a large mixture-of-experts (MoE) model, is that stack written as flags.
This primer turns the cost model of one forward pass from 00 into a price per token, then gives each lever a short card: its cost, its gain, where the gain ends, and one measured number. The mechanisms are in the inference series, which each card names.
From a forward pass to a price
Recall the two phases from 00. Prefill processes the prompt in one pass and is compute-bound: its cost is FLOPs. Decode produces one token per pass for each sequence and is memory-bound: each step reads the weights it uses plus every sequence’s KV cache, and its cost is bytes. If sequences decode together, one read of the weights serves all , so the tokens per step grow with while the weight bytes stay fixed. That is why batching is the main lever. Its gain ends when the KV bytes, which grow with and with context length, take over the step, or when the caches no longer fit in memory. In an MoE model a larger batch also reads more experts, until every expert is read: for GLM-5.3, with 8 of 256 experts per token, from a batch of about 128 if routing were uniform (C2). The expert layers would turn compute-bound only near 9,000 tokens per step (computed with 00’s H100 ridge), so at decode memory or the latency promise stops the batch first.
A serving system is described by three numbers. Time to first token (TTFT) is how long a user waits before the answer starts; it is mostly prefill and queueing. Time per output token (TPOT, also called inter-token latency, ITL) is the gap between tokens after that; its inverse is the per-user speed, in tokens per second per user. Benchmarks often report a percentile: P90 is the value that 90% of requests meet or beat. Throughput is the total number of tokens the hardware produces, usually given per GPU. The bill depends on throughput, the user’s experience on the other two:
Throughput and per-user speed pull against each other: a larger batch gives more tokens per GPU and slower tokens for each user. Fix the model, its number format, the GPU type, the engine and the workload; the best achievable pairs over all batch sizes and layouts form the throughput–interactivity curve, a Pareto frontier: no point on it improves one number without giving up the other. TTFT is a separate promise that the curve doesn’t show.
Before reading one, check which tokens it counts. Agentic benchmarks such as AgentX count every token the GPU processes, including input tokens read from cache, and about 95% of the input is cached on that traffic (Wu, citing AgentX’s methodology). Output is under 1% of the counted tokens, so a GPU that “produces” 300,000 tokens per second here produces about 2,000 output tokens. Figure 1 shows a curve measured this way.
TP4 on 4 GPUsTP2 on 2 GPUsTP2 + EP2, DP attention, 2 GPUsa per-user promise: 100 or 200 tok/s per user
DeepSeek-V4.1-Flash, B300, SGLang, InferenceX run 2026-10-29. Labels give concurrency, the sessions per instance. Hover or focus a point for its layout and output tokens.
Every serving benchmark in the reports is one point on such a curve, and a throughput without its per-user speed can’t be compared with another. The Nemotron 3.5 report gives “up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user”. Its tables show how that headline was picked: the report calls a point usable if each user gets at least 50 tokens per second, then takes the usable point with the highest throughput. The 4,856 point runs at 89.9 tokens per second per user (the report’s Table 24). A point at 5,728 output tokens per second per GPU, 18% more, ran at 49.1 per user, just under the floor, and was left out.
Three workloads, three points on the curve
Interactive chat and reasoning are dominated by decode: with a long chain of thought, the answer is much longer than the question (Nemotron 3 Ultra’s headline comparison uses 8K tokens in and 64K out). Per-user speed is the promise.
Agentic coding is dominated by a long, reused context: each turn adds a tool result of a few thousand tokens to a context of 100K or more and asks for a short answer. If the engine still holds the earlier turns’ cache, a turn costs a small prefill; if not, a prefill of the whole context. TTFT matters as much as per-user speed.
Offline generation and reinforcement-learning (RL) rollouts have no user waiting on a token, so the target is throughput, except that an RL step waits for its slowest sample. The GLM-5 report: “For RL rollouts, the optimization target is not aggregate throughput but end-to-end latency, dominated by the slowest (long-tail) sample in each step” (§3.6.2). The long tail runs at a small batch, where per-user speed counts again.
Utilization
Wu defines utilization as “tokens you actually sold ÷ tokens the benchmark says you could sell. Not GPU busy time.” He reports that a new OpenRouter provider without direct customers “lands at 10–30%”, and that DeepInfra puts typical self-hosted inference at 10–20%. We could check neither, and no lab report gives a number. Even with steady demand, a server can’t sell its benchmark throughput. A benchmark keeps a fixed number of requests in flight, while real requests arrive at random, and as load nears capacity the queue, and with it TTFT, grows steeply, without limit as load reaches capacity (a standard queueing result for random arrivals), so a server keeps its TTFT promise only well below its benchmark throughput. Demand also varies over the day.
Selling tokens and serving yourself
Two 2026 deployments of models in the map show the formula from both sides. Some hardware names first. NVIDIA’s Hopper generation (H100, H200) multiplies in FP8 at the lowest; the Blackwell generation (B200, B300, and GB200 and GB300, which pair them with Grace CPUs in racks of up to 72 GPUs called NVL72) also multiplies in FP4. Inside a rack, or an 8-GPU server, the GPUs share NVLink, a fast interconnect; between servers they use a network about an order of magnitude slower.
Selling tokens: DeepSeek-V4.1-Flash at market prices
Wu’s analysis (“Is hosting an open-source inference engine profitable?”, 2026-11-02) fixes one model (DeepSeek-V4.1-Flash), one workload (AgentX), one promise (100 output tokens per second per user) and one price, and varies the hardware. The price is what OpenRouter providers collected: $0.013 per million input tokens, blended over cached and uncached, and $0.57 per million output tokens. At a 99-to-1 mix of input to output tokens that is $0.0186 per million tokens of either kind; Wu uses $0.0188 throughout, and so do we. Rent is the lowest public on-demand price: $3.99 per GPU-hour for H200 and $7.40 for B300; a GB300 NVL72 rack has no public price, so Wu takes $15.21 per GPU from a price index. Throughput is the best public open-source result for each card on InferenceX.
Revenue per GPU-hour at full load is tokens/s × 3,600 × price, so the break-even utilization is rent ÷ (tokens/s × 3,600 × price ÷ 10⁶), with the price per million tokens. Wu’s results, recomputed from his inputs:
| Hardware and software | All tokens/s per GPU, incl. cached input | Break-even utilization |
|---|---|---|
| 8×H200 | about 24,000 (implied) | 244%: never |
| 8×B300 | about 316,000 (implied) | 35% |
| GB300 NVL72, standard configuration | about 267,000 (implied) | 84% |
| GB300 NVL72, NVIDIA Dynamo | about 469,000 (implied) | 48% |
| GB300 NVL72, vLLM with prefill/decode split and wide expert parallelism | 954,000 (measured by vLLM) | 24% |
Wu’s table gives rent per million tokens, from which the throughputs follow; the last row is the vLLM team’s own measurement. The three GB300 rows are one rack with three software configurations: “A 3.5x spread from configuration alone.” That row was measured at 196 tokens per second per user, not 100, with a P90 TTFT of 8 seconds, on 64 of the rack’s GPUs joined by NVLink; its wide expert parallelism would cost far more across a network. Wu’s 99-to-1 mix also overstates the output share: in the vLLM run’s AgentX traffic, output was 0.72% of all tokens, which puts the market price at $0.0170 per million and every break-even about a tenth higher, 35% becoming 38% (computed).
Wu attributes the H200’s loss to its missing FP4 units: “Hopper has no FP4 compute units, so a model built around FP4 runs at 1/13 the throughput of a B300”. That can be only part of it. Decode is bound by bytes, and 4-bit weights save an H200 the same bytes whatever its arithmetic; the quantization card below puts the gain from native FP4 arithmetic at about 1.3×. The rest is memory and software: an H200 has about half a B300’s memory, so a smaller batch fits at the same promise, and about 60% of its bandwidth, and kernels for Hopper now lag (FlashMLA dropped Hopper in September 2026). No source separates these.
Wu also varies the business inputs on the B300 node. Promising 200 tokens per second per user instead of 100 moves break-even from 35% to 93%; promising 50, to about 20%. His prices and rents each span a factor of two and move it between 26% and 52%. Break-even is proportional to rent and inversely proportional to throughput and price; the promise moves it furthest because going from 50 to 200 tokens per second per user cuts throughput 4.6-fold. The promise values reach us through a secondary write-up of an image in Wu’s article, and InferenceX’s 2026-10-29 B300 run in Figure 1 puts the 200 point nearer 83% (computed). Figure 2 lets you move all three inputs over Wu’s ranges.
10–30%: new provider, Wu's estimate
50 tok/s per user: 20.0% · 100 tok/s per user: 34.6% · 200 tok/s per user: 93.0%
u = rent / (tok/s × 3,600 × price / 10^6), with rent $7.40 per GPU-hour and price $0.0188 per M tokens
The sliders span Wu's ranges: price from DeepSeek's off-peak to its list price, rent from reserved to a median index. The 0.72% preset prices Wu's $0.013 in / $0.57 out at the output share of the vLLM run's traffic.
Serving yourself: Revolut and GLM
Revolut’s David Abitbol described the other side at AI Engineer NYC (2026-10-13). Revolut serves coding agents for about 2,000 engineers with GLM-5.1 and then GLM-5.3, on 56 B200 GPUs (“we roughly need 56 B200”). The blended cost is “30 cents per million”, and the invoice is “one-tenth of what we were spending on Anthropic and OpenAI”. The targets are empirical: “sub-3 second time to first token”, because “as soon as you exceed 5 seconds, my Slack inbox starts to blow up”, and about 200 tokens per second per user, at which “people are fine”. Between the migration and the stable setup, on the same 56 GPUs, requests per second grew 76% while TTFT fell from 14.26 s to 1.68 s.
He named four of this primer’s levers: FP4 weights on Blackwell; TP8 rather than TP4, because “even though GLM fits on TP4, you don’t have enough KV cache”; a cache hit rate “above 90%” for coding traffic; and a context cap of 256K tokens out of a possible 1M: “long context is a scam … 256 is really what you need.” He listed speculative decoding and the prefill/decode split as options to look into, without saying whether Revolut runs them. The conference’s own summary says these numbers “describe Revolut’s deployment and utilization, not a portable price”.
A provider controls neither its price, which a routing algorithm sets, nor its utilization. An internal fleet has captive utilization through the working day; its levers are the latency it promises, the context it reserves memory for, and the cache hit rate.
Fitting the model on one node
Before any lever, the model has to fit. On each GPU, memory holds its share of the weights, the KV cache of every sequence it serves, and working space for activations, CUDA graphs (recorded sequences of GPU kernels replayed with one call; see “Kernels”) and the sampler. NVIDIA lists 141 GB per H200, 1,440 GB for an 8-GPU DGX B200 (180 GB per GPU) and 2.1 TB for a DGX B300. These are binary units: an H200 reports 143,771 MiB (151 GB), a B200 183,359 MiB (192 GB), and 2.1 TiB ÷ 8 is about 288 GB per B300. We compute with those byte counts, which are what vLLM’s memory fraction applies to.
Take GLM-5.3, a 753B-parameter MoE with multi-head latent attention (MLA) and a sparse indexer (T1, T3). MLA caches one compressed vector per token and layer, plus a short part that carries the position; the indexer is a small scorer that picks which cached tokens each query reads. The FP8 release is 756 GB of weights. vLLM publishes a recipe per model, a set of tested command lines per GPU type; GLM-5.3’s says that “the FP8 checkpoint fits on a single 8xH200” node and, “with FP8 KV cache”, “reaches the full 1M-token context on 8xB200”. Its H200 command, with speculative decoding selected, reads:
vllm serve zai-org/GLM-5.3 --tensor-parallel-size 8 --kv-cache-dtype fp8 \
--max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' ...
(the rest sets the tokenizer and parsers). Here we need the first three flags.
The KV cache per token follows from the config. Each of the 78 layers caches the latent,
kv_lora_rank: 512 plus qk_rope_head_dim: 64 values, 1,152 bytes in BF16. vLLM’s FP8 format
stores the 512 latent values in FP8 with 16 bytes of scales and keeps the 64 position values in BF16:
656 bytes. The 21 layers with an indexer also cache its keys (index_head_dim: 128), 132 bytes in
FP8 whatever the main format is. The multi-token-prediction layer used for drafting has a cache of its
own, with an indexer. In total, computed: (78 + 1) × 1,152 + (21 + 1) × 132 = 93,912 bytes per token
in BF16, and (78 + 1) × 656 + 22 × 132 = 54,728 in FP8.
| KV format | Bytes per token | One 128K sequence | One 1M sequence |
|---|---|---|---|
| BF16 | 93,912 | 12.3 GB | 98.5 GB |
| FP8 | 54,728 | 7.2 GB | 57.4 GB |
Tensor parallelism (TP8, the first flag) decides where the cache lives. It splits each weight matrix across the 8 GPUs, and for attention it gives each GPU a share of the heads (00). The latent cache has one head, so every GPU holds the full cache of every sequence. The weights take 94.5 GB per GPU (756 GB ÷ 8). More GPUs shrink that share and leave more room for cache on each, which is why Revolut’s Abitbol advises TP8 over TP4, but they don’t shrink one sequence’s cache.
So on 8×H200, with vLLM’s default of using 92% of memory and an assumed 10 GB per GPU of working space,
about 34 GB per GPU is left for cache: four 128K sequences in FP8, and about 60% of one 1M
sequence (computed). The recipe’s H200 command caps context at 256K, which fits this arithmetic. On
8×B200 the same calculation leaves 72 GB, about one 1M sequence: “reaches the full 1M-token context”
means one user at a time. The recipe adds that “whether the full window fits is a
KV-cache VRAM question, so the lever is --max-num-seqs”, the cap on how many sequences share the
pool. Figure 3 lets you vary formats, context and layout.
weights, 1/8 per GPU10 GB working reserve (our assumption)KV cache, one cell per sequenceusable: 92% of memory (vLLM default)does not fit
Weights 94.5 GB per GPU (FP8) · KV 54,728 bytes per token, 7.17 GB per 128K sequence per GPU (FP8, copied to all 8 GPUs)
Most 128K sequences per node: H200 4 · B200 10 · B300 22
The weight format decides how many GPUs the model needs: in BF16, GLM-5.3 doesn’t fit on one H200 or B200 node. The KV format, with the context the deployment reserves, decides how many users share each GPU.
Lever cards
Splitting the model across GPUs
Tensor parallelism shards each layer’s weight matrices, so all GPUs work on every token. After the attention block and after the feed-forward block of each layer, the GPUs combine their partial results with an all-reduce, in which each GPU sends its part and receives the sum. Expert parallelism (EP) places whole experts on different GPUs and sends each token to its experts’ GPUs, an all-to-all exchange. Attention data parallelism (DP attention) has each GPU run attention for its own requests with its own cache, while EP splits the expert layers across all of them. vLLM recipes call TP plus EP “TEP” and DP attention plus EP “DEP”.
All three layouts add communication. An all-reduce per layer is cheap inside one NVLink domain (8 GPUs in a server, up to 72 in an NVL72 rack) and expensive outside it, so TP rarely leaves the domain; EP’s all-to-all moves every token to its experts and back. DP attention also copies the attention weights to every GPU: for Kimi K3, “around 61 GB for KDA plus 11 GB for MLA” per GPU (LMSYS; KDA is K3’s linear-attention layer).
In return they give memory and speed. TP spreads the weights, so each GPU reads fewer weight bytes per
step and one user’s tokens come faster, though for a latent cache each GPU still reads every
sequence’s full cache. DP attention keeps each request’s cache on one GPU, so the node holds more
sequences, fewer than eight times as many because of the copied weights (Figure 4). Decode context
parallelism (DCP, -dcp 8 in vLLM) keeps TP for the weights and splits each sequence’s cache along its
tokens, so each GPU holds and reads an eighth of every sequence, at the cost of gathering queries and
merging partial results in every layer. On Kimi-K2-Thinking on 8×H200, vLLM’s recipe reports that DCP8
raised the KV capacity from 715,072 tokens to 5,721,088 (8.0×), and the whole node’s output by 43% in
a saturated random-token benchmark.
TP helps least at high concurrency, where the copied cache limits the batch; DP attention and DCP help least at low concurrency, where there is no batch to gain and the communication adds latency. SGLang’s documentation: “DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.” vLLM’s Mistral Large 4 recipe: “DCP8 only pays off when KV capacity binds.” vLLM’s recipe for DeepSeek-V4-Pro on 8×B200 under agent traffic gives the crossover as a schedule: TP8 for 1 to 14 concurrent sessions, attention in two TP4 groups at 32 and in four TP2 groups at 48 and 64, with the experts spread over all eight GPUs, and DEP8 from 64 to 192. EP and DP attention also lose when load is uneven: each step waits for the GPU with the busiest experts or the longest sequences, so engines copy hot experts to spare GPUs. I3 covers the layouts, wide EP across many GPUs, and expert load balancing.
attention weights (TP: a 1/8 slice; DP: all of them)expert weightsKV cache, one bar = one sequence's worth of memory
Scheduling, and splitting prefill from decode
Engines batch continuously and mix prompt chunks into decode steps (00); vLLM schedules
all pending decodes first and fills the rest of the step’s token budget with prefill. A prefill
chunk large enough to make a step compute-bound lengthens it for every user decoding in it. Two flags control the balance.
--max-num-seqs caps how many sequences decode at once (per data-parallel rank);
--max-num-batched-tokens caps the tokens per step, draft tokens included, and so the chunk size.
vLLM’s tuning guide: “Smaller values (e.g., 2048) achieve better ITL because there are fewer prefills
slowing down decodes”, higher values give better TTFT, and “For optimal throughput, we recommend
setting max_num_batched_tokens > 8192 especially for smaller models on large GPUs.” When the cache runs
out, vLLM preempts a sequence and recomputes it later, which shows up as tail latency, so deployments
cap concurrency below the point where the pool fills.
Prefill/decode disaggregation (P/D) puts prefill and decode on separate pools of GPUs and transfers each prompt’s KV cache between them, so no prefill lands in a decode step and each pool gets its own layout. The costs: two pools to size and keep busy, each phase running on fewer GPUs, and a transfer per request (Figure 5). For an agent turn the transfer is the whole context’s cache unless the decode pool still holds it: 7.8 GB for 142K tokens of GLM-5.3 in FP8, against 0.13 GB for DeepSeek-V4.1-Flash’s compressed global cache of 890 bytes per token in its trained FP4 format, plus a sliding-window part of fixed size (computed; larger in vLLM’s FP8 format). The architecture decides whether P/D is cheap.
decode step (6,144 tokens)step carrying a prefill chunkKV transfertokens, and a stall between them
Colocated, 2,048-token chunks: first token after 8 (A) and 13 (B) units; longest gap between tokens 1 unit.
Disaggregated 4 + 4: first token after 7.6 (A) and 7.8 (B) units, transfer included; every gap 1.6 units, even but longer than the node's clean step of 1 (4 GPUs read more weight bytes each). Toy units.
Toy roofline: a step takes max(1, tokens in the step ÷ 8,192) units on 8 GPUs, so a chunk is nearly free until the step reaches the ridge, then stretches it. On 4 + 4, a prompt takes twice as long to prefill and a decode step takes 1.6 units.
The sources disagree on what the split gains. vLLM’s documentation gives two reasons to use it, “Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately” and “Controlling tail ITL”, and then: “Disaggregated prefill DOES NOT improve throughput.” Two sources measured throughput gains on their own setups. vLLM’s recipe for Mistral Large 4 on GB300 says “Disaggregated 4+4 beats colocated 8-GPU serving by 1.8–2.2x at 8k and 4–5x at 32k/60k, because colocated decode shares steps with prefill chunks”. The comparison is output tokens per second per GPU at a minimum per-user speed, on unique prompts with prefix caching off, and the prefill pool also uses expert parallelism, so not all of the gain is the split, and agent traffic with cached context would gain less. The Nemotron 3.5 report measured a split with no TP or EP on either side: on 11 VR200 GPUs (the generation after Blackwell) replaying SWE-bench Pro sessions with Nemotron 3.5 Super VL, 7 prefill plus 4 decode GPUs gave 10.3% more output tokens per GPU and 61% more per user than 11 colocated ones, at their best usable points, while median TTFT rose from 0.49 s to 1.12 s. Each result has its own scope, and none is a head-to-head on another’s setup.
The split helps little with short prompts, where prefill rarely stalls decode, and in small deployments, where two pools can’t each be kept busy. Large-cluster results usually combine it with wide EP, so the gain can’t be credited to the split alone (I4).
Not recomputing: prefix caching and tiered KV
An agent turn repeats the whole earlier context. Prefix caching keeps the KV cache of finished requests and reuses it when a new request starts with the same tokens. Engines store the cache in fixed-size blocks of tokens, like memory pages, and a hit reuses whole blocks, skipping their prefill; a miss on a 100K-token context is a full prefill of 100K tokens.
Caching needs memory and routing. Cached prefixes live in memory that live sequences aren’t using, so a large batch evicts them; engines move idle prefixes down a hierarchy, from GPU memory to CPU memory (DRAM) to SSD. And a hit happens only if the request reaches the GPUs that hold its prefix. vLLM’s Mistral Large 4 recipe routes every request of a session to the same attention GPU and reports that without this “the hit rate drops 14–22 points and TTFT p99 is 2–3x worse”.
DeepSeek’s deployment notes describe “roughly 100 PiB of SSD storage for the KV cache, with a TTL (time-to-live) of 3 days for agent traffic”. Kimi K3’s report pins each session to a primary and a secondary cluster by consistent hashing, because “a typical coding input carries a prefix of 400K tokens but requires a prefill increment of only 4K tokens” (§5.4.3); the secondary must re-prefill on failover. In an LMSYS post, the team running Iflytek’s GLM-5.1/5.2 coding service reports a hit rate “around 30%” without SGLang’s HiCache tiered cache and 95% with it and external storage, with median TTFT below 3 seconds. Pricing also favours caching: Wu reports that OpenRouter keeps a session with one provider only if its cache-read price is below its input price, so a provider that doesn’t cache loses the repeat traffic.
Caching gains nothing on traffic without repeated prefixes, or when sessions don’t land where their cache lives. Hybrid models make it harder, since a recurrent state can be resumed only where a snapshot was saved (00). vLLM: “a prefix-cache hit can resume only at a block boundary.” Kimi K3’s report says the shared block size for its mix of layers was “forced to 1024–6144 tokens”, “at such a coarse granularity caching is nearly useless”, and describes its fix, which reaches any 512-token boundary. I4 covers tiers and routing, and T2 the recurrent states.
More than one token per pass: speculative decoding
Speculative decoding lets a small drafter propose tokens, then runs the large model once over them; the model keeps the drafted tokens it agrees with, up to the first it rejects, plus one token of its own (P1). With sampling, an acceptance test based on rejection sampling makes the output distribution exactly the large model’s (Leviathan et al., 2022; Chen et al., 2023). If each drafted token is accepted with probability , independently, a step yields on average
At low concurrency a step that checks tokens takes about as long as a plain step: the cache and the dense weights are read once for all of them. Nemotron 3.5’s report: “In the low-concurrency regime targeted here, verifying a block of draft tokens costs similarly to generating a single token.” In an MoE the drafted tokens also pick their own experts, so at a small batch verification reads more expert weights than one token would, and the gain on a large MoE is smaller than on a dense model.
The gain is fewer decode steps per answer, so higher per-user speed. The cost is the drafter’s compute,
the large model’s work on rejected tokens, and KV cache for drafted tokens. Multi-token prediction (MTP) heads ship with the model: GLM-5.3’s config has
num_nextn_predict_layers: 1, and the recipe runs that layer five times per step. EAGLE-3 is a
separately trained small head. DFlash and DeepSeek’s DSpark draft a block of tokens at once; in
DeepSeek-V4.1-Flash, DSpark picks the number of tokens to verify per request “to maximize expected
system-wide token throughput under the current system load” (§2.4.3). I5 covers the drafters and their training.
The gain shrinks as concurrency grows. A batch of sequences verifies positions per step. As the batch grows, each verified position starts to cost compute time. In the limit where even a plain step is compute-bound, a verify step costs about plain steps, and the speedup falls toward the expected tokens divided by , which is below 1. LMSYS reports an accepted length around 2.7 on chat traffic with 8 verified positions for Kimi K3: up to 2.7× when verification is free, 0.34× when it is fully compute-bound. Figure 6 shows the falloff measured in the DFlash paper. In its SGLang serving table (Table 3: Qwen3-8B on Math500, one B200), DFlash goes from 5.1× at one request to 2.8× at 32.
DFlash, 10-token blockEAGLE-3, 60-token tree
Both lines: DFlash paper, Table 5, as printed: Llama-3.1-8B-Instruct, GSM8K, SGLang (FlashInfer, Spec-v1), one B200.
SGLang’s presets for the original DeepSeek-V4 checkpoints draft three tokens (four verified positions) for “low-latency”, one for “balanced”, and none for “high-throughput”, because “at saturation the verify step costs more than it saves”. vLLM’s Qwen3.5 recipe warns that MTP “degrades throughput under high concurrency because speculative tokens consume KV cache capacity”.
Speculative decoding helps most for one user and for the long tail of RL rollouts, where the last samples decode at a tiny batch: GLM-5’s report says MTP “provides disproportionately large benefits on the long tail”. A drafter is not always the lab’s: Kimi K3’s report trains an EAGLE-3 head, while vLLM and SGLang serve K3 with DSpark drafters trained by others.
Smaller numbers: quantization
Quantization stores numbers in fewer bits: the weights, the activations, or the KV cache, with a different gain for each. A format named WxAy has x-bit weights and y-bit activations; the arithmetic runs at the activations’ width.
Fewer bits per weight mean fewer bytes per decode step and fewer GPUs to hold the model. Experts hold most of an MoE’s parameters, so they go first. FP8 has 8 bits; FP4 formats have 4 bits plus a shared scale per small block of values, which MXFP4 sets per 32 values and NVFP4 per 16 (I1).
For activations, the gain is faster matrix multiplication, but only where the tensor cores, the GPU’s matrix-multiply units, support the format. Hopper has FP8 tensor cores and no FP4 ones. On Hopper, vLLM runs NVFP4 experts “via Marlin W4A16”: the Marlin kernel expands 4-bit weights to 16 bits before multiplying, which saves bytes but not arithmetic. Blackwell multiplies FP4 natively (W4A4). Nemotron 3.5’s report measures the difference for its NVFP4 Super VL checkpoint on an agentic benchmark at 50 tokens per second per user: against BF16 weights, 1.86× the output per GPU on an 80 GB H100 with W4A16 and 2.43× on GB200 with W4A4 (the report’s Figure 10). Native FP4 arithmetic is worth about 1.3× there, across two GPU generations. DeepSeek’s FP4 experts run as FP4 times FP8 even on Blackwell; the report notes that “the peak FLOPs for FP4 FP8 operations are currently the same as FP8 FP8 on existing hardware”.
A smaller KV cache fits more sequences per GPU, its main use. For a model that reads its whole cache every step it also shortens each step at long context; for GLM-5.3, whose sparse attention reads 2,048 selected entries per step, it is mainly a capacity lever. GLM-5.3’s FP8 cache is 0.58 of BF16, as computed above. DeepSeek-V4.1-Flash went further and trained an FP4 main KV cache into the model, “where FP4 reduces storage rather than accelerates matrix multiplication” (§2.4.4); vLLM’s recipe still serves it with an FP8 cache.
Quantization fails in two ways. One is hardware without the format, as on Hopper with FP4 arithmetic. The other is quality loss that shows up on the deployer’s traffic but not in the averages. Nemotron 3.5 reports a median accuracy recovery of 99.62% across 36 benchmarks for its NVFP4 checkpoint, and answers 0.44% longer on average. The serving company umans.ai ran a four-day public trial of a third-party NVFP4 conversion of GLM-5.2: it served “200+ tokens per second”, a volunteer reported “COT collapse, and not even at deep context”, and the company concluded “the speed was real, the quality was not” and unlisted the model.
Some labs train their low-bit format in with quantization-aware training (QAT), so the shipped weights are the evaluated ones: Kimi K3 (MXFP4 experts, QAT “throughout the entire post-training stage”), DeepSeek-V4 (FP4 experts) and Kimi-K2-Thinking (INT4). gpt-oss-120b was post-trained with MXFP4 experts and evaluated in that format. GLM-5’s report applies INT4 QAT in supervised fine-tuning, yet GLM-5.3 ships in FP8, and its 4-bit versions are third-party conversions, to be tested on your own traffic. O2 covers training in low precision.
The other end: one user, one machine
At a batch of one, nothing is shared, so the batching levers are gone and every lever that makes one
user’s tokens faster gives its full gain. An MoE model makes local serving possible at all: each token
uses only a few experts, so the experts can sit in CPU memory and be computed there by the CPU, while
attention, shared layers and the cache sit on the GPU. DRAM bandwidth then sets the speed. llama.cpp
exposes this as flags: --cpu-moe keeps “all Mixture of Experts (MoE) weights in the CPU”, --n-cpu-moe N those of the first N layers, and -ot places tensors by name pattern. KTransformers is built around
the same split. GGUF, llama.cpp’s file format, carries the local weight formats, such as Q4_K_M, a
4-bit format with per-block scales (I1).
KTransformers ran DeepSeek-V3/R1 at 12.4 to 13.4 tokens per second (8K to 2-token prompts) in Q4_K_M on a 24 GB RTX 4090 and a dual-socket Xeon; a single socket needs 382 GB of RAM. Its guide for DeepSeek-V4-Flash reports “20+ tok/s on a single RTX 5090” with at least 200 GB of system memory, for one request at 16K context. Unsloth’s 1.8-bit Kimi-K2-Thinking runs at “around ~1-2 tokens/s” on one 24 GB GPU with 256 GB of RAM.
The floor is server-class memory: hundreds of gigabytes of RAM for the experts. A model shipped in a low-bit format lowers it: gpt-oss-120b in native MXFP4 runs “on a single 80GB GPU”. Local speculative decoding exists too: Poolside’s llama.cpp fork for Laguna-S-2.1 carries a DFlash drafter with seven draft tokens.
Kernels
A kernel is the GPU program for one operation, and a new architecture saves nothing until its kernels exist in the serving engine. Kimi K3 wrote “fused kernels, KDA Context Parallelism, and state-aware prefix caching” for its linear-attention layers. Day-0 recipes therefore pin engine builds: K3 needs a dedicated vLLM image and CUDA 13, and Mistral Large 4 needed a dedicated image before any release supported it. Kernel libraries also move on: FlashMLA removed Hopper support on 2026-09-30. Each kernel family reduces one cost term from 00, and I2 covers them.
| Kernel family | Cost term it reduces |
|---|---|
| FlashAttention, FlashMLA | attention bytes (tiles kept on chip; MLA and sparse variants) |
| DeepGEMM, fused MoE kernels | expert matrix multiplications with few tokens per expert |
| DeepEP | expert-parallel all-to-all (dispatch and combine) |
| CUDA graphs | launch overhead at small batch; with some attention kernels, only for steps without prefill |
Three recipes side by side
The same model gets different settings for different workloads. Each cell gives the setting, then its source; the second column says which kind of lever the row is, in the callout’s terms.
| Lever | Kind | Interactive chat, few users | Agentic coding, long reused context | Throughput, offline and RL rollouts |
|---|---|---|---|---|
| Parallelism | both | TP8 (Qwen3.5 “Latency-Focused”; Mistral Large 4 “best interactivity”); DP attention off (SGLang) | TP8 at 1–14 sessions, DEP8 at 64–192 (DeepSeek-V4-Pro, B200 agent lane) | DP8 + EP (Qwen3.5 “Throughput-Focused”); EP64 and DP64 over 8 nodes, for KV room (GLM-5 RL) |
| Prefill/decode split | batch, tail latency | colocated in the latency presets cited here | on, 4 + 4, DCP on decode (Mistral Large 4 agentic) | on for short inputs (Mistral Large 4); on for tail latency (GLM-5 RL) |
| Prefix cache | batch, TTFT | on by default; off in Qwen3.5’s latency preset (a hybrid model, where prefix caching is experimental) | on, with session-sticky routing (Mistral Large 4; DeepSeek-V4-Pro session ID); SSD tier (DeepSeek) | on (Qwen3.5 throughput preset); DRAM pool for RL rollouts (Kimi K3) |
| Drafter | speed per user | 3 drafts (SGLang DeepSeek-V4 “low-latency”); MTP-1 (Qwen3.5) | DSpark with 6 (DeepSeek-V4-Pro agent preset); none (Mistral Large 4 agentic, DCP decode) | 1 draft “balanced”, none “high-throughput” (SGLang DeepSeek-V4); block-6 DFlash for RL (MiMo-V2.6) |
The vLLM recipes cited here, except Qwen3.5’s, set an FP8 KV cache whatever the workload; SGLang’s DeepSeek-V4 presets and the RL reports don’t state one. Within single recipes, draft length shrinks as concurrency grows. Mistral’s agentic preset runs no drafter at all, because DSpark doesn’t run on its DCP decode pool.
Where the models in the map sit
Reference deployments from recipes, model cards and reports, as of 2026-11-07. The recipes change: the GLM-5.3 generator now defaults to B300 with MTP off, and DeepSeek-V4-Pro’s default serves the 0813 checkpoint. A — means the source is silent; “default” means no KV format is set, so the engine uses the model’s dtype.
| Model | Reference deployment | Weights shipped | KV format | Drafter |
|---|---|---|---|---|
| GLM-5.3 | 8×H200 TP8 at 256K; 8×B200 for 1M | FP8; NVFP4 and W4A8 third-party | FP8 | MTP (depth 1), 5 drafts |
| DeepSeek-V4-Pro(-0813) | 8×H200 TP8 + EP; GB200 DEP8 + DEP8 split; 4×GB300 DP4 + EP (card) | FP4 experts, FP8 rest (QAT) | FP8 | DSpark, 5–7 drafts |
| DeepSeek-V4.1-Flash | 4×GB200 TP4 | FP4 experts, FP8 rest | FP8 in vLLM; FP4 main KV trained (report) | DSpark |
| Mistral Large 4 | 4×GB300 TP4 (NVFP4); 8×GB300 TP8 or 4 + 4 split; 2×8 H200 (FP8) | FP8; NVFP4; MXFP4 on AMD | FP8 | DSpark (in the checkpoint), 3 drafts |
| Kimi K3 | 8×B300 TP8; at least 8×GB300, multi-node for production | MXFP4 experts (QAT) | FP8 | DSpark in vLLM and SGLang; EAGLE-3 in the report |
| Kimi-K2-Thinking | 8×H200 TP8, + DCP8 for throughput | INT4 weights (QAT) | default | — |
| Qwen3.5-397B-A17B | 8×H200 TP8 (BF16); GB200 DP4 + EP (NVFP4) | BF16; FP8; NVFP4; GPTQ INT4 | default (vLLM) | MTP-1; DFlash (third-party) |
| MiniMax-M3 | 8×H200 TP8 at 64K; DP8 + EP | BF16; MXFP8 and FP4 variants | FP8 opt-in | EAGLE-3 (third-party), 3 drafts |
| gpt-oss-120b | 1×H200 (card: any single 80 GB GPU) | MXFP4 experts (post-trained) | default; FP8 on Blackwell | — |
| MiMo-V2.6 | 2 nodes, SGLang TP16 with DP attention 2 and EP16 | FP8, MXFP4 experts (QAT) | — | DFlash block-6 in RL |
| Nemotron 3.5 Super VL | 1×GB200 | BF16; NVFP4 (W4A4 on Blackwell, W4A16 elsewhere) | FP8 | MTP (5 drafts in the headline benchmark) |
| Laguna-S-2.1 | TP4 (card; GPU not stated) | BF16; FP8, NVFP4, INT4, GGUF | — | DFlash, 7 drafts |
The code check for multi-token prediction finds an MTP module in GLM-5.3, the DeepSeek-V4 family, Qwen3.5 and MiniMax-M3, and none in Kimi K3, Mistral Large 4, gpt-oss-120b and Laguna-S-2.1. Kimi K3 and Laguna are served with separate drafters, Mistral’s DSpark drafter ships in the checkpoint but outside the model code, and gpt-oss has none. MiniMax-M3 declares MTP modules that its recipe doesn’t use.
What the reports leave out
No lab report gives a realized price per token. The only first-party revenue figure is DeepSeek’s: one cluster “serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at the off-peak price)”, without its GPU count or price mix. Production hit rates appear only as targets (Revolut’s 90%), platform averages and vendor case studies. Benchmarks also differ from production in drafting: AgentX fixes the drafter’s acceptance length per model from a separate benchmark, “because synthetic tokens can distort draft-token acceptance”, and GLM-5.3’s recipe warns that “MTP’s acceptance rate is usually low in synthetic runs”.
Easy to get wrong
- A throughput without its per-user speed, its TTFT and the tokens it counts. Agentic benchmarks count cached input, about 95% of the input; the output price is about 30 times the blended price per token on that traffic.
- “Fits on 8 GPUs” without context and concurrency: GLM-5.3’s 1M context on 8×B200 is for about one sequence.
- Speedups from speculative decoding measured at one request, or on a dense model. DFlash’s 5.1× on an 8B model is 2.8× at 32 requests on the same GPU, and in the same paper a large EAGLE-3 draft tree is a slowdown from 16 requests.
- FP4 weights are not FP4 arithmetic. On Hopper, FP4 weights are expanded to 16 or 8 bits before the multiplication; they save memory and bandwidth, not FLOPs, and the missing FP4 units explain only a small part of why Hopper loses.
- A median accuracy recovery says nothing about the tails, where “COT collapse” shows up.
Check yourself
1. A node rents for $7.40 per GPU-hour and sells tokens at $0.0188 per million. At a promise of 100 tokens per second per user it produces 316,000 tokens per second per GPU. What utilization does it need to break even, and what happens if the price per token is computed with a 0.72% output share instead?
Revenue at full load is 316,000 × 3,600 × 0.0188 / 10⁶ = $21.4 per GPU-hour, so break-even is 7.40 / 21.4 = 35%. With a 0.72% output share the price is 0.9928 × 0.013 + 0.0072 × 0.57 = $0.0170 per million, revenue $19.3 per hour, and break-even 38%.
2. GLM-5.3 runs on 8×H200 with TP8 and holds four 128K-token users. Why does each of them cost the same 7.2 GB on every GPU, and which two layouts avoid that?
MLA caches one latent head per layer, which tensor parallelism can’t split, so every GPU holds the full
cache of every sequence: 54,728 bytes per token × 131,072 tokens = 7.2 GB, on each of the eight GPUs.
Decode context parallelism (-dcp 8) splits each sequence’s cache along its tokens, so each GPU holds an
eighth. DP attention gives each GPU its own sequences instead, but then every GPU also holds the full
attention weights, which takes back part of the room.
3. A drafter’s tokens are accepted 2.7 at a time on average, and each step verifies 8 positions. What speedup do you expect at one request and at saturation, and why do recipes cut the draft length as concurrency grows?
At one request a verify step costs about as much as a plain step, so the speedup approaches 2.7×. At saturation the step is compute-bound and costs about 8 plain steps, so the speedup is about 2.7 / 8 = 0.34×, a slowdown. In between, each extra draft adds a little accepted output and a growing amount of verification work, so the best draft length shrinks; drafted tokens also take KV cache that would hold more sequences.
4. vLLM’s documentation says prefill/decode disaggregation does not improve throughput; the vLLM recipe for Mistral Large 4 measures 1.8–2.2× at 8K and 4–5× at 32K–60K more output per GPU with it. What would you check before believing either for your deployment?
The operating point (the recipe’s ratio is at a minimum per-user speed), the input length (the gain grows with it), what else changed (its prefill pool also uses expert parallelism), whether prefix caching was on (it was off), and the KV transfer size for your model. vLLM’s statement carries no operating point; read it as raw throughput with no latency target.
In the map
Methods
- Continuous batching 1 source
- Chunked prefill 6 sources
- Prefix caching 10 sources
- KV-cache offloading 1 source
- Tensor parallelism 6 sources
- Expert parallelism 12 sources
- Attention data parallelism 5 sources
- Decode context parallelism 1 source
- Prefill-decode disaggregation 4 sources
- Speculative decoding 15 sources
- Multi-Token Prediction 19 sources
- DFlash 6 sources
- DSpark 14 sources
- EAGLE 10 sources
- FP8 KV-cache quantization 12 sources
- NVFP4 quantization 4 sources
- MXFP4 Quantization 3 sources
- W4A16 quantization 1 source
- W4A4 quantization 1 source
- GGUF 2 sources
- CUDA Graph capture and replay 4 sources
- vLLM 7 sources
- SGLang 4 sources
- llama.cpp 1 source
- KTransformers 2 sources
Taxonomy nodes
- inference & serving 9 methods
Further reading
- Yingjun Wu, “Is hosting an open-source inference engine profitable?” (X, 2026-11-02): the full break-even model, rent indices and practices; its tables are images.
- vLLM blog post on DeepSeek-V4.1-Flash with prefill/decode split and wide EP on GB300 (vllm.ai, 2026-10-23): the curve data behind the rack-scale row, and the DeepInfer comparison with its limits.
- David Abitbol’s Revolut talk at AI Engineer NYC (2026-10-13), with transcript on ai.engineer; Luca Berton’s write-up for a summary.
- vLLM recipes for GLM-5.3, DeepSeek-V4-Pro, Mistral Large 4, Kimi K3 and Kimi-K2-Thinking (recipes.vllm.ai): command lines per GPU and workload, and the measured tables quoted here.
- Nemotron 3.5 Super and Lightning technical report, §4–5: quantization and serving measured under one usability floor.
- Chen et al., “DFlash” (arXiv 2602.06036): speculative-decoding speedups by concurrency.
- Origins: continuous batching is Orca’s iteration-level scheduling (Yu et al., 2022); chunked prefill is from Sarathi (Agrawal et al., 2023); the prefill/decode split from DistServe and Splitwise (2024); speculative decoding from Leviathan et al. (2022) and Chen et al. (2023); paged KV from vLLM’s PagedAttention (Kwon et al., 2023).
- The inference series: I1 (quantization), I2 (kernels), I3 (parallelism), I4 (scheduling and the KV-cache system), I5 (speculative decoding).
