Open Models Almanac

Inference & serving · I3

Parallelism for serving large MoE models: what each layout copies, and what it sends

Every layout copies something onto every GPU: tensor parallelism the single-head KV cache, data-parallel attention the non-expert weights. The batch and the context decide which copy costs more, and memory often decides before speed does

Illustration of eight small wooden desks in a gentle arc. Each desk holds the same thick ink-blue ledger, and small teal parcels travel between the desks along thin gold rails. One desk carries a slightly taller stack of parcels than the others.

vLLM publishes a tested recipe for serving DeepSeek-V4-Pro (its 0813 checkpoint), a 1.6-trillion-parameter mixture-of-experts (MoE) model, on one node of eight B200 GPUs under replayed agent traffic. The recipe splits the model across the eight GPUs in four different ways as the load grows. Up to 14 concurrent sessions it runs tensor parallelism over all eight GPUs (TP8). At 32 sessions it splits attention into two groups of four (TP4×DP2), at 48 and 64 into four groups of two (TP2×DP4), and from 64 to 192 (the recipe lists 64 under both) it gives every GPU its own sessions and spreads the experts over all eight (DEP8). The model, GPUs and benchmark stay the same, though the data-parallel lanes add a host-memory cache tier and a router that keeps each session on one GPU, and the recipe gives the layout at each concurrency, not the measurements behind it.

The serving primer (02) gave the rule of thumb behind that schedule: tensor parallelism at low concurrency, data-parallel attention with expert parallelism at high. This article explains why it holds, where the switch sits, and what changes across a whole rack.

What one decode step costs, and what can be split

Take one decoder layer of an MoE model (C2): an attention block, then a router that sends each token to kk of EE experts, small feed-forward networks of which only the chosen ones run, plus, in most of these models, one shared expert that every token uses. DeepSeek-V4-Pro’s config has n_routed_experts: 384, num_experts_per_tok: 6, n_shared_experts: 1, hidden_size: 7168 (dd) and num_hidden_layers: 61 (LL). Counting from the config and the modeling code, the routed experts are 1,547B of its 1,573B parameters, 98.4% (computed; the safetensors index adds up to 864.7 GB, with the experts in FP4 and the rest mostly in FP8).

A decode step runs this stack once for every sequence being served, producing one new token each, or a few when speculative decoding drafts ahead (02). Call the number of sequences in the step SS (the concurrency; the sources count sessions or requests), the tokens per sequence per step qq (1 plus any drafted tokens), and the tokens per step across the node B=qSB = qS; TT is the context length. As 00 showed, decode is limited by memory bandwidth: a step takes roughly the bytes each GPU reads from its memory (HBM) divided by its bandwidth, 4.8 TB/s on an H200 and 8 TB/s on a B200 (NVIDIA’s figures). Each GPU reads the weights of the layers it runs once per step, however many sequences share them (for experts, only those some token chose), and the KV cache of every sequence whose attention it computes. Prefill, the pass over the prompt, is the opposite case: thousands of tokens share each weight read, and the cost to count is FLOPs.

Splitting a layer across nn GPUs adds communication, in collective operations: an all-reduce sums one vector from every GPU and gives every GPU the sum; an all-gather gives every GPU all the pieces; an all-to-all has each GPU send a different piece to each of the others. Each GPU’s share of such a job is a rank. Inside one server the GPUs talk over NVLink, 450 GB/s per GPU each way on H100 and H200 and 900 GB/s on Blackwell (NVIDIA quotes both directions summed). Between servers each GPU has one network card of 50 GB/s (400 Gb/s) on H100, H200 and B200 and 100 GB/s on B300 and GB300, a ninth to an eighteenth of its NVLink bandwidth. A GB200 or GB300 NVL72 rack joins 72 GPUs in one NVLink domain.

We carry DeepSeek-V4-Pro on 8×B200 through the article, with GLM-5.3, the primer’s example, as the contrast: its KV cache per token is twelve times larger, which moves every number below.

Tensor parallelism splits every matrix

Tensor parallelism (TP) gives each of the nn GPUs a slice of every weight matrix, the split Megatron-LM introduced for training (Shoeybi et al., 2019). In attention, each GPU takes h/nh/n of the query heads with their slices of the projections; in a feed-forward block, the first matrix is split by columns and the second by rows, so each GPU computes a partial result of the full width. After the attention block and after the feed-forward block the GPUs add their partial results with an all-reduce: two per layer. With TP over eight GPUs every GPU works on every token, and every GPU reads one eighth of every weight that the step touches. For a single user that is the fastest a node can go, since the step’s weight read is divided by eight.

What TP sends is the hidden state. Counted as a ring all-reduce over nn GPUs, which passes partial sums once around the ring to add them and once more to hand out the result, each GPU sends about 2n−1n B d b2\frac{n-1}{n}\,B\,d\,b bytes, with b=2b = 2 for BF16 activations. For DeepSeek-V4-Pro at B=1B = 1 that is 25 KB per all-reduce and 122 all-reduces per step (computed). At that size the transfer takes tens of nanoseconds; the time goes to the fixed latency of each collective (we assume 10 µs; no source in our set measures it), paid 122 times on the critical path, and the engines use faster kernels than a ring for it (LMSYS, on Kimi K3: “one-shot multicast stores for small messages, NVLS in-switch reduction for large ones”, the latter sending each vector once). The ring’s bytes take as long as a 10 µs latency only at about 360 tokens per step on eight B200s (computed), still small beside a decode step’s weight read; the bytes become the cost in prefill, with thousands of tokens per step, or once TP crosses a network card.

TP splits attention by head, so it cannot split a cache that has only one head. Multi-head latent attention (MLA; T1) caches one compressed vector per token and layer, shared by all query heads. GLM-5.3, Kimi K3 and Mistral Large 4 use it (code check: multi-head-latent-attention present). DeepSeek-V4 has no latent (kv_lora_rank is absent, and the code check finds no MLA for V4-Pro) but caches one shared key-value head per entry, num_key_value_heads: 1 with head_dim: 512. Either way there is nothing to split by head, so under TP8 every GPU holds and reads the whole cache of every sequence (“every rank holds a full copy”, LMSYS writes of Kimi K3). vLLM’s documentation gives the general rule: with gg KV heads on n≥gn \ge g GPUs the cache is copied n/gn/g times, so a grouped-query model with 4 KV heads at TP8, its example Qwen3-235B-A22B, keeps two copies. The small projections that produce a single head, such as GLM-5.3’s kv_a_proj_with_mqa, are copied too, which puts TP8 at 95.8 GB of weights per GPU for GLM-5.3 rather than the primer’s 756 GB ÷ 8 = 94.5 (computed).

Pope et al. (2022) met the same copy with multiquery attention on TPUs (the shared key and value “must be replicated on each chip”) and split attention by batch instead, which is the next layout.

Data-parallel attention with expert parallelism

Expert parallelism (EP) places whole experts on GPUs, as MoE training systems such as GShard (Lepikhin et al., 2020) already did: with EP8, DeepSeek-V4-Pro’s 384 experts sit 48 to a GPU. A token’s six chosen experts may live on six different GPUs, so before the experts run, each GPU sends each of its tokens to the GPUs that hold its experts (the dispatch), and afterwards the results come back and are summed (the combine). Both are all-to-alls, and DeepEP is DeepSeek’s library for them. DeepSeek’s reports send the dispatch in FP8 and the combine in BF16 (some engine paths send the combine in FP8 too). Per token and MoE layer that is about kn−1n d (1+2)k\frac{n-1}{n}\,d\,(1 + 2) bytes to the other GPUs, fewer when two of a token’s experts share a GPU: V4-Pro’s six experts sit on about 8(1−(7/8)6)=4.48(1-(7/8)^6) = 4.4 of the eight GPUs on average (computed, uniform routing).

Data-parallel attention (DP attention) gives each GPU its own sequences. Each GPU runs attention, at full width, only for its own sessions and keeps only their cache, so the single-head cache is stored once. vLLM and SGLang usually pair it with EP for the experts; vLLM’s recipes call the pair DEP (data and expert parallelism), and DEP8 means eight GPUs. The experts’ all-to-all is then the only exchange in a layer. DP attention does not require EP: both engines can keep the experts tensor-parallel behind it, gathering every token onto every GPU before the MoE layer and scattering the sums after it, and SGLang’s recipe for DeepSeek-V4 decode on AMD MI355X does that.

The engines count GPUs differently. vLLM’s DEP8 is --data-parallel-size 8 --enable-expert-parallel with a TP size of

  1. In SGLang, --tp 8 is the size of the whole group and --attn-dp-size 8 splits attention into eight groups of one GPU, so --tp 8 --attn-dp-size 8 with an expert-parallel backend (--moe-a2a-backend deepep or megamoe) is vLLM’s DEP8.

In exchange, everything outside the routed experts (attention, the shared expert, the dense layers, the embedding and output tables) is copied onto every GPU instead of split. For DeepSeek-V4-Pro that is 26.0 GB of weights, WsW_s below: 110.1 GB per GPU under TP8, which holds an eighth of it, and 132.8 GB under DEP8, which holds all of it (computed, release formats). For Kimi K3, LMSYS gives “around 61 GB for KDA plus 11 GB for MLA” per GPU, its two kinds of attention layer. Those bytes come out of the memory left for the cache, and all but the embedding table are read at every step.

All eight GPUs still meet at every MoE layer: vLLM’s documentation notes that “expert layers across all ranks are required to synchronize during every forward pass, even when there are fewer requests to be processed than DP ranks”. An idle GPU runs a dummy pass, joining every all-to-all and running its experts for the other GPUs’ tokens, and a step lasts as long as its busiest GPU.

A third layout sits between the two. TEP (tensor and expert parallelism) keeps attention under TP and moves only the experts to EP. Weights per GPU, the copied cache and the traffic are TP’s: every GPU already holds every token after the attention all-reduce, so each runs the tokens routed to its own experts and the results are summed as under TP. The matrix multiplications run on whole experts instead of an eighth of each, which keeps quantization blocks whole: TEP is forced on Qwen3.8-Flash-Next on H200, where “plain TP8 is incompatible with its 128-wide quantization blocks”, and it is the vLLM recipe builder’s default for DeepSeek-V4-Pro on H200, B200 and B300.

  • routed experts
  • other weights (attention, shared expert, embeddings…)
  • KV cache
  • bytes sent
TP8Attention: 16 of the 128 query heads on eachGPU; all of them read the one KV head, held onall eight (dashed: a copy)qkv050100150GB per GPU12345678Cache: the same 96 sequences of 128K on everyGPU (dashed: a copy)experts 104.4 GB · other weights 5.6 GB · cachepool 56.8 GB · 10 GB working spaceper layer: an all-reduce of the hidden stateafter attention and another after the MoE12.85 MBsent per GPU per layer, 256 tokens per stepTP8Attention: 16 of the 128 query heads on each GPU; all of them read the one KV head, heldon all eight (dashed: a copy)128 query headsthe one KV head (cache)050100150GB per GPU (linear, from 0)GPU 1GPU 2GPU 3GPU 4GPU 5GPU 6GPU 7GPU 8routed experts 104.4 GBother weights 5.6 GBcache pool 56.8 GB10 GB working space92% of memory (vLLM)Cache: the same 96 sequences of 128K on every GPU (dashed: a copy)per layer: an all-reduce of the hidden state after attention and another after theMoE12.85 MBsent per GPU per layer at 256 tokens per step (bar to scale; one block per collective)

Weights per GPU: 110.06 GB (routed experts 104.4 GB, 1/8 of each of 384 experts; everything else 5.6 GB, split eight ways).

Cache pool per GPU: 56.8 GB (92% of 192 GB, minus 10 GB working space, minus the weights).

Sequences of 128K the node holds: 96, each one's cache copied on all eight GPUs.

Sent per GPU per layer at 256 tokens per step: 12.85 MB (7 × B × d bytes, BF16), in 2 collectives: all-reduce and all-reduce.

Computed from the config, the release's weight formats and an FP8 cache; memory rules as in the inference primer (92% of HBM, 10 GB working space).

Figure 1.What each GPU holds and sends for DeepSeek-V4-Pro on eight B200s (computed from the config, release formats, FP8 cache). TP8 and TEP8 divide the non-expert weights by eight, but every GPU keeps the whole cache of every sequence, since the cache has a single head; DEP8 keeps each sequence on one GPU and copies 26 GB of non-expert weights onto all eight, which leaves 34.1 GB for cache per GPU instead of 56.8.

The crossover

Put the two lists side by side for one GPU in a decode step on nn GPUs. Assume every step is limited by memory reads, every sequence has the same context, and the sequences split evenly over the GPUs. Let WsW_s be the non-expert weights that TP splits and DEP copies, and κ\kappa the bytes of cache read per sequence per step. Under TP a GPU reads Ws/nW_s/n of those weights and SκS\kappa of cache, every sequence’s; under DEP it reads WsW_s and Sκ/nS\kappa/n, its own sequences only. The expert bytes are the same in both, on average. The two reads are equal when Ws/n+Sκ=Ws+Sκ/nW_s/n + S\kappa = W_s + S\kappa/n; both sides carry the factor (n−1)/n(n-1)/n, so for any nn the tie is at

S∗=Wsκ.S^{*} = \frac{W_s}{\kappa}.

Below S∗S^{*} sequences TP reads less; above it, DEP does. This is a toy: it leaves out communication and arithmetic and assumes perfect balance. It does show which quantities move the switch. The copied weights are fixed. The copied cache grows with every sequence and every token of context, so the longer the contexts, the earlier DEP wins.

S∗S^{*} is where TP8 and DEP8 tie, but at S∗S^{*} neither is best. Splitting attention into groups of aa GPUs, as vLLM’s TP4×DP2 and TP2×DP4 do, divides the weights by aa and copies each sequence’s cache aa times, so a GPU reads Ws/a+aSκ/nW_s/a + aS\kappa/n. That is smallest near a=nS∗/Sa = \sqrt{nS^{*}/S}: on eight GPUs, TP8 up to S∗/4S^{*}/4, TP4×DP2 up to S∗S^{*}, TP2×DP4 up to 4S∗4S^{*}, and DEP8 beyond (derived). It is the order of vLLM’s schedule, though vLLM switches at far fewer sessions.

For DeepSeek-V4-Pro WsW_s is 26.0 GB. Its cache is unusually small: compressed attention keeps one 512-byte FP8 entry per four tokens in 30 layers and per 128 tokens in 31 (T3 explains the design). The 30 four-token layers also run an indexer, a small scoring layer that keeps a 68-byte key per entry, reads all of them at each step and picks the 1,024 entries the attention then reads. In all the cache holds 4,474 bytes per token (computed). For a 128K-token sequence a step reads 66.8 MB of indexer keys, 15.7 MB of selected entries, 16.3 MB for the 128-token layers, which are read in full, and 4.1 MB of 128-token local windows: κ\kappa = 102.9 MB (computed). That puts S∗S^{*} at 253 sequences for 128K contexts, 641 at 32K and 38 at 1M. GLM-5.3 selects 2,048 entries with an indexer too; 363 MB of its 468 MB per 128K sequence are indexer keys, and its S∗S^{*} is 41.

At 128K, though, TP8 on eight B200s holds only 96 V4-Pro sequences (next section): the copied cache runs out of memory long before its reads outweigh the weights, so on this node capacity forces the switch first.

Adding communication moves these numbers without changing their order. Per GPU and layer, for a step of BB tokens, TP8 sends 2×2⋅78⋅Bd⋅2=7Bd2 \times 2\cdot\frac78\cdot B d\cdot 2 = 7Bd bytes, because every GPU takes part in every token’s all-reduce. Under DEP8 a GPU sends only for its own B/8B/8 tokens, to six experts of which 7/8 are remote, at about 3 bytes a value: B8⋅6⋅78⋅3.03 d≈2.0 Bd\frac{B}{8}\cdot 6\cdot\frac78\cdot 3.03\,d \approx 2.0\,Bd. So DEP8 sends 0.28 times a ring TP8’s bytes per GPU (0.50 against NVLS) with a BF16 combine (0.36 for GLM-5.3, computed), at every batch size, in two collectives per layer like TP8. TP’s extra bytes move the crossover earlier; an all-to-all’s higher fixed latency (assumed below) moves it later, and counts like extra copied weight: 10 µs more per collective is worth 11 GB of WsW_s on B200, 43% of V4-Pro’s (derived).

Figure 2 runs the full toy, with the communication and an assumed fixed latency per collective (10 µs per all-reduce, 20 µs per all-to-all; we found no published figure). On 8×B200 with one token per sequence it puts the switch at 584 sequences at 32K and 296 at 128K for V4-Pro, and at 152 and 72 for GLM-5.3. The hybrids, which pay both an all-reduce and an all-to-all per layer, never win in it on B200. With six drafted tokens per sequence, which multiply the communicated tokens but not the cache read, V4-Pro’s switch at 128K falls to 144 sequences.

Speculation does not combine with every layout: SGLang’s balanced and high-throughput DeepSeek-V4 recipes drop the draft model because they use DP attention (“DSpark is incompatible with” it “on current releases”), though its MI355X agentic recipes and vLLM’s DEP lanes keep it.

  • TP8
  • DEP8
  • TP4×DP2, TP2×DP4 (thin)
  • beyond the layout's cache memory
  • TP8 out of cache memory
  • toy crossover
  • S* = Ws/κ
DeepSeek-V4-Pro, B200, 128K01,0002,0003,0004,0008321285122,048tokens/s per GPUconcurrent sequences S (log scale)S* 253× 296TP8 out ofcache memorybeyond 96DeepSeek-V4-Pro, B200, 128K01,0002,0003,0004,00081632641282565121,0242,0484,096decode tokens/s per GPU (linear, from 0)concurrent sequences S (log scale)S* 253crossover 296TP8 out of cachememory beyond 96

Toy crossover: DEP8 overtakes TP8 at S = 296 sequences.

S* = W_s/κ = 253 sequences (26.0 GB of copied weights ÷ 102.9 MB of cache read per sequence per step).

TP8 holds at most 96 sequences of 128K; DEP8 holds 456.

Best layout at S = 256: TP8, but it holds only 96 sequences. Fastest layout by S: TP8, then DEP8 from 296.

Toy model: memory-bound reads plus communication, no overlap, assumed collective latencies (10 µs per all-reduce, 20 µs per all-to-all). With 7 tokens per sequence per step the y axis still counts one per sequence, since how many drafted tokens are accepted is not modelled.

Figure 2.A toy model of one decode step on eight GPUs (computed; assumed collective latencies, no overlap). TP8 reads every sequence's copied cache, DEP8 reads its copied weights, so DEP8 overtakes at the concurrency where the cache outweighs the weights: hundreds of sequences for DeepSeek-V4-Pro's small cache at 128K, tens for GLM-5.3's. Before that, TP8 on eight B200s runs out of memory at 96 V4-Pro sequences. With memory reads only, the four layouts take over in vLLM's order; the switching points do not match vLLM's.

In the full toy the switch comes far later than the recipe’s 32 to 64 sessions, and most of what it leaves out would move the real switch earlier. Agent sessions carry long contexts, and the recipe’s TP8 lane fixes “a fixed 50 GiB KV cache per GPU”: 12.0M tokens for the node, since every GPU holds a copy, or about 84 sessions at the 142,000-token median context of 02’s agent traces (computed); past that TP8 has to evict, whatever its read time. The DEP lanes add “the Simple CPU KV tier” behind the GPUs, while the TP8 points use “no host KV offload”; prefill chunks share decode steps; and small kernels run below their peak. Load imbalance, which the toy also ignores, pulls the other way, so for this workload the toy’s crossover is likely, not surely, above the real one.

NVIDIA’s Nemotron 3 Ultra report puts the rule in words: “At small batch sizes inference is bound by weight-read memory bandwidth, and wider TP wins … At large batch sizes communication of activation tensors becomes the bottleneck, and EP wins”.

Large batches matter for the experts too. Each of V4-Pro’s 384 experts gets 6B/384=B/646B/384 = B/64 of a step’s BB tokens on average. An FP4 weight costs 0.53 bytes with its scale and gives 2 FLOPs per token, so with FP8 activations on a B200 (4.5 PFLOP/s over 8 TB/s, 562 FLOPs per byte) an expert stops being limited by its read only at 562×0.53/2=149562 \times 0.53 / 2 = 149 tokens, a batch of some 9,600 (computed). Decode never gets there, so every added token shares expert reads paid anyway. Wide EP adds tokens through the weight memory it frees: the SGLang team’s DeepSeek-V3 decode on 72 H100s credits its 5.2× over a TP16 baseline that already used DP attention “largely” to “4× larger batch sizes enabled by EP”, which cut “per-GPU memory consumption of model weights”.

Capacity: what copying does to the cache pool

The layouts also differ in how many sequences fit at all. With the primer’s rules (92% of memory usable, 10 GB per GPU of working space) and an FP8 cache, a 128K-token sequence of V4-Pro takes 0.59 GB. Under TP8 each B200 has 56.8 GB left for cache and holds every sequence, so the node holds 96; under DEP8 each has 34.1 GB for its own sequences, 57 each, 456 in all, 4.75 times as many rather than eight (computed). For GLM-5.3 the counts are 9 and 56, 6.2 times. Where memory is tight, nothing may be left: on 8×H200 V4-Pro’s DEP8 weights exceed the usable memory by 4 GB per GPU (vLLM’s guide runs DEP8 there anyway, with the context capped “to leave KV headroom”), and Kimi K3’s exceed 8×B300’s by 39 GB per GPU (computed).

Measured pools show the same shortfall. On Mistral Large 4’s decode side on four GB300s (NVFP4, 256K-token context), vLLM reports 1.96M cache tokens under TP4, 4 × 1.51M = 6.04M under DP4 with EP (3.1×), and 7.81M with TP4 plus decode context parallelism, the layout of the next section (4.0×).

Figure 3.How many sequences fit. Left, computed from the configs (FP8 cache, 92% of memory, 10 GB working space per GPU): DEP8 holds 4.75 to 6.2 times what TP8 holds on eight GPUs, and Kimi K3's DEP8 weights do not fit on eight B300s at all. DCP8 reaches 6.9 times for Kimi K3; with the indexer keys copied on every GPU, as SGLang does, GLM-5.3 and DeepSeek-V4-Pro hold about what DEP8 holds. Right, measured by vLLM on Mistral Large 4: DP attention gives 3.1 times the cache tokens of TP4 on four GPUs, decode context parallelism 4.0 times.

Splitting the cache along its tokens

If the cache cannot be split by head, it can be split by position. Decode context parallelism (DCP) keeps TP for the weights and gives each of the cc GPUs in the group every cc-th token of every sequence: in SGLang’s words, “rank r owns position p when p mod c = r”, a cyclic split that keeps a growing sequence balanced. Each GPU computes attention over its own tokens for all query heads, and the partial outputs are merged by weighting each with its share of the softmax normalizer (each GPU’s log-sum-exp of its scores), the rescaling FlashAttention uses between tiles on one GPU. Unlike ring attention (Liu et al., 2023), which passes key-value blocks around in training and prefill, DCP leaves the cache in place and moves the one-token query and the partial outputs: one more exchange per attention layer, a single all-to-all in SGLang with the queries replicated. In vLLM, -dcp 8 on top of -tp 8 still uses eight GPUs and divides the cache copy by eight.

Where the whole cache splits by position, the capacity gain is close to the GPU count: vLLM’s recipe for Kimi-K2-Thinking (not in the map) on 8×H200 reports 715,072 cache tokens under TP8 and 5,721,088 with DCP8.

DCP adds time per token. On Mistral Large 4, vLLM found that “DCP8 only pays off when KV capacity binds” (5.44 ms per output token for DCP8 against 4.02 ms for TP8 at one 32K request, with a draft model). On replayed agent traces with a mean input of 101K tokens, the four-GPU TP4 decode side of a prefill/decode split ran out of cache at about 32 sessions and its median time to first token reached 14.95 s, against 0.72 s with DCP4; but “DCP4 decode costs 12–25% ITL” (ITL, the time per output token), “so at ≤16 sessions” plain TP4 was better.

DCP cannot split two parts of these models, at least in SGLang. A sparse-attention indexer scores every token, so SGLang keeps “the index-K cache … replicated on every rank” (on GLM-5.3, “about 6% at DCP2 and 17% at DCP4” of each GPU’s cache budget). Every GPU then also reads all the indexer keys. For V4-Pro at 128K that is 66.8 of the 102.9 MB a step reads per sequence, so DCP8 would cut the read only 1.4 times and would hold 429 sequences on eight B200s, fewer than DEP8’s 456 (computed; no engine recipe offers DCP for V4). GLM-5.3 would hold 57, against DEP8’s 56. vLLM seems to split these keys (our inference): Mistral Large 4 has an indexer, and its DCP4 held 4.0 times TP4’s cache tokens. And a linear-attention layer’s state has no positions to split. Kimi K3’s 69 KDA layers (Kimi Delta Attention, its linear-attention layer) keep a fixed state per request, 96 heads of 128 × 128 FP32 values per layer, which TP splits by head (LMSYS: “about 54 MB under TP=8, covering all 69 KDA layers”) and DCP, DP and EP cannot shrink further. Once DCP removes the cache copy, “the running-request cap becomes the binding limit”, and SGLang’s cookbook calls the state pool “the concurrency ceiling” for K3.

Prefill context parallelism (PCP) splits one long prompt across GPUs instead. On Mistral Large 4 with eight GPUs it gave “~3x lower c1 TTFT than TP8 on long prompts, but the lowest throughput” (“c1” is one concurrent request): 0.54 s instead of 1.87 s for a 32K prompt at one request, but at best 86.2 output tokens per second per GPU at 8K inputs against TP8’s 178.3 (vLLM, eight GB300s, with a draft model, measured without EP). Those numbers suit a separate prefill pool for long prompts whose first token must come fast.

Pipeline parallelism, for prefill and across nodes

Pipeline parallelism (PP) splits the model by layers: GPU one runs the first eighth of the layers and passes the hidden state on. Each GPU then holds one eighth of the weights and caches only its own layers, and the only communication is the hand-off between stages, which crosses a slow link cheaply. That made PP the standard way across nodes for dense models (vLLM’s documentation: “Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes.”); for MoE models the recipes more often spread EP across nodes (below).

PP does not make one sequence faster: its token still passes through all stages in turn. For prefill, with many chunks queued, the stages stay busy, and PP avoids TP’s all-reduces, which LMSYS calls “a barrier that cannot overlap with compute”. It also copies no cache: SGLang’s Kimi K3 cookbook notes that above TP1 “the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16”. On 16 GB200s SGLang measured Kimi K3 prefill (8,192-token prompts, 32 concurrent requests) at 4,550 tokens/s per GPU with PP16, 2,407 with TEP16 and 1,652 with TP16, and below about 8 requests “the pipeline cannot fill”. In decode a token passes the stages one after another, so its step takes as long as one GPU reading the whole model; more batches in flight only keep the other stages busy (our inference; no engine source says it in these words). That is why PP appears at decode only where the model does not fit one node: vLLM serves Kimi K3 on two B200 nodes as TP8 × PP2 with DCP8, and Mistral Large 4 on two H200 nodes as TP16 or PP2 × TP8.

Load imbalance and replicated experts

Under EP and DP attention, every MoE layer ends with all GPUs waiting for the slowest. A GPU is slower when its sequences are longer (more cache to read) or when its experts were chosen more often in this step. Even with uniform routing a small batch is lumpy: simulating V4-Pro’s routing over EP8, the busiest GPU gets 1.20 times the mean number of expert-token pairs at 64 tokens per step, 1.10 at 256 and 1.05 at 1,024 (computed). Pairs measure work, which sets the time where each expert sees many tokens (prefill, wide EP); in EP8 decode the time is the weights of the experts touched, and at 256 tokens nearly all are touched, so the bytes even out sooner. Real routing is not uniform. With expert popularity falling off as one over its popularity rank (a Zipf law; our assumption, for illustration) and the experts placed at random, the ratio stays near 1.75 at any batch (computed): the GPU holding the hottest experts gets more in every step. The training-time balancing of C3 evens out use over a long training batch, but nothing in it makes the experts on one GPU equally popular in a given hour of traffic.

The serving-time fix is expert-parallel load balancing (EPLB). From each expert’s token count over a recent window, DeepSeek’s EPLB code gives each spare slot to the expert with the highest load per copy, then places the copies heaviest first on the least-loaded GPU with a free slot, in the manner of Graham’s longest-processing-time rule for scheduling. It removes the skew that persists, not one step’s lumpiness. The DeepSeek-V3 report sets 32 redundant experts for prefill, chosen “based on statistics collected during the online deployment” and “adjusted periodically (e.g., every 10 minutes)”; vLLM’s EPLB by default rebalances every 3,000 steps on the load of the last 1,000. The copies cost memory, about 2.4 GB per GPU for one redundant slot in every MoE layer of DeepSeek-V3 according to vLLM’s documentation.

SGLang reports EPLB giving “1.49x (prefill) and 2.54x (decode)” on DeepSeek-V3 “in large-scale settings”, with expert statistics taken from the same distribution as the test data. On DeepSeek-V4.1-Flash on DEP32 across 32 GB300 GPUs, with a draft model whose experts were balanced too, vLLM measured EPLB raising balancedness (mean GPU load over maximum, so 1.75 above is 0.57) “from 0.35 to 0.94” and decode throughput by about 25% (computed from its figure data at 72 requests per GPU).

EPLB balances the experts, not the sequences. Under DP attention each GPU also has its own cache, so a router has to send each session back to the GPU that holds its prefix while keeping the GPUs’ loads even, and the two goals pull apart. On Mistral Large 4’s agent traces, vLLM found that without routing each session to the same DP rank the cache hit rate “drops 14–22 points and TTFT p99 is 2–3x worse”. Under TP there is one cache pool and no such choice.

  • work on the GPU's own 48 experts
  • work on a hot expert's copy (EPLB)
  • mean
  • busiest GPU
B 2560200400600expert-token pairs (work) per GPU16112192160311943615282611771178everyone waits for GPU 5: 1.88 × meanB 2560200400600expert-token pairs (work) per GPU, one MoE layer, one step (linear, from 0)161GPU 1219GPU 2160GPU 3119GPU 4361GPU 5282GPU 6117GPU 7117GPU 8mean 192everyone waitsfor this one: 1.88 × mean

This step: the busiest GPU has 1.88 × the mean work (balancedness = mean/max = 0.53).

Over 200 simulated steps: max/mean 1.75, balancedness 0.58.

Toy routing for DeepSeek-V4-Pro: 384 experts, 6 per token, 48 per GPU on eight GPUs; popularity is uniform or Zipf over a random ranking of the experts. EPLB here only copies the 8 hottest experts into one spare slot per GPU; engines also move the others.

Figure 4.A simulated MoE layer on eight GPUs (computed; toy routing, counted in expert-token pairs). Under expert parallelism the step lasts as long as the busiest GPU. Small batches are lumpy even when every expert is equally popular, and with skewed popularity the busiest GPU stays far above the mean at any batch; copying the eight hottest experts brings it from about 1.75 to about 1.33 times the mean.

Hiding the exchange behind the work

The dispatch and combine sit on the critical path: the experts cannot start before their tokens arrive, and the next layer cannot start before the results return. Overlap hides them by splitting each step’s batch in two, the double buffering of classical parallel code: while one half computes attention or experts, the other half’s tokens are in flight. The DeepSeek-V3 report describes it for serving (communication–computation overlap; it overlaps “the attention and MoE of one micro-batch with the dispatch and combine of another”); SGLang calls it two-batch overlap (TBO, --enable-two-batch-overlap), vLLM dual-batch overlap (DBO, --enable-dbo).

Each half is a smaller batch, so the GPU runs less efficiently. On DeepSeek-V3 SGLang measured TBO adding 27% to 35% to prefill throughput at the same tokens per GPU, and in decode “-27% at 32 tokens/device”, with a gain only above “a threshold between 64 and 128 tokens” and 25.5% at 256. vLLM splits a decode-only batch only above 32 tokens and a batch containing prefill only above 512, by default. DBO also needs DP attention, EP and a DeepEP backend, and “cannot be combined with … pipeline parallelism, prefill context parallelism, multimodal … or hybrid models” (Kimi K3, the Qwen3.5 family and Nemotron 3 mix in linear-attention or Mamba layers). None of vLLM’s recipes, and only SGLang’s MI355X cells for DeepSeek-V4, switch it on; elsewhere the overlap comes from fused kernels.

DeepSeek-V4 fuses the exchange into the expert kernel itself (MegaMoE, in DeepSeek’s DeepGEMM kernel library): experts are scheduled in waves, and “computation of current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently”. The report measures 1.50 to 1.73 times the speed of unfused kernels “for general inference workloads” and gives a rule for when the exchange hides completely. Per token and expert, the expert does 6 d de6\,d\,d_e FLOPs (three d×ded \times d_e matrices at 2 FLOPs per weight, ded_e the expert width) and the exchange moves 3d3d bytes, so at arithmetic rate CC and link bandwidth β\beta the exchange hides when 6dde/C≥3d/β6dd_e/C \ge 3d/\beta, that is C/β≤2deC/\beta \le 2d_e: 6,144 FLOPs per byte for V4-Pro. A B200 running its experts at the full 4.5 PFLOP/s of dense FP8 arithmetic would need about 730 GB/s of interconnect bandwidth by that rule (computed), within NVLink’s 900 GB/s per direction and far beyond a 50 GB/s network card. That is the regime of prefill. In decode the experts wait on their weight reads, which leaves the exchange more time: on eight B200s its bytes fit under V4-Pro’s expert reads at any decode batch over NVLink (the bound, some 13,000 tokens per step, lies past the 9,600-token ridge) and up to about 740 tokens over a network card (computed). The kernels belong to I2.

Figure 5.Two-batch overlap, schematic. With the step's batch split in two, one half's tokens travel to and from their experts while the other half computes. The saving is the communication time that fits under the other half's work; the cost is that each half is a smaller batch, which in SGLang's DeepSeek-V3 decode lost 27% at 32 tokens per GPU and gained 25.5% at 256.

Wide EP and the rack

Once DP attention and EP are in place, nothing ties the layout to one server. Spread over more GPUs, each holds fewer experts and reads fewer expert bytes per step, while the copied non-expert weights stay the same per GPU. DeepSeek’s February 2025 production overview describes a decode unit of 18 nodes with “Routed Expert EP144, MLA/Shared Expert DP144”, serving on average about 14.8k output tokens per second per node of eight H800s (a variant of the H100).

On H100 or H200, EP beyond eight GPUs crosses the network card, at a ninth of NVLink’s bandwidth or less. On a rack-scale machine that limit goes away: on GB200 NVL72, NVIDIA’s Nemotron 3 Ultra report notes, “all 72 GPUs share a single NVLink domain, so wide EP can span the full system without paying for cross-domain interconnects in the all-to-all path.” The SGLang team’s DeepSeek-V3 decode on 48 GPUs of a GB200 NVL72 reported 13,386 output tokens per second per GPU with FP8 attention and NVFP4 experts, and 9,087 at the H100 run’s precision, 3.3 times the H100’s per-GPU rate (computed; their headline 4.8× compares different precisions).

Each doubling of the EP group gains less. vLLM’s DeepSeek-V4.1-Flash decode on GB200 at 128K context gains “from 22% to 3%” per doubling between DEP4 and DEP32: the expert kernel drops from 13.4 to 6.0 ms as each GPU holds fewer experts, while “the remaining work stays around 10 ms” (vLLM’s Figure 5; the step includes a draft model). Only the expert part shrinks, as in Amdahl’s law, and not in proportion: experts per GPU fall 7.5 times, the kernel’s time 2.2 times. For prefill the same post found “DEP4 gave only 83% of DEP2’s per-GPU throughput”: a long prompt already fills a few GPUs.

Wide EP usually comes with a split of prefill and decode onto separate GPUs (P/D; I4), for two reasons. Colocated, decode steps share the GPUs with prefill chunks, and under DP attention one GPU’s chunk stretches the step for all of them. Even against colocated TP8 (with a draft model, at 100 tokens/s per user or more), vLLM measured a 4+4 split of Mistral Large 4 at 1.8 to 2.2 times the throughput at 8K inputs and 4 to 5 times at 32K and 60K, “because colocated decode shares steps with prefill chunks”. And the all-to-all kernels come in a prefill kind and a decode kind: DeepEP’s low-latency mode, the decode kind, has the fixed shapes that CUDA graphs need, and vLLM’s documentation warns that both “may show poor performance for mixed workloads”.

Figure 6.Decode on GB200 at 128K context, measured by vLLM on DeepSeek-V4.1-Flash with a draft model and EPLB. Going from 4 to 32 GPUs per EP group cuts the expert kernel from 13.4 to 6.0 ms per step because each GPU holds fewer experts, but the rest of the step stays near 10 ms, so the per-GPU gain per doubling falls from 22% to 3%.

What the architecture does to the layout

The layout follows from a few config fields. Whether TP copies the cache depends on the number of KV heads: one latent or shared head (GLM-5.3, Kimi K3, Mistral Large 4, DeepSeek-V4) means a full copy per GPU, while a grouped-query model with at least eight KV heads, such as Hy3 or MiMo-V2.5-Pro (num_key_value_heads: 8), splits its cache cleanly over eight GPUs.

DeepSeek-V4’s cache is 8% of GLM-5.3’s per token, so its crossover comes several times later. A sparse indexer cuts the cache read per step but not its size, and its own key cache is one more single-head structure that TP copies and SGLang’s DCP does not split.

A linear-attention or state-space layer keeps a fixed state per request that TP splits by head and PP by layer, but no layout splits along positions; Kimi K3’s is 434 MB per request (computed), carried whole by one GPU under DP attention.

The all-to-all moves kk times the dispatched width per token; the expert size does not enter. LatentMoE (C2) projects the hidden state down before the experts: Kimi K3 dispatches 3,584 values instead of 7,168 and Nemotron 3 Ultra 2,048 instead of 8,192, half and a quarter of the volume, if the engine sends the latent (NVIDIA’s Nemotron 3.5 report says its LatentMoE “exchanges latent vectors”). How many experts a model has sets which EP sizes divide evenly and how many distinct GPUs a token reaches: Kimi K3’s 16 of 896 experts land on 7.1 of 8 GPUs, V4-Pro’s 6 of 384 on 4.4 (computed). The weight format constrains the layout too: besides the Qwen3.8-Flash-Next case above, DeepSeek-V4’s FP4 checkpoint is “TP-only” on Hopper in SGLang’s cookbook.

The layouts side by side

LayoutSplitsCopies on every GPUSends per layerBest atCosts
TPevery weight matrixsingle-head KV cache and projections2 all-reduces of the hidden stateone user, low concurrencycache copy limits batch and context; all-reduce latency
TEPattention by head, experts wholeas TPas TPexperts whose quantization blocks can’t be slicedas TP for the cache
DP attention + TP expertsexperts by slice, sequences between GPUsattention, dense layers, embeddingstoken all-gather + reduce-scatterdecode with few tokens per step (SGLang, MI355X)copied weights; split caches
DEPexperts whole, sequences between GPUsas the row abovedispatch + combine all-to-allhigh concurrency, long contextscopied weights, dummy passes, waits for the busiest GPU; prefix hits need routing; no draft model in some SGLang recipes
TP + DCPweights as TP; cache by token positionin SGLang, indexer keys and draft model’s cache; linear-attention state stays at TP’s shareTP’s all-reduces + one attention exchangelong contexts where the cache binds12–25% more time per token (Mistral)
PCPone prompt’s tokensas its base layoutattention exchange in prefilltime to first token on long promptslowest throughput
PPlayersnothingstage hand-offprefill, models larger than a nodeno gain for one sequence; needs work in flight

Where the models in the map sit

The table lists the layouts each engine’s recipe or cookbook names for a model, with the hardware; a blank means none was found. Of 16 models from 12 labs in the vLLM recipe builder, 13 default to plain TP on every GPU type; DeepSeek-V4-Pro and Qwen3.8-Flash-Next default to TEP, and Inkling’s two-node H100 default is TP8 × DP2 with EP. No default is DEP, which appears in the presets for measured traffic.

ModelDefault (vLLM builder)Other named layoutsSource
DeepSeek-V4-ProTEP8 (H200, B200, B300)guide: DEP8 on H200/B300; B200 agent lane TP8 → TP4×DP2 → TP2×DP4 → DEP8; GB200 P/D DEP8 + DEP8; SGLang on B200: TP8 for latency, DEP8 with MegaMoE for throughput; SGLang on H200 in FP4: TP8 onlyvLLM recipe, SGLang cookbook
DeepSeek-V4.1-FlashTP4 (H200, B200, GB200), TP2 (B300, GB300)agent lanes TP4 at low concurrency, DEP4 to DEP8 from 64 (B300: DEP2 from 8); GB300 P/D DEP4 prefill + DEP16 decodevLLM recipe
GLM-5.3TP8TP8 + EP for NVFP4 on Blackwell; SGLang DEP8 with DeepEP for throughputvLLM recipe, SGLang cookbook
Kimi K3TP8 on B300; TP16 over two nodes (H200, B200) or four trays (GB200); TP8 over two GB300 trays; TP32 over four H100 nodesTP8 + DCP8; TP8 × PP2 + DCP8 on two B200 nodes; SGLang P/D prefill PP8, decode TP8/DCP8; SGLang 8-GPU default prefill TP8vLLM recipe, LMSYS, SGLang cookbook
Mistral-Large-4TP8 (B200, B300), TP4 on GB300DP8 + EP for throughput; TP8 + DCP8 for long context; TP1×PCP8×DCP8 + EP for first-token latency (measured without EP); H200 TP16 or PP2 × TP8 (unmeasured)vLLM recipe
Qwen3.5-397B-A17BTP8DP8 + EP “for maximum text throughput under high concurrency”; DP4 + EP on GB200vLLM recipe
MiMo-V2.5-ProSGLang: attention TP8 × DP2, EP16 (--tp 16 --dp 2 --enable-dp-attention --ep 16)model card
Nemotron 3 UltraTP8 (B200, B300; NVFP4 too), TP16 over two H200 nodes (BF16)TP8 + EP (model card); report: wide TP for latency, wide EP for throughputvLLM recipe, report

Easy to get wrong

  • “EP moves more data than TP.” Per GPU it moves less inside one server: DeepSeek-V4-Pro’s dispatch and combine are 0.28 times the bytes of TP’s two all-reduces per layer (computed), and the share falls as the group grows. What DEP pays for is copied weights, a higher fixed latency per collective and, once it spans servers, bytes that cross the network card.
  • “DP attention multiplies the cache by the number of GPUs.” By less, since the copied weights come out of the pool: 3.1 times on four GPUs in vLLM’s Mistral Large 4 measurement, 4.75 to 6.2 times on eight in our arithmetic, and zero where the copy does not fit.
  • “The MLA cache problem is about MLA.” It is about the number of KV heads. DeepSeek-V4 has no latent and the same problem; a GQA model with two KV heads keeps four copies at TP8.
  • “A recurrent state cannot be split.” TP splits Kimi K3’s KDA state by head. What nothing splits is the state of one request along positions, so DCP raises cache capacity without raising the state’s concurrency limit.
  • “Decode context parallelism multiplies the cache by its group size.” Only where the whole cache splits by position. Where an indexer’s keys stay copied on every GPU, as in SGLang, DeepSeek-V4-Pro on DCP8 would hold fewer 128K sequences than on DEP8.

Check yourself

On eight GPUs, a model copies 20 GB of non-expert weights under DEP8 that TP8 splits, and reads 200 MB of a single-head cache per sequence per decode step, all of it in proportion to the context. At about what concurrency do TP8 and DEP8 read the same bytes in the memory-only toy? What happens to that number if the contexts double?

S∗=Ws/κ=20 GB/200 MB=100S^{*} = W_s/\kappa = 20\text{ GB} / 200\text{ MB} = 100 sequences, where TP4×DP2 and TP2×DP4 would read less than either. If the cache read per sequence doubles with the context, S∗S^{*} halves to 50: DEP8 wins earlier for longer contexts. Communication and latency shift the real value, which is why the article calls this a toy.

Why does decode context parallelism nearly multiply Kimi K3’s cache capacity by its group size while DP attention does not?

DCP keeps TP’s split of the weights and divides the cache of every sequence across the group. K3’s MLA cache splits whole by position, so the only thing DCP removes from each GPU is the cache copy. DP attention removes the copy too, but copies the attention and other non-expert weights onto every GPU, which shrinks the pool on each (for K3 on eight B300s, below zero). A model with an indexer gains less in SGLang, which keeps the indexer’s keys copied.

Kimi K3 on DCP8 still hits a concurrency limit even though its MLA cache has room. Why?

Its 69 KDA layers keep a fixed state per request, 54 MB per GPU under TP8. DCP splits caches along token positions, and the state has none, so each new request still costs the full state share on every GPU.

In the map

Methods

Taxonomy nodes

Further reading

  • Pope et al., “Efficiently Scaling Transformer Inference” (2022): partitioning dense models on TPUs, and batch-sharded multiquery attention, the ancestor of DP attention.
  • DeepSeek-V3 technical report, §3.4, and DeepSeek’s “Inference System Overview” (February 2025): prefill and decode units, redundant experts, two-micro-batch overlap, the production layout.
  • DeepSeek-V4 report, §3.1: the fused expert kernel and its bandwidth rule.
  • LMSYS, “Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs” (2025): DP attention, DeepEP modes, EPLB and two-batch overlap, with ablations.
  • LMSYS, “SGLang and Miles Add Day-0 Support for Kimi K3” (2026): PP for prefill, DCP for decode, the hybrid-state limit.
  • vLLM, “Serving DeepSeek-V4.1-Flash at Scale” (2026): wide EP on NVL72 racks, EPLB at DEP32, small prefill groups.
  • vLLM’s documentation on expert-, data- and context-parallel deployment, and its recipes for DeepSeek-V4-Pro, Kimi K3 and Mistral Large 4, where the layout tables live.
  • NVIDIA, Nemotron 3 Ultra technical report, §5.2: a lab’s statement of the TP-to-EP crossover and of what NVL72 changes.
↑