Serve GLM-5.3 on eight H200 GPUs to a single user, and each decode step has to read about 6.6 GB per GPU: 3.7 GB of weights that every token uses, 2.8 GB for the eight experts each layer picks, and 0.13 GB of cache. At the H200’s 4.8 TB/s that takes about 1.4 ms (computed below), the floor that 00 and 02 build their prices on. Run the same step without CUDA graphs, launching each GPU program separately, and the GPU runs about 1,840 of them, each followed by an idle gap of about 2.1 µs (Hazy Research’s measurement on an H100): some 3.9 ms of gaps, close to three times the work (our estimate, below).
The cost model assumes the GPU reaches its roofline. Whether it does is decided by kernels: whether attention streams the compressed cache at full bandwidth, whether an expert reads its weights as fast as memory allows when it sees one or two tokens, whether the exchange between GPUs stalls the step, and whether the GPU waits between operations. Serving recipes are full of these choices (--moe-backend deep_gemm_mega_moe, --attention-backend TOKENSPEED_MLA, cudagraph_mode: FULL_DECODE_ONLY). This article says which cost term each one moves.
What a kernel is, and where its data lives
A kernel is one program launched on the GPU: a matrix multiplication, a normalization, an attention computation. The CPU asks for it, the GPU runs it across its streaming multiprocessors (SMs, 132 on an H100 and 148 on a B200), and the result lands in high-bandwidth memory (HBM), the GPU’s main memory. The next kernel reads it from there. A kernel’s work is cut into thread blocks, each running on one SM, and the threads of a block run in groups of 32 called warps.
Three kinds of memory matter. HBM is large and comparatively slow: an H200 holds 141 GB and reads 4.8 TB/s. Each SM has a small, fast scratchpad, shared memory, of 228 KB on Hopper and Blackwell, plus registers. Data in shared memory or registers costs far less to reread than data in HBM, though shared memory has its own bandwidth limit, which on Blackwell becomes a bottleneck for attention (below). Matrix products run on tensor cores, units inside each SM; a thread block multiplies tiles of 64 or 128 rows.
00 gave the roofline: an operation that does floating-point operations (FLOPs) on bytes of HBM traffic has arithmetic intensity . Below the ridge, peak FLOP/s divided by bandwidth, the operation waits on memory and takes ; above it, it waits on arithmetic and takes . For an H200 with dense FP8 at 1,979 TFLOP/s the ridge is 412 FLOPs per byte; for BF16 at 989 TFLOP/s it is 206 (computed from NVIDIA’s datasheet, whose tensor-core figures are quoted “with sparsity” and halved here).
00 applied this to a whole forward pass. A step is a sequence of kernels, each at its own point on the roofline, and when they run one after another the step takes at least the sum of each one’s own bound, , plus the gaps between them. A single max over the whole step would let a compute-bound kernel hide behind a memory-bound one (off by up to a factor of two). The sum is a floor: every kernel at full bandwidth or at peak, overlapping only where the stack arranges it.
Which roof each decode kernel of GLM-5.3 faces
Take the worked example from 02: GLM-5.3, a 753B-parameter mixture-of-experts (MoE) model, served in FP8 with tensor parallelism over eight H200s (TP8: most weight matrices split eight ways, every GPU holding a slice of all 256 experts; the small latent projections and the cache are copied to every GPU). Its attention is multi-head latent attention (MLA) with a sparse indexer: each layer keeps one compressed latent vector per token, 656 bytes in FP8, and an indexer picks 2,048 of the cached tokens for each new query. Leave speculative decoding aside for now (one new token per sequence per step), group the decode step’s kernels into five classes and compute each class’s intensity per GPU at 8K context (compute.py §A):
| Class | What it reads | FLOPs per byte, batch 1 | batch 64 |
|---|---|---|---|
| Dense matrix products (attention projections, shared expert, router, output layer) | 3.7 GB of weights per GPU | 1.9 | 120 |
| Routed experts | the slices of every expert some token picked | 2.0 | 4.6 |
| Sparse latent attention | 2,048 latent entries per layer per sequence | 26.5 useful, 212 as run | same |
| Indexer scoring | an indexer key for every cached token, in 21 layers | 7.8 | 7.8 |
| All-reduce between GPUs | two per layer, over NVLink | latency-bound, not modelled |
The classes that read weights sit far below the ridge at every batch that fits: under TP8 the KV cache pool holds 73 sequences of 8K tokens (02), and at 64 the dense products reach 120 FLOPs per byte against a ridge of 412. They gain from batching because every token reuses the same weights. The routed experts barely gain, because 64 tokens spread over about 222 experts give each only 2.3 rows (Expert multiplications, below). The step’s floor is the sum of the classes’ times: 1,381 µs at batch 1 and 18,907 µs at batch 64 (computed), and at 64 the routed experts are 87% of it.
Two costs lie outside that floor. Real kernels this small don’t stream at full bandwidth (below). And the 156 all-reduces per step carry a few kilobytes each at batch 1, so they cost their latency, which no source we saved prices; it is why vLLM’s largest listed fusion is the one into the all-reduce.
- dense products
- routed experts
- sparse latent attention
- indexer
- attention, useful work of 8 heads (hollow)
- prefill chunk (hollow)
- FP8 roof
- BF16 roof
| decode, B = 1 | FLOP/B | µs | bound by |
|---|---|---|---|
| dense products | 1.9 | 764 | bytes |
| routed experts | 2.0 | 590 | bytes |
| sparse latent attention | 212 as run (useful 26.5) | 22.5 | FLOPs |
| indexer | 7.8 | 4.73 | bytes |
| step (sum) | 1,381 µs |
Prefill chunk of 2,048 tokens: dense products 806 FLOP/B (compute roof), routed experts 101 (bandwidth roof), counting activation reads and writes.
Computed, not measured: per GPU under TP8, FP8 weights and KV cache, 8K context, ideal kernels. Each class takes the longer of its bytes at 4.8 TB/s and its arithmetic at the dense peak it uses (FP8 1,979 TFLOP/s; attention in BF16, 989). Decode counts weight and cache bytes only; experts touched assume uniform routing, 8 of 256 per token. vLLM pads GLM-5.3's 8 attention heads per GPU to 64 in its FP8 sparse decode kernel, so the attention kernel does 8× the useful arithmetic; its time here is that of the padded kernel. "Verify 6" is a step of MTP with five drafts, as GLM-5.3's 8×H200 vLLM recipe serves it: six positions per sequence. Prefill marks also count activation reads and writes and leave out the output layer.
So the question for most decode kernels is how close they get to 4.8 TB/s. The average kernel of the batch-1 step reads about 3.6 MB, under a microsecond at full bandwidth, and in that time it has to start, fill the memory pipeline with requests and drain it. On batch-1 decode of Llama-3.2-1B on an H100, Hazy Research found that vLLM and SGLang “are only able to use at most 50% of available GPU bandwidth”; DeepGEMM’s table shows dense FP8 products with 64 rows moving 1.7 to 2.7 TB/s, half to four fifths of an H800’s 3.35 TB/s.
The attention kernel is the exception. With MLA every query head of a sequence reads the same latent entry, so per selected key it does FLOPs on 656 bytes, with the query heads on one GPU, the query positions computed at once, latent and position dimensions. FlashMLA’s documentation puts the crossover at for a BF16 cache on an H800 (about 62 for the 656-byte FP8 entry on an H200) ( is its ) and notes that DeepSeek decodes without tensor parallelism, so ” is 128 and the kernel is compute-bound”. Under TP8 a GPU of GLM-5.3 holds 8 of the 64 heads: 26.5 useful FLOPs per byte. But vLLM’s FP8 sparse kernel “only supports h_q = 64 or 128”, so it pads the 8 heads to 64 and computes 212 FLOPs per byte, at the BF16 ridge, discarding seven eighths of the arithmetic. SGLang pads the same way for DeepSeek-V4, reasoning that “MLA decode is typically memory-bandwidth-bound on KV reads”; here, at 212 against 206, it is nearly a tie.
Speculative decoding (I5) moves every class. The GLM-5.3 recipe for 8×H200 drafts five tokens with its MTP layer, so a step verifies six positions per sequence. With dense MLA the six would share their entries and the attention’s intensity would rise six times; with the sparse indexer each position selects its own 2,048, so it rises only by the overlap of the selections, from 212 to at most 1,274 (compute.py §H2). Six tokens also touch about 44 experts per layer instead of 8, and the batch-1 floor rises from 1,381 to 4,178 µs, for up to six tokens (computed).
Fusion: read once, write once
Most operations between the large matrix products are elementwise or row-wise: normalization, an activation, rounding to FP8. Each does a few FLOPs per element. Run as separate kernels, every one reads its input from HBM and writes its output back, and the next kernel reads that output again. Fusion merges a chain of such operations, often into the epilogue (the final step that writes the output) of the matrix product before or after them, so the intermediate stays in registers or shared memory. It is loop fusion, an old compiler transformation, and CNN libraries fused convolution, bias and ReLU for the same reason; what is new are the targets, such as FP8 rounding with its scales, collectives and the labs’ own layers.
Take RMSNorm followed by rounding to FP8, before GLM-5.3’s first attention projections in prefill. Unfused, the normalization writes its BF16 output to HBM and the rounding kernel reads it back, 2 bytes each way per element of width : 24,576 extra bytes per token, 5.1 ns at 4.8 TB/s. The projection that follows (6,144 inputs to 2,624 outputs, at FP8 peak) takes 16.3 ns per token, so the three-kernel chain takes 25.3 ns and the fused one 20.2 ns, about 20% less (computed, §H4); fusing SwiGLU with the rounding before an expert’s down projection saves about 10%. In the per-kernel sum, fusion deletes memory-bound kernels and their bytes and leaves the product’s term as it was. These are upper bounds, with the product at peak, and for GLM-5.3 vLLM runs none of its fusion passes (Compilation, below). vLLM lists the measured end-to-end gain of these two fusions as “1-4%”, and of fusing the all-reduce with the following RMSNorm as “5-20%”.
At decode the bytes saved are small: at batch 64 one MoE layer reads about 1.1 GB of weights per GPU and the fused activations save 3.7 MB of it, 0.34% (computed). At decode the gain is fewer kernels, for the reason in the opening (Launches, below).
The labs fuse much further. DeepSeek-V4.1 runs its hyper-connections (a widened residual stream mixed by small learned matrices) through one kernel, Mega-mHC, that combines “residual update, input mixing, and coefficient prediction” with the pre-norm and the FP8 conversion; with its other fused kernels, “each CSA2 Reuse Mode layer executes with only 15 kernels during prefill and 11 during decode”. Kimi K3 fuses its attention-residual merge and the following RMSNorm into the tensor-parallel all-reduce.
Attention kernels
FlashAttention: the score matrix never leaves the chip
Attention as written in a textbook computes the scores , a matrix per head, then the softmax , then . A naive implementation writes and to HBM between three kernels. For a dense head of GLM-5.3’s width at a 128K-token prefill, the FP32 scores alone would be 68.7 GB, and at 8K the three kernels move 0.82 GB per head (computed, §D). GLM-5.3 itself attends to 2,048 selected entries in prefill too, and under TP8 vLLM runs its prefill through the same FP8 sparse kernel as decode; the dense head only shows the term that tiling removes.
FlashAttention (Dao et al., 2022) computes the same result in one kernel. It splits , and into blocks that fit in shared memory and, for each block of query rows, walks through the key and value blocks, keeping a running output. The obstacle is the softmax, whose denominator needs every score of the row. The fix is the online softmax. Keep, for each query row, the largest score seen so far and the running sum of exponentials relative to it, starting from , , . When a new block of scores arrives:
and at the end the output is . The factor rescales what was accumulated under the old maximum, so the result equals the full softmax exactly in real arithmetic (in double precision a random 64-key example differs by , compute.py §D), and the matrix is never written. Two partial states merge by the same rescaling in any order, so a walk over blocks, a split across SMs and any tree of merges agree in exact arithmetic and differ only in rounding, which matters under “Determinism”. The paper proves the kernel needs HBM accesses for shared memory of size , against for the standard form.
- scores in this block
- the new maximum
- running maximum m
- before this block
- rescaled by e^(m − m')
- after adding this block
Block 1: m = 0.92, l = 1.927. Nothing has been summed before, so nothing is rescaled.
Toy numbers from a fixed seed, not a model's. After each block: m is the largest score so far, l the sum of e^(s − m) over the keys read, o the same sum weighted by each key's 2-dimensional value. Dividing o by l at the end gives the softmax-weighted average of the values, and no score is kept once its block has been added.
The kernels have been rewritten for each GPU generation. FlashAttention-2 reached “50-73% of the theoretical maximum FLOPs/s on A100” but about 35% on Hopper. FlashAttention-3 rebuilt it around Hopper’s asynchronous copy unit (TMA) and warp specialization, in which some warps only move data and others only compute, for “740 TFLOPs/s (75% utilization)” in FP16 on an H100. On Blackwell the tensor cores doubled while the exponential units did not, and “shared memory traffic and exponential operations now dominate execution time”; FlashAttention-4 (Zadouri et al.) adds a software exponential and conditional rescaling, reaching 1,613 TFLOP/s in BF16 on a B200 (71%). These are prefill-style, compute-bound benchmarks.
Decode has one query row per sequence, so the work per sequence is a single long walk over its cache. To keep all SMs busy at small batch, decode kernels split the cache of one sequence across several SMs and merge the partial outputs with the same rescaling rule (split-KV, also called flash-decoding), into a number of pieces that depends on the batch.
FlashMLA: latent caches, sparse selection, low precision
DeepSeek’s FlashMLA is the attention library for MLA. Its dense decode kernel reached “up to 3000 GB/s in memory-bound configuration and 660 TFLOPS in computation-bound configuration on H800 SXM5” (April 2025), about 90% of the H800’s 3.35 TB/s (computed). Later releases added kernels for the sparse attention of DeepSeek-V3.2 and V4, which read only the entries an indexer selects (the top 2,048 in V3.2, 1,024 in V4-Pro and 512 in V4-Flash, by their configs’ index_topk).
The sparse decode kernel shows how far a kernel is cut to one format. Its FP8 cache for V3.2 stores, per token, 512 FP8 values, four FP32 scales and the 64 position dimensions in BF16: 656 bytes. The arithmetic runs in BF16, so every selected entry is converted on the fly, and on Hopper that conversion became the limit: by DeepSeek’s count, for 64 query heads (what TP8’s padding produces), about 34 cycles of matrix work per cached token against “at least … ≈50 cycles for dequantizing each token”. Pairing two thread blocks on neighbouring SMs, each converting half the entries for both, took the kernel from 250 to 410 TFLOP/s on an H800 (128 heads, two query positions). V4 changed the layout to 584 bytes per token, V4.1 to 528, with an FP4 variant of 288 bytes.
Each layout comes with its kernel. On 2026-09-30 FlashMLA removed “support for the Hopper architecture and for earlier models (including DeepSeek V3 / V3.2 / V4.0)” and “changed the FP8 / FP4 KV cache format”; the current README requires a Blackwell GPU (compute capability SM100 or SM103) or Huawei’s Ascend 950 and gives sparse decode “up to 1024 TFlops on B200”. Hopper deployments, the worked example included, run on the engines’ own copies: vLLM vendors a FlashMLA build from before the drop, and vLLM’s FlashInfer sparse MLA backend covers FP8 caches on Hopper for models without position dimensions, such as GLM-5.3-Flash. The DeepSeek-V4 report states the co-design directly: the cache’s block size can be “any multiple of lcm(m, m′)” of its two compression ratios.
The indexer has its own kernels. It scores every cached token against each new query (in GLM-5.3, 32 heads of width 128 in 21 layers), and its reads grow with the context while the attention’s stay fixed at 2,048 entries: at 8K the indexer reads 22.7 MB per sequence per step against 104.8 MB for the selected latents, and from about 38K tokens it reads more than the attention it serves (computed, §F). DeepSeek’s scoring kernels live in DeepGEMM, which added an FP4 indexer in April 2026; the V4 report cut the top-k selector’s time in half by moving its scores from FP32 to BF16, “while preserving a 99.7% recall rate of KV entries”.
Expert multiplications
An MoE layer sends each token to of its experts, so the layer’s product splits into small products of different heights, which a grouped GEMM (general matrix multiplication) runs in one launch. DeepGEMM, DeepSeek’s library, offers two layouts. In the contiguous one, for prefill and training, each expert’s tokens sit back to back, padded to the kernel’s block height, and the CPU must know the counts. In the masked one, for decode, every expert gets a fixed-size slot and a mask says how many rows are real, because “when CUDA graph is enabled … the CPU is unaware of the number of tokens each expert receives” (graphs: see “Launches”).
For GLM-5.3 under TP8 a decode batch of 64 sequences touches about 222 of the 256 experts, with , 2.3 rows each on average (uniform routing, compute.py §A; real routing is skewed and under TP reads fewer experts). A 128-row tile then holds 2 useful rows, and the padding costs no time. Each weight is used once per row, one multiply and one add, so rows do FLOPs per weight; the token rows are small next to the weights, so the product does FLOPs per weight byte ( for FP8) and stays below the ridge of 412 until reaches about 206. Padded to 128 rows it does 256 FLOPs per byte, still below 412, so the arithmetic finishes before the weight bytes arrive (compute.py §B). Padding first costs time when a product needs a second tile, from 129 rows. With 2.3 rows the product uses about 1% of the tensor cores’ peak.
- bandwidth cap r/206: the weight read sets the time
- tile utilisation, rows padded to 128
- share reached, the lower of the two
- time lost to padding
- GLM-5.3 decode under TP8
A model of an ideal kernel, not a measurement: one expert's FP8 weights streamed once at 4.8 TB/s, r rows padded up to a multiple of 128 (an assumed tile height) and multiplied at 1,979 TFLOP/s. Up to 128 rows the padded tile's arithmetic is shorter than the weight read, so padding is free; from 129 rows on, every r that is not a multiple of 128 loses time to it. Decode rows assume uniform routing, 8 of 256 experts per token, for 1 to 73 sequences (the TP8 cache pool at 8K).
So a decode expert kernel is judged on weight bandwidth, and a tile-shaped grouped GEMM falls short of it too. Kimi K3’s report: “at small batch sizes, the group GEMMs reduce to memory-bound streaming of weight matrices — a regime for which conventional tile-centric kernels are poorly suited”. Moonshot replaced them with a token-centric kernel in which each warp computes one output value and streams its weights straight from memory, laid out offline.
Weight-only quantization (I1) pays off in this regime. Marlin, the W4A16 kernel vLLM and SGLang have used on Hopper, keeps 4-bit weights in memory and expands them in registers just before the multiplication; its authors report “close to ideal (4x) speedups up to batchsizes of 16-32 tokens” on an A10, falling towards 1.5× at 128 “as the problem becomes compute-bound”. For an MoE the batch that counts is rows per expert.
At prefill the regime moves to the knee. A chunk of 8,192 tokens gives each expert 256 rows on average, past the ridge by weight bytes alone; but under TP8 each GPU holds a slice only 256 columns wide, the activations it reads and writes outweigh the weights, and counting them the product stays just below the FP8 ridge (about 250 FLOPs per byte, the figure’s prefill mark). There tile utilization matters too: DeepGEMM’s README discusses block sizes that keep “112 out of 132 SMs” busy with 128-wide blocks and all 132 with 112-wide ones. Tuning is per shape and per GPU: vLLM’s Triton MoE kernel reads a configuration file named by expert count, width, GPU and data type, mapping “an irregular grid of batch sizes” to tile settings, and runs on defaults without one.
Kernels that move tokens between GPUs
With expert parallelism (EP) each GPU holds a subset of the experts, and every MoE layer sends each token to the GPUs owning its experts (dispatch) and gathers the results back (combine). I3 derives when this exchange can hide behind the expert products; here the question is what the kernels do.
DeepEP, DeepSeek’s all-to-all library, shipped two kernel families. The normal kernels, for training and prefill, forward data from NVLink (the GPUs’ link inside a node) to RDMA (direct transfers between nodes’ memories) at high throughput. The low-latency kernels, for decode, were RDMA-only at release and receive through a hook that “does not occupy any SM resource”. The split mattered for graphs: the normal dispatch waits on the CPU for the counts it must allocate, which “is not compatible with CUDA graph”, while the low-latency kernels preallocate fixed buffers and can be captured. DeepEP’s second version (April 2026) merged the two into one interface that uses fewer SMs, keeping the choice as a flag (“CUDA graph compatible, or faster CPU sync in saving memory mode”), and version 2.5 removed version 1. SGLang’s DeepSeek-V4 cookbook still sets --deepep-mode normal on prefill servers and low_latency on decode servers.
The other route keeps the exchange inside the GPUs’ shared address space. vLLM’s default EP exchange was an all-gather of every token to every rank and a reduce-scatter back, “intrinsically wasteful” in NVIDIA’s words, since each rank needs only its routed tokens. NVIDIA measured the exchange at 15% to 20% of runtime for Nemotron 3 Ultra on prefill-heavy work and adopted FlashInfer’s NVLink one-sided all-to-all, in which each GPU writes its tokens directly into the receiver’s memory, for “roughly 5%” end to end on a GB200 NVL72 rack.
The furthest step fuses the exchange into the expert kernel. DeepSeek’s MegaMoE, in DeepGEMM, runs dispatch, the two expert matrices, the activation and combine as “a single pipelined kernel” in which “computation of current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently”, the software pipelining distributed linear algebra has long used. The DeepSeek-V4 report measures “1.50∼1.73×” over “strong non-fused baselines” for general inference and up to 1.96× for latency-sensitive work such as RL rollouts. FlashInfer added fused MoE kernels in CuTe DSL for Blackwell in July 2026, and the vLLM recipe for Mistral Large 4 uses one on the prefill side of a disaggregated preset: on 4 + 4 GB300s, 241 against 206 output tokens per second per GPU at 32K-token inputs and a floor of 40 tokens per second per user (+17%, computed), 154 against 149 at a floor of 100.
The fused kernel has costs as plain as its gains. MegaMoE needs expert parallelism over NVLink with symmetric memory on every GPU (the Kimi K3 recipe: “not compatible with cross-node RDMA”), and the DeepSeek-V4-Pro recipe tells aggregated tensor-parallel serving not to pick an EP MoE backend. It is cut for formats: DeepGEMM’s first Mega MoE supported “Only FP8 x FP4 MoE”, and vLLM’s deep_gemm_mega_moe is “FP8-only”, with a separate variant for V4-Pro’s mixed checkpoint. FlashInfer’s Hopper MegaMoE needs the token count to “be equal on every rank”, so short ranks are padded. The Mistral Large 4 recipe uses it on decode servers only “with --max-num-batched-tokens below 512”; at 8,192 it “lost 10% at 8k”. On an expert-parallel decode preset at 8K-token inputs, MegaMoE on both sides gave 766 against 723 tokens per second per GPU at the floor of 40 (+6%), and 98 against 178 at the floor of 100 (−45%, computed).
Launches and CUDA graphs
A kernel launch has two costs. The CPU has to issue it (in eager PyTorch: Python, PyTorch’s dispatcher and the driver), a cost none of our sources measures for these models. And the GPU sits idle between one kernel’s end and the next one’s start: Hazy Research timed that gap on an H100 at about 2.1 µs for kernels queued on a stream and about 1.3 µs inside a CUDA graph, “time spent with the GPU doing no useful work”. No source counts GLM-5.3’s kernels, but vLLM measured Kimi K3, a large MoE mixing MLA and linear (KDA) attention layers, at 23 kernels per layer on its stock path: 1,817 kernels for GLM-5.3’s 78 layers and MTP layer, about 1,840 with an assumed 20 for the embedding, output layer and sampling (our estimate, compute.py §C). At batch 1 the floor of 1,381 µs is 0.75 µs of GPU work per kernel, less than the gap after it.
Counting each kernel’s work plus the gap after it (the serialized model used from here on), an eager step at batch 1 takes about 5,240 µs, three quarters of it gaps. A CPU running ahead of the GPU hides its own issue cost, not the GPU’s gap: with kernels of GPU time (each with its gap) and µs to issue each, the step lies between and .
A CUDA graph records a sequence of kernels with their arguments once and replays it with a single launch from the CPU, which removes the CPU’s issue cost and shortens the GPU’s gap. Everything is frozen at capture: tensor shapes, memory addresses, the number of thread blocks. Engines therefore capture one graph per batch size (vLLM’s older default: 1, 2, 4, then every 8 up to 256 and every 16 up to 512) and pad each batch to the nearest captured size. Padding is free for the dense products, and the engines keep pad tokens out of the experts: vLLM sets their expert ids to −1 so supporting MoE kernels drop them (VLLM_MOE_SKIP_PADDING, on by default), and SGLang’s MoE routing skips “much of the padded region”. Graphs also hold memory: for GLM-5.2, SGLang measured 2.4 GB for 42 prefill shapes, yet the total came out 1.10 GB below running without graphs, because capturing up to the chunk size removes a per-request activation peak.
- CPU launching a kernel
- GPU idle between kernels
- dense products
- routed experts
- sparse attention
- indexer
| whole step, B = 1 | µs | below eager |
|---|---|---|
| eager | 5,239 | – |
| piecewise graph | 3,896 | 25.6% |
| full graph | 3,769 | 28.1% |
| fewer kernels (13 per layer) | 2,742 | 47.7% |
GPU work alone (per-operation roofline): 1,381 µs.
A model with stated inputs, not a trace. Each kernel does its roofline work and is followed by a gap in which the GPU does nothing useful: 2.1 µs launched from a CUDA stream, 1.3 µs inside a CUDA graph (Hazy Research's H100 measurement); 5 µs is Hazy's estimate of a typical stall. Kernels per step: 23 per layer (vLLM's count for Kimi K3, assumed for GLM-5.3) × 79 layers + 20 = 1,837; with 13 per layer, 1,047. The piecewise graph runs the indexer and attention kernels of every layer outside the graph (an assumption). The kernel list and the split of each layer's work among its kernels are ours; kernels that only move activations (norms, rope, all-reduce) carry no work in the roofline and show as slivers. The CPU is assumed not to run ahead of the GPU.
Not every kernel can be frozen. Attention kernels often plan their work on the CPU from the current sequence lengths, which change every step. Engines have three answers. A full graph captures the attention too, which needs a kernel whose launch configuration doesn’t depend on the lengths; FlashInfer, for instance, copies each step’s plan into a fixed buffer read by a kernel that stays resident with a fixed number of thread blocks. A piecewise graph leaves attention out: vLLM’s compiler (next section) splits the model at every attention call, captures the pieces between, and runs attention eagerly, so a layer costs one graph replay plus the eager launches of its attention, cache write and indexer. A breakable graph, introduced in SGLang in February 2026 and adopted by vLLM, captures the whole forward in one pass, ending the capture at each incompatible operation, running it eagerly and resuming, with no compiler involved; SGLang uses it for prefill by default and measured prefill-only steps 1.60× faster than eager on GLM-5.2.
vLLM’s default mode, FULL_AND_PIECEWISE, uses full graphs for uniform decode batches (every sequence with the same query length) and piecewise graphs for anything with prefill in it. FULL_DECODE_ONLY drops the piecewise graphs to save their memory and leaves prefill and mixed steps eager; DeepSeek-V4-Pro’s vLLM recipe uses it even for aggregated serving on 8×H200. Speculative decoding changes the shape: a step that verifies drafted tokens has query positions per sequence, so the V4-Pro recipe’s GB200/GB300 section asks for “explicit capture sizes n*(K+1) for n=1..S” (its is our , the sequences per engine); vLLM’s newer model runner derives these sizes itself. Hybrid models add state: each sequence in a full decode graph needs its own block of recurrent (Mamba) state, so the state blocks cap the batch a graph can hold, and current vLLM refuses to start when the capture size exceeds them. For GLM-5.3, current vLLM (commit of 2026-11-11) uses breakable graphs by default, and its sparse MLA backend supports full graphs for uniform decode batches, speculative ones included, so a decode step can replay as one graph.
A graph shortens the gaps but keeps one per kernel. In the serialized model a full graph cuts the batch-1 eager step from 5,239 to 3,769 µs (28%), and the 2,388 µs of gaps left still exceed the 1,381 µs of work; at batch 64 it saves 6.5% (compute.py §H5). Only fewer kernels remove gaps: at 13 kernels per layer the batch-1 graph step falls to 2,742 µs. Async scheduling (vLLM) and the overlap scheduler (SGLang), both on by default, take the CPU’s share off the path by preparing the next batch while the GPU runs the current one. A persistent kernel goes furthest: it stays resident and pulls work from a queue instead of being launched per operation. Hazy Research’s megakernel runs a whole forward pass of a 1B model as one kernel, written for that model and that GPU. vLLM’s specialized decode for Kimi K3 cuts a layer from 23 kernels to 13 with fusion, small-matrix kernels and overlap with communication, from 130 µs to 69 µs per layer on a GB300 (the 10 fewer gaps are some 13 to 21 µs of that, our estimate): 1.58× the tokens per second at batch 1 on eight GPUs, 1.31× at batch 64.
Compilation
Hand-written kernels cover the large operations. For the rest, vLLM compiles the model with torch.compile: Dynamo traces the forward pass into a graph, with attention wrapped as one opaque custom operation, and Inductor generates fused Triton kernels for the pieces between attention calls. vLLM’s own CUDA versions of RMSNorm, activations and the like are switched off where Inductor’s are faster (on NVIDIA “Inductor actually generates a faster fused kernel than our custom CUDA kernel” for RMSNorm with FP8 rounding), and pattern-matching passes fuse across the custom kernels that remain and into collectives, as with the all-reduce and the following RMSNorm; “any fusion involving custom ops (attention, collectives, sub-byte quantization) continues to require custom passes”. The compiled graph is also what piecewise CUDA graphs are cut from.
Compilation runs at startup and is cached on disk; SGLang measured torch.compile at 78–86% of the time spent preparing prefill graphs, 158 seconds on GLM-5.2. Kernel libraries compile just in time too: a cold start of NVIDIA’s RL stack on more than 1,000 GPUs spent 38.8 minutes compiling kernels, 28 of them FlashInfer’s; with those baked into the container image and the other caches warm, that phase took 0.4 minutes.
On NVIDIA GPUs vLLM doesn’t compile several of the newest architectures at all. Breakable graphs set the compilation mode to none, and the architectures using them by default include GLM-5.3, DeepSeek-V4 and V4.1, Kimi K3, MiniMax-M3 and Step-5. In the worked example, then, every elementwise operation runs as vLLM’s own custom kernel or a plain PyTorch kernel, unfused, and the fusions that matter are written by hand into the model’s kernels, as DeepSeek’s fused attention kernel and Mega-mHC are. On AMD GPUs the default list is shorter (“Breakable CUDA graphs currently regress performance on ROCm”), and the recipes compile the same models, except DeepSeek-V4.1, which “does not support torch.compile”, and run AITER, AMD’s kernel library. SGLang leaves torch.compile off by default.
Determinism: the same token whatever else is in the batch
Run the same prompt at temperature 0 twice on a busy server and the outputs can differ. Thinking Machines sampled Qwen3-235B 1,000 times at temperature 0 and got 80 different completions. Their explanation: most inference kernels use no atomic additions, so a run on the same batch gives the same bits, but a kernel processes a request differently depending on the batch around it. RMSNorm, matrix products and attention all reduce (sum) over a dimension, and at small batch a fast kernel splits that sum across more SMs (split-K for products, split-KV for attention). Floating-point addition isn’t associative, so a different batch, through a different split, gives a slightly different sum. What’s missing is batch invariance: the output for a request is bitwise the same in every batch that contains it. Across GPUs the all-reduce and the expert combine are sums too, and kernels that accumulate with atomics, such as some expert combines, aren’t even repeatable on one batch.
The fix is a reduction order that doesn’t depend on the batch: no split-K, a fixed tiling along the summed dimension, and attention split into chunks of fixed size rather than a fixed number of chunks; one kernel configuration for all shapes is the simplest way. Compensated (Kahan) or pairwise summation makes the sum more accurate but still order-dependent, so it doesn’t help. vLLM’s batch-invariant mode also keeps its own all-reduce “with a fixed reduction order” and turns off the faster all-reduce kernels. It costs speed: Thinking Machines’ batch-invariant matrix product was about 20% slower than cuBLAS, and a Qwen3-8B run took 42 s instead of 26 s; SGLang’s deterministic mode measured slowdowns of 25% to 45% (average 34%) on Qwen3-8B on an H200.
A serving stack pays this for RL (S5): if the rollout engine and the trainer compute bitwise the same probabilities, the policy (the model being trained) that samples and the one trained are the same function, which needs batch-invariant kernels in the engine and the same reduction orders in the trainer. Of the 14 labs we searched (13 through their own documents, Tencent only through a vLLM recipe), one states batch-invariant inference kernels. DeepSeek-V4 replaced cuBLAS “end-to-end” with DeepGEMM, abandoned split-K “in most scenarios”, and gave up split-KV in decode attention, which leaves the last partly filled wave of sequences on few SMs. Its answer is a pair of decode kernels: one computes a whole sequence on one SM, the other spreads the last wave’s sequences across several SMs with “the same” accumulation order, so both give identical bits; the report calls the overhead “negligible” without a number. Aleph Alpha states the opposite for Kolibri’s RL: “none of the kernels are batch-invariant”. DeepGEMM’s kernels were “fully deterministic and batch-invariant” until September 2026, when a switch made that opt-in “so some kernels can run faster”.
The kernel families side by side
| Kernel family | Cost term it moves | Phase where it pays | What it is cut for | Examples |
|---|---|---|---|---|
| Fusion of elementwise chains | HBM writes and rereads of intermediates; launches | prefill (bytes), decode (launches) | one chain of operations, layout, format | RMSNorm + FP8 rounding; Mega-mHC |
| FlashAttention family | the score traffic | prefill and training | head width, GPU generation (FA3 Hopper, FA4 Blackwell) | FlashAttention 2, 3, 4 |
| Latent and sparse decode attention | cache bytes per step, conversion cycles, or padded-head FLOPs (TP, speculation) | decode | cache layout (656, 584, 528, 288 B per token), heads per GPU | FlashMLA, FlashInfer MLA, TOKENSPEED_MLA |
| Grouped expert products | weight bandwidth at decode, tensor-core use at prefill | both, differently | expert shape, format, GPU, rows per expert | DeepGEMM, Marlin, Triton MoE, token-centric decode |
| All-to-all kernels | exchange time, SMs taken from compute | EP serving | interconnect (NVLink or RDMA), buffer sizes | DeepEP, NVLink one-sided |
| Exchange fused into experts | exposed exchange time | EP prefill and decode | NVLink domain, symmetric memory, tokens per rank | MegaMoE (DeepGEMM, FlashInfer) |
| CUDA graphs, persistent kernels | CPU issue time (graphs); GPU gaps per kernel (shorter in graphs, fewer with fused or persistent kernels) | small-batch decode | batch sizes, memory addresses; megakernels one model | full, piecewise, breakable graphs; megakernels |
| Batch-invariant kernels | none; costs speed | RL rollouts, debugging | fixed reduction order | DeepSeek-V4, SGLang and vLLM modes |
Where the models in the map sit
Kernel work named in each lab’s own documents, and the kernel flags its engine recipes set (vendor recipes marked as such). A blank means we searched and found nothing.
| Model | Lab’s own documents | Recipe flags (vendor) |
|---|---|---|
| DeepSeek-V4 / V4.1 | MegaMoE; TileLang kernels replacing “the vast majority” of PyTorch operators; batch-invariant kernels; V4.1: fused attention kernel, Mega-mHC, 11 kernels per decode layer | vLLM V4-Pro on 8×H200: "cudagraph_mode": "FULL_DECODE_ONLY", --block-size 256; Blackwell with EP: deep_gemm_amxf4_mega_moe (V4-Pro), deep_gemm_mega_moe (V4.1); SGLang Hopper: Marlin W4A16 or FlashInfer W4A8 experts |
| Kimi K3 | FlashKDA (training and prefill); fused KDA decode kernel replaying inputs for speculative verification; AttnRes merge fused into the all-reduce; token-centric expert decode kernel | vLLM Blackwell: --attention-backend TOKENSPEED_MLA; NVLink EP: flashinfer_nvlink_one_sided all-to-all; SGLang Hopper: Marlin experts, FlashMLA decode |
| Mistral Large 4 | context-parallel collectives with in-kernel overlap: one index-producing layer’s decode step 240 to 153 µs on GB300 | vLLM: dedicated image; breakable graphs and FlashMLA sparse by default; MegaMoE (flashinfer_moe_ep_cutedsl) on EP prefill |
| Nemotron 3 Ultra / 3.5 | NVLink one-sided all-to-all (about 5%); weights padded at load for kernel alignment; 3.5: Mamba state replay for verification, multi-wave KV split | Ultra model card (NVIDIA’s own): Blackwell only for now; SGLang --disable-piecewise-cuda-graph |
| GLM-5.3 (GLM-5 report) | one quantization kernel bitwise identical in training and inference; deterministic top-k in RL | vLLM: breakable graphs, no compilation (engine default); SGLang: --cuda-graph-max-bs-decode twice the concurrency |
| MiniMax-M3 | sparse-attention kernels: a top-k without exponentials, a persistent grid over key blocks | vLLM: --block-size 128 “mandatory” |
| Step-5-Preview | engine commands in StepFun’s README: SGLang FlashAttention-4 and MegaMoE on B300, FlashAttention-3 and Triton MoE on H200; MegaMoE needs expert parallelism | vLLM recipe: the same MegaMoE split |
| gpt-oss-120b | Triton MoE kernel with MXFP4 (a reference implementation, “not expected to be run in production”) | vLLM: Marlin MXFP4 on A100, FlashInfer TRT-LLM MXFP4 on B200 |
Where one model’s documents cover both GPU generations, the kernels split by generation: for DeepSeek-V4, Kimi K3 (in SGLang) and Step-5-Preview, Blackwell gets MegaMoE, FlashAttention-4 or TRT-LLM MLA and Hopper gets Triton, Marlin, FlashAttention-3 or FlashMLA, and gpt-oss makes the same split between B200 and A100. They split by vendor too: the AMD cells of the same recipes run AITER kernels and, except for DeepSeek-V4.1, compile the model. MegaMoE requires expert parallelism in every recipe that explains it.
Easy to get wrong
- “Decode attention is memory-bound.” With MLA it depends on how many query heads share each cached entry on a GPU, how many positions are verified at once and how the kernel pads heads. Under TP8 GLM-5.3’s useful work is 26.5 FLOPs per byte, but vLLM’s kernel computes 64 padded heads and sits at the BF16 ridge, and DeepSeek’s decode with 128 heads is compute-bound by FlashMLA’s own analysis.
- “Few rows per expert waste the tensor cores, so expert kernels are slow at decode.” The tensor cores are mostly idle, but the time is set by reading the weights. A decode expert kernel should be compared with the bandwidth roof, not with peak FLOP/s.
- “CUDA graphs remove launch overhead.” They remove the CPU’s issue cost and shorten the GPU’s gap between kernels (2.1 to 1.3 µs in Hazy Research’s measurement), and the kernels inside run as before; SGLang says of its breakable graphs that they do “not reduce model FLOPs or make compute-bound kernels cheaper”. At batch 1 the gaps left inside a graph still exceed GLM-5.3’s work; only fewer kernels remove them.
- “Deterministic” and “batch-invariant” are the same thing. A kernel run twice on the same batch usually gives the same bits; what changes the output is a different batch around the request.
- “vLLM compiles every model.” On NVIDIA GPUs it runs GLM-5.3, DeepSeek-V4, Kimi K3 and others with breakable graphs and compilation off; their fusions come from hand-written kernels.
Check yourself
- A decode step of GLM-5.3 at batch 8 under TP8 has a floor of 5,216 µs by the per-operation roofline, spread over about 1,840 kernels. In the serialized model, with a GPU gap of 2.1 µs after each kernel run eagerly and 1.3 µs inside a CUDA graph, how long are the eager and the full-graph steps? Why does a path with 13 kernels per layer instead of 23 still help inside a graph?
Answer
Eager: µs. Full graph: µs, about 16% less; the 2,390 µs of gaps left are almost half the work. The gaps scale with the number of kernels, so 13 per layer (about 1,050 kernels) leaves some 1,360 µs of gaps and a step of about 6,580 µs, 28% below eager (computed, compute.py §H5, with the article’s assumptions). The CPU’s issue cost comes on top in eager mode wherever it exceeds a kernel’s time.
- An expert product reads one FP8 expert slice of bytes and multiplies it by token rows. On an H200 (4.8 TB/s, 1,979 TFLOP/s FP8), at what does it stop waiting on memory, and why doesn’t padding rows to a 128-row tile slow it down?
Answer
It does FLOPs per weight byte, so it reaches the ridge of at . At padded to 128 rows it computes FLOPs per byte, still below 412, so the padded arithmetic finishes before the weight bytes arrive. Time is TB/s either way.
- Why can’t a decode attention kernel that splits each sequence’s cache into “as many pieces as fill the GPU” be batch-invariant, and what does DeepSeek-V4 do instead?
Answer
The number of pieces depends on how many other sequences share the step, and each piece’s partial result is merged with the online-softmax rescaling, a floating-point sum whose order changes with the number of pieces. DeepSeek-V4 gives up the split: one kernel processes a whole sequence on one SM, and a second kernel, for the last partly filled wave, spreads a sequence over several SMs but accumulates in the same order, so both give the same bits.
In the map
Methods
- Kernel fusion 2 sources
- FlashAttention 2 sources
- FlashAttention-3 2 sources
- FlashAttention-4 1 source
- FLASHMLA_SPARSE 1 source
- FlashInfer 2 sources
- DeepEP 4 sources
- MegaMoE 2 sources
- FlashInfer NVLinkOneSided all-to-all 2 sources
- Triton MoE kernel 1 source
- Marlin MoE backend 1 source
- WarpDecode token-centric routed-expert decoding 1 source
- CUDA Graph capture and replay 4 sources
- Breakable CUDA graphs 1 source
- CUDA graph capture size reduction 2 sources
- Operator fusion using torch.compile 1 source
- Persistent kernel 1 source
- Batch-invariant deterministic kernels 2 sources
- Dual-kernel batch-invariant attention decoding 1 source
- Mega-mHC 1 source
- CUTEDSL 1 source
- Triton 1 source
Taxonomy nodes
- inference kernel 59 methods
- kernel & quantization library 25 methods
Further reading
- Dao et al., FlashAttention (2022): the IO analysis and the lower bound; FlashAttention-2 (2023) and -3 (2024), and Zadouri et al.’s FlashAttention-4 (2026), for how each GPU generation changed the kernel.
- FlashMLA,
docs/on GitHub: the 2025-04-22 deep dive (why MLA decode is compute-bound) and the 2025-09-29 one (the FP8 sparse kernel and dequantization-bound decode). - DeepGEMM and DeepEP READMEs and release PRs: grouped-GEMM layouts, Mega MoE benchmarks, the two DeepEP generations.
- DeepSeek-V4 report §3 (fused EP kernel, TileLang, batch-invariant kernels) and the V4.1 report §2.4.1 (Mega-mHC’s traffic count).
- Kimi K3 report §5.4.2: decode kernels for KDA, attention residuals and latent MoE.
- vLLM design docs
cuda_graphs.md,torch_compile.mdandfusions.md; SGLang’s “Advanced CUDA Graph Techniques” post (August 2026) for breakable graphs. - Horace He and Thinking Machines, “Defeating Nondeterminism in LLM Inference” (2025): batch invariance kernel by kernel.
- Frantar et al., MARLIN (2024): why weight-only quantization pays up to a batch of about 32.
- Hazy Research, “Look Ma, No Bubbles” (2025): launch overhead and a whole forward pass as one kernel.
