Open Models Almanac

Inference & serving · I2

Kernels and graph compilation: where the cost model meets the GPU

A kernel gets close to the roofline by keeping intermediate results on chip, streaming what the phase needs at full bandwidth and hiding launches and communication behind work, and each of those gains is cut for one shape, one format and one GPU generation

Illustration of a compact workbench where one coral ribbon passes through a press, rollers and a cutter in a row and comes out cut into the shapes of the moulds hanging above, far from a tall shelf of jars.

Serve GLM-5.3 on eight H200 GPUs to a single user, and each decode step has to read about 6.6 GB per GPU: 3.7 GB of weights that every token uses, 2.8 GB for the eight experts each layer picks, and 0.13 GB of cache. At the H200’s 4.8 TB/s that takes about 1.4 ms (computed below), the floor that 00 and 02 build their prices on. Run the same step without CUDA graphs, launching each GPU program separately, and the GPU runs about 1,840 of them, each followed by an idle gap of about 2.1 µs (Hazy Research’s measurement on an H100): some 3.9 ms of gaps, close to three times the work (our estimate, below).

The cost model assumes the GPU reaches its roofline. Whether it does is decided by kernels: whether attention streams the compressed cache at full bandwidth, whether an expert reads its weights as fast as memory allows when it sees one or two tokens, whether the exchange between GPUs stalls the step, and whether the GPU waits between operations. Serving recipes are full of these choices (--moe-backend deep_gemm_mega_moe, --attention-backend TOKENSPEED_MLA, cudagraph_mode: FULL_DECODE_ONLY). This article says which cost term each one moves.

What a kernel is, and where its data lives

A kernel is one program launched on the GPU: a matrix multiplication, a normalization, an attention computation. The CPU asks for it, the GPU runs it across its streaming multiprocessors (SMs, 132 on an H100 and 148 on a B200), and the result lands in high-bandwidth memory (HBM), the GPU’s main memory. The next kernel reads it from there. A kernel’s work is cut into thread blocks, each running on one SM, and the threads of a block run in groups of 32 called warps.

Three kinds of memory matter. HBM is large and comparatively slow: an H200 holds 141 GB and reads 4.8 TB/s. Each SM has a small, fast scratchpad, shared memory, of 228 KB on Hopper and Blackwell, plus registers. Data in shared memory or registers costs far less to reread than data in HBM, though shared memory has its own bandwidth limit, which on Blackwell becomes a bottleneck for attention (below). Matrix products run on tensor cores, units inside each SM; a thread block multiplies tiles of 64 or 128 rows.

00 gave the roofline: an operation that does FF floating-point operations (FLOPs) on MM bytes of HBM traffic has arithmetic intensity F/MF/M. Below the ridge, peak FLOP/s divided by bandwidth, the operation waits on memory and takes M/bandwidthM/\text{bandwidth}; above it, it waits on arithmetic and takes F/peakF/\text{peak}. For an H200 with dense FP8 at 1,979 TFLOP/s the ridge is 412 FLOPs per byte; for BF16 at 989 TFLOP/s it is 206 (computed from NVIDIA’s datasheet, whose tensor-core figures are quoted “with sparsity” and halved here).

00 applied this to a whole forward pass. A step is a sequence of kernels, each at its own point on the roofline, and when they run one after another the step takes at least the sum of each one’s own bound, ∑imax⁡(Mi/bandwidth, Fi/peak)\sum_i \max(M_i/\text{bandwidth},\ F_i/\text{peak}), plus the gaps between them. A single max over the whole step would let a compute-bound kernel hide behind a memory-bound one (off by up to a factor of two). The sum is a floor: every kernel at full bandwidth or at peak, overlapping only where the stack arranges it.

Which roof each decode kernel of GLM-5.3 faces

Take the worked example from 02: GLM-5.3, a 753B-parameter mixture-of-experts (MoE) model, served in FP8 with tensor parallelism over eight H200s (TP8: most weight matrices split eight ways, every GPU holding a slice of all 256 experts; the small latent projections and the cache are copied to every GPU). Its attention is multi-head latent attention (MLA) with a sparse indexer: each layer keeps one compressed latent vector per token, 656 bytes in FP8, and an indexer picks 2,048 of the cached tokens for each new query. Leave speculative decoding aside for now (one new token per sequence per step), group the decode step’s kernels into five classes and compute each class’s intensity per GPU at 8K context (compute.py §A):

ClassWhat it readsFLOPs per byte, batch 1batch 64
Dense matrix products (attention projections, shared expert, router, output layer)3.7 GB of weights per GPU1.9120
Routed expertsthe slices of every expert some token picked2.04.6
Sparse latent attention2,048 latent entries per layer per sequence26.5 useful, 212 as runsame
Indexer scoringan indexer key for every cached token, in 21 layers7.87.8
All-reduce between GPUstwo per layer, over NVLinklatency-bound, not modelled

The classes that read weights sit far below the ridge at every batch that fits: under TP8 the KV cache pool holds 73 sequences of 8K tokens (02), and at 64 the dense products reach 120 FLOPs per byte against a ridge of 412. They gain from batching because every token reuses the same weights. The routed experts barely gain, because 64 tokens spread over about 222 experts give each only 2.3 rows (Expert multiplications, below). The step’s floor is the sum of the classes’ times: 1,381 µs at batch 1 and 18,907 µs at batch 64 (computed), and at 64 the routed experts are 87% of it.

Two costs lie outside that floor. Real kernels this small don’t stream at full bandwidth (below). And the 156 all-reduces per step carry a few kilobytes each at batch 1, so they cost their latency, which no source we saved prices; it is why vLLM’s largest listed fusion is the one into the all-reduce.

  • dense products
  • routed experts
  • sparse latent attention
  • indexer
  • attention, useful work of 8 heads (hollow)
  • prefill chunk (hollow)
  • FP8 roof
  • BF16 roof
1101001,00010,0001101001,000FLOPs per byte (log scale)TFLOP/s per GPU (log scale)FP8 1,979BF16 989ridge 412206dense, prefilldense 1.9experts 2.0indexer 7.8useful 26.5experts, prefillattention, padded 2121101001,00010,0001101001,000arithmetic intensity, FLOPs per HBM byte (log scale)attainable TFLOP/s per GPU (log scale)FP8 1,979BF16 989ridge 412206HBM 4.8 TB/s × intensitydense, prefill 2,048dense products 1.9routed experts 2.0indexer 7.8attention, useful (8 heads) 26.5experts, prefillattention, as run (64 heads) 212
decode, B = 1FLOP/Bµsbound by
dense products1.9764bytes
routed experts2.0590bytes
sparse latent attention212 as run (useful 26.5)22.5FLOPs
indexer7.84.73bytes
step (sum)1,381 µs

Prefill chunk of 2,048 tokens: dense products 806 FLOP/B (compute roof), routed experts 101 (bandwidth roof), counting activation reads and writes.

Computed, not measured: per GPU under TP8, FP8 weights and KV cache, 8K context, ideal kernels. Each class takes the longer of its bytes at 4.8 TB/s and its arithmetic at the dense peak it uses (FP8 1,979 TFLOP/s; attention in BF16, 989). Decode counts weight and cache bytes only; experts touched assume uniform routing, 8 of 256 per token. vLLM pads GLM-5.3's 8 attention heads per GPU to 64 in its FP8 sparse decode kernel, so the attention kernel does 8× the useful arithmetic; its time here is that of the padded kernel. "Verify 6" is a step of MTP with five drafts, as GLM-5.3's 8×H200 vLLM recipe serves it: six positions per sequence. Prefill marks also count activation reads and writes and leave out the output layer.

Figure 1.Kernel classes of one GLM-5.3 decode step on eight H200s (computed, ideal kernels). The classes that read weights wait on memory; the attention kernel as vLLM runs it, padded to 64 heads, sits at the BF16 ridge, and a prefill chunk of a few hundred tokens moves the dense products past the FP8 ridge.

So the question for most decode kernels is how close they get to 4.8 TB/s. The average kernel of the batch-1 step reads about 3.6 MB, under a microsecond at full bandwidth, and in that time it has to start, fill the memory pipeline with requests and drain it. On batch-1 decode of Llama-3.2-1B on an H100, Hazy Research found that vLLM and SGLang “are only able to use at most 50% of available GPU bandwidth”; DeepGEMM’s table shows dense FP8 products with 64 rows moving 1.7 to 2.7 TB/s, half to four fifths of an H800’s 3.35 TB/s.

The attention kernel is the exception. With MLA every query head of a sequence reads the same latent entry, so per selected key it does 2 hGPU sq (2dc+dR)2\,h_{\text{GPU}}\,s_q\,(2d_c + d_R) FLOPs on 656 bytes, with hGPUh_{\text{GPU}} the query heads on one GPU, sqs_q the query positions computed at once, dc=512d_c = 512 latent and dR=64d_R = 64 position dimensions. FlashMLA’s documentation puts the crossover at hqsq≥128h_q s_q \ge 128 for a BF16 cache on an H800 (about 62 for the 656-byte FP8 entry on an H200) (hqh_q is its hGPUh_{\text{GPU}}) and notes that DeepSeek decodes without tensor parallelism, so ”hqh_q is 128 and the kernel is compute-bound”. Under TP8 a GPU of GLM-5.3 holds 8 of the 64 heads: 26.5 useful FLOPs per byte. But vLLM’s FP8 sparse kernel “only supports h_q = 64 or 128”, so it pads the 8 heads to 64 and computes 212 FLOPs per byte, at the BF16 ridge, discarding seven eighths of the arithmetic. SGLang pads the same way for DeepSeek-V4, reasoning that “MLA decode is typically memory-bandwidth-bound on KV reads”; here, at 212 against 206, it is nearly a tie.

Speculative decoding (I5) moves every class. The GLM-5.3 recipe for 8×H200 drafts five tokens with its MTP layer, so a step verifies six positions per sequence. With dense MLA the six would share their entries and the attention’s intensity would rise six times; with the sparse indexer each position selects its own 2,048, so it rises only by the overlap of the selections, from 212 to at most 1,274 (compute.py §H2). Six tokens also touch about 44 experts per layer instead of 8, and the batch-1 floor rises from 1,381 to 4,178 µs, for up to six tokens (computed).

Fusion: read once, write once

Most operations between the large matrix products are elementwise or row-wise: normalization, an activation, rounding to FP8. Each does a few FLOPs per element. Run as separate kernels, every one reads its input from HBM and writes its output back, and the next kernel reads that output again. Fusion merges a chain of such operations, often into the epilogue (the final step that writes the output) of the matrix product before or after them, so the intermediate stays in registers or shared memory. It is loop fusion, an old compiler transformation, and CNN libraries fused convolution, bias and ReLU for the same reason; what is new are the targets, such as FP8 rounding with its scales, collectives and the labs’ own layers.

Take RMSNorm followed by rounding to FP8, before GLM-5.3’s first attention projections in prefill. Unfused, the normalization writes its BF16 output to HBM and the rounding kernel reads it back, 2 bytes each way per element of width d=6,144d = 6{,}144: 24,576 extra bytes per token, 5.1 ns at 4.8 TB/s. The projection that follows (6,144 inputs to 2,624 outputs, at FP8 peak) takes 16.3 ns per token, so the three-kernel chain takes 25.3 ns and the fused one 20.2 ns, about 20% less (computed, §H4); fusing SwiGLU with the rounding before an expert’s down projection saves about 10%. In the per-kernel sum, fusion deletes memory-bound kernels and their bytes and leaves the product’s term as it was. These are upper bounds, with the product at peak, and for GLM-5.3 vLLM runs none of its fusion passes (Compilation, below). vLLM lists the measured end-to-end gain of these two fusions as “1-4%”, and of fusing the all-reduce with the following RMSNorm as “5-20%”.

At decode the bytes saved are small: at batch 64 one MoE layer reads about 1.1 GB of weights per GPU and the fused activations save 3.7 MB of it, 0.34% (computed). At decode the gain is fewer kernels, for the reason in the opening (Launches, below).

The labs fuse much further. DeepSeek-V4.1 runs its hyper-connections (a widened residual stream mixed by small learned matrices) through one kernel, Mega-mHC, that combines “residual update, input mixing, and coefficient prediction” with the pre-norm and the FP8 conversion; with its other fused kernels, “each CSA2 Reuse Mode layer executes with only 15 kernels during prefill and 11 during decode”. Kimi K3 fuses its attention-residual merge and the following RMSNorm into the tensor-parallel all-reduce.

Attention kernels

FlashAttention: the score matrix never leaves the chip

Attention as written in a textbook computes the scores S=QK⊤S = QK^\top, a T×TT \times T matrix per head, then the softmax PP, then PVPV. A naive implementation writes SS and PP to HBM between three kernels. For a dense head of GLM-5.3’s width at a 128K-token prefill, the FP32 scores alone would be 68.7 GB, and at 8K the three kernels move 0.82 GB per head (computed, §D). GLM-5.3 itself attends to 2,048 selected entries in prefill too, and under TP8 vLLM runs its prefill through the same FP8 sparse kernel as decode; the dense head only shows the T2T^2 term that tiling removes.

FlashAttention (Dao et al., 2022) computes the same result in one kernel. It splits QQ, KK and VV into blocks that fit in shared memory and, for each block of query rows, walks through the key and value blocks, keeping a running output. The obstacle is the softmax, whose denominator needs every score of the row. The fix is the online softmax. Keep, for each query row, the largest score seen so far mm and the running sum ℓ\ell of exponentials relative to it, starting from m=−∞m = -\infty, ℓ=0\ell = 0, o=0o = 0. When a new block of scores sjs_j arrives:

m′=max⁡(m, max⁡jsj),ℓ′=em−m′ℓ+∑jesj−m′,o′=em−m′ o+∑jesj−m′ vjm' = \max\big(m,\ \max_j s_j\big),\qquad \ell' = e^{m - m'}\ell + \sum_j e^{s_j - m'},\qquad o' = e^{m - m'}\,o + \sum_j e^{s_j - m'}\, v_j

and at the end the output is o/ℓo/\ell. The factor em−m′e^{m-m'} rescales what was accumulated under the old maximum, so the result equals the full softmax exactly in real arithmetic (in double precision a random 64-key example differs by 2.8×10−172.8 \times 10^{-17}, compute.py §D), and the T×TT \times T matrix is never written. Two partial states (m,ℓ,o)(m, \ell, o) merge by the same rescaling in any order, so a walk over blocks, a split across SMs and any tree of merges agree in exact arithmetic and differ only in rounding, which matters under “Determinism”. The paper proves the kernel needs Θ(T2dh2/M)\Theta(T^2 d_h^2 / M) HBM accesses for shared memory of size MM, against Θ(Tdh+T2)\Theta(T d_h + T^2) for the standard form.

  • scores in this block
  • the new maximum
  • running maximum m
  • before this block
  • rescaled by e^(m − m')
  • after adding this block
Scores of one query row against 16 keys (toy numbers)block 1block 2block 3block 4m = 0.92After block 1 of 4first block: nothing to rescale yetl1.927o1−1.400o2−0.401o / l: (−0.727, −0.208)full softmax, all 16 keys: (0.177, −0.338)difference 0.903 (keys 5–16 not read yet)Scores of one query row against 16 keys (toy numbers)block 1block 2block 3block 4m = 0.92After block 1 of 4first block: nothing to rescale yetl1.927o1−1.400o2−0.401o / l: (−0.727, −0.208)full softmax, all 16 keys: (0.177, −0.338)difference 0.903 (keys 5–16 not read yet)

Block 1: m = 0.92, l = 1.927. Nothing has been summed before, so nothing is rescaled.

Toy numbers from a fixed seed, not a model's. After each block: m is the largest score so far, l the sum of e^(s − m) over the keys read, o the same sum weighted by each key's 2-dimensional value. Dividing o by l at the end gives the softmax-weighted average of the values, and no score is kept once its block has been added.

Figure 2.Online softmax on a toy row: each new block can raise the running maximum, which shrinks what was summed before by e^(m - m'). After the last block, o divided by l equals the full softmax output.

The kernels have been rewritten for each GPU generation. FlashAttention-2 reached “50-73% of the theoretical maximum FLOPs/s on A100” but about 35% on Hopper. FlashAttention-3 rebuilt it around Hopper’s asynchronous copy unit (TMA) and warp specialization, in which some warps only move data and others only compute, for “740 TFLOPs/s (75% utilization)” in FP16 on an H100. On Blackwell the tensor cores doubled while the exponential units did not, and “shared memory traffic and exponential operations now dominate execution time”; FlashAttention-4 (Zadouri et al.) adds a software exponential and conditional rescaling, reaching 1,613 TFLOP/s in BF16 on a B200 (71%). These are prefill-style, compute-bound benchmarks.

Decode has one query row per sequence, so the work per sequence is a single long walk over its cache. To keep all SMs busy at small batch, decode kernels split the cache of one sequence across several SMs and merge the partial outputs with the same rescaling rule (split-KV, also called flash-decoding), into a number of pieces that depends on the batch.

FlashMLA: latent caches, sparse selection, low precision

DeepSeek’s FlashMLA is the attention library for MLA. Its dense decode kernel reached “up to 3000 GB/s in memory-bound configuration and 660 TFLOPS in computation-bound configuration on H800 SXM5” (April 2025), about 90% of the H800’s 3.35 TB/s (computed). Later releases added kernels for the sparse attention of DeepSeek-V3.2 and V4, which read only the entries an indexer selects (the top 2,048 in V3.2, 1,024 in V4-Pro and 512 in V4-Flash, by their configs’ index_topk).

The sparse decode kernel shows how far a kernel is cut to one format. Its FP8 cache for V3.2 stores, per token, 512 FP8 values, four FP32 scales and the 64 position dimensions in BF16: 656 bytes. The arithmetic runs in BF16, so every selected entry is converted on the fly, and on Hopper that conversion became the limit: by DeepSeek’s count, for 64 query heads (what TP8’s padding produces), about 34 cycles of matrix work per cached token against “at least … ≈50 cycles for dequantizing each token”. Pairing two thread blocks on neighbouring SMs, each converting half the entries for both, took the kernel from 250 to 410 TFLOP/s on an H800 (128 heads, two query positions). V4 changed the layout to 584 bytes per token, V4.1 to 528, with an FP4 variant of 288 bytes.

Each layout comes with its kernel. On 2026-09-30 FlashMLA removed “support for the Hopper architecture and for earlier models (including DeepSeek V3 / V3.2 / V4.0)” and “changed the FP8 / FP4 KV cache format”; the current README requires a Blackwell GPU (compute capability SM100 or SM103) or Huawei’s Ascend 950 and gives sparse decode “up to 1024 TFlops on B200”. Hopper deployments, the worked example included, run on the engines’ own copies: vLLM vendors a FlashMLA build from before the drop, and vLLM’s FlashInfer sparse MLA backend covers FP8 caches on Hopper for models without position dimensions, such as GLM-5.3-Flash. The DeepSeek-V4 report states the co-design directly: the cache’s block size can be “any multiple of lcm(m, m′)” of its two compression ratios.

The indexer has its own kernels. It scores every cached token against each new query (in GLM-5.3, 32 heads of width 128 in 21 layers), and its reads grow with the context while the attention’s stay fixed at 2,048 entries: at 8K the indexer reads 22.7 MB per sequence per step against 104.8 MB for the selected latents, and from about 38K tokens it reads more than the attention it serves (computed, §F). DeepSeek’s scoring kernels live in DeepGEMM, which added an FP4 indexer in April 2026; the V4 report cut the top-k selector’s time in half by moving its scores from FP32 to BF16, “while preserving a 99.7% recall rate of KV entries”.

Expert multiplications

An MoE layer sends each token to kk of its EE experts, so the layer’s product splits into EE small products of different heights, which a grouped GEMM (general matrix multiplication) runs in one launch. DeepGEMM, DeepSeek’s library, offers two layouts. In the contiguous one, for prefill and training, each expert’s tokens sit back to back, padded to the kernel’s block height, and the CPU must know the counts. In the masked one, for decode, every expert gets a fixed-size slot and a mask says how many rows are real, because “when CUDA graph is enabled … the CPU is unaware of the number of tokens each expert receives” (graphs: see “Launches”).

For GLM-5.3 under TP8 a decode batch of 64 sequences touches about 222 of the 256 experts, E(1−(1−k/E)B)E(1-(1-k/E)^B) with k=8k = 8, 2.3 rows each on average (uniform routing, compute.py §A; real routing is skewed and under TP reads fewer experts). A 128-row tile then holds 2 useful rows, and the padding costs no time. Each weight is used once per row, one multiply and one add, so rr rows do 2r2r FLOPs per weight; the token rows are small next to the weights, so the product does 2r/b2r/b FLOPs per weight byte (b=1b = 1 for FP8) and stays below the ridge of 412 until rr reaches about 206. Padded to 128 rows it does 256 FLOPs per byte, still below 412, so the arithmetic finishes before the weight bytes arrive (compute.py §B). Padding first costs time when a product needs a second tile, from 129 rows. With 2.3 rows the product uses about 1% of the tensor cores’ peak.

Figure 3.Share of peak an ideal FP8 expert product reaches with r rows (a model, not a measurement). Decode sits at the far left, where the weight read sets the time; padding to 128-row tiles first costs time at 129 rows, and beyond about 206 rows, where the product is compute-bound, every partly filled tile does.

So a decode expert kernel is judged on weight bandwidth, and a tile-shaped grouped GEMM falls short of it too. Kimi K3’s report: “at small batch sizes, the group GEMMs reduce to memory-bound streaming of weight matrices — a regime for which conventional tile-centric kernels are poorly suited”. Moonshot replaced them with a token-centric kernel in which each warp computes one output value and streams its weights straight from memory, laid out offline.

Weight-only quantization (I1) pays off in this regime. Marlin, the W4A16 kernel vLLM and SGLang have used on Hopper, keeps 4-bit weights in memory and expands them in registers just before the multiplication; its authors report “close to ideal (4x) speedups up to batchsizes of 16-32 tokens” on an A10, falling towards 1.5× at 128 “as the problem becomes compute-bound”. For an MoE the batch that counts is rows per expert.

At prefill the regime moves to the knee. A chunk of 8,192 tokens gives each expert 256 rows on average, past the ridge by weight bytes alone; but under TP8 each GPU holds a slice only 256 columns wide, the activations it reads and writes outweigh the weights, and counting them the product stays just below the FP8 ridge (about 250 FLOPs per byte, the figure’s prefill mark). There tile utilization matters too: DeepGEMM’s README discusses block sizes that keep “112 out of 132 SMs” busy with 128-wide blocks and all 132 with 112-wide ones. Tuning is per shape and per GPU: vLLM’s Triton MoE kernel reads a configuration file named by expert count, width, GPU and data type, mapping “an irregular grid of batch sizes” to tile settings, and runs on defaults without one.

Kernels that move tokens between GPUs

With expert parallelism (EP) each GPU holds a subset of the experts, and every MoE layer sends each token to the GPUs owning its experts (dispatch) and gathers the results back (combine). I3 derives when this exchange can hide behind the expert products; here the question is what the kernels do.

DeepEP, DeepSeek’s all-to-all library, shipped two kernel families. The normal kernels, for training and prefill, forward data from NVLink (the GPUs’ link inside a node) to RDMA (direct transfers between nodes’ memories) at high throughput. The low-latency kernels, for decode, were RDMA-only at release and receive through a hook that “does not occupy any SM resource”. The split mattered for graphs: the normal dispatch waits on the CPU for the counts it must allocate, which “is not compatible with CUDA graph”, while the low-latency kernels preallocate fixed buffers and can be captured. DeepEP’s second version (April 2026) merged the two into one interface that uses fewer SMs, keeping the choice as a flag (“CUDA graph compatible, or faster CPU sync in saving memory mode”), and version 2.5 removed version 1. SGLang’s DeepSeek-V4 cookbook still sets --deepep-mode normal on prefill servers and low_latency on decode servers.

The other route keeps the exchange inside the GPUs’ shared address space. vLLM’s default EP exchange was an all-gather of every token to every rank and a reduce-scatter back, “intrinsically wasteful” in NVIDIA’s words, since each rank needs only its routed tokens. NVIDIA measured the exchange at 15% to 20% of runtime for Nemotron 3 Ultra on prefill-heavy work and adopted FlashInfer’s NVLink one-sided all-to-all, in which each GPU writes its tokens directly into the receiver’s memory, for “roughly 5%” end to end on a GB200 NVL72 rack.

The furthest step fuses the exchange into the expert kernel. DeepSeek’s MegaMoE, in DeepGEMM, runs dispatch, the two expert matrices, the activation and combine as “a single pipelined kernel” in which “computation of current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently”, the software pipelining distributed linear algebra has long used. The DeepSeek-V4 report measures “1.50∼1.73×” over “strong non-fused baselines” for general inference and up to 1.96× for latency-sensitive work such as RL rollouts. FlashInfer added fused MoE kernels in CuTe DSL for Blackwell in July 2026, and the vLLM recipe for Mistral Large 4 uses one on the prefill side of a disaggregated preset: on 4 + 4 GB300s, 241 against 206 output tokens per second per GPU at 32K-token inputs and a floor of 40 tokens per second per user (+17%, computed), 154 against 149 at a floor of 100.

The fused kernel has costs as plain as its gains. MegaMoE needs expert parallelism over NVLink with symmetric memory on every GPU (the Kimi K3 recipe: “not compatible with cross-node RDMA”), and the DeepSeek-V4-Pro recipe tells aggregated tensor-parallel serving not to pick an EP MoE backend. It is cut for formats: DeepGEMM’s first Mega MoE supported “Only FP8 x FP4 MoE”, and vLLM’s deep_gemm_mega_moe is “FP8-only”, with a separate variant for V4-Pro’s mixed checkpoint. FlashInfer’s Hopper MegaMoE needs the token count to “be equal on every rank”, so short ranks are padded. The Mistral Large 4 recipe uses it on decode servers only “with --max-num-batched-tokens below 512”; at 8,192 it “lost 10% at 8k”. On an expert-parallel decode preset at 8K-token inputs, MegaMoE on both sides gave 766 against 723 tokens per second per GPU at the floor of 40 (+6%), and 98 against 178 at the floor of 100 (−45%, computed).

Launches and CUDA graphs

A kernel launch has two costs. The CPU has to issue it (in eager PyTorch: Python, PyTorch’s dispatcher and the driver), a cost none of our sources measures for these models. And the GPU sits idle between one kernel’s end and the next one’s start: Hazy Research timed that gap on an H100 at about 2.1 µs for kernels queued on a stream and about 1.3 µs inside a CUDA graph, “time spent with the GPU doing no useful work”. No source counts GLM-5.3’s kernels, but vLLM measured Kimi K3, a large MoE mixing MLA and linear (KDA) attention layers, at 23 kernels per layer on its stock path: 1,817 kernels for GLM-5.3’s 78 layers and MTP layer, about 1,840 with an assumed 20 for the embedding, output layer and sampling (our estimate, compute.py §C). At batch 1 the floor of 1,381 µs is 0.75 µs of GPU work per kernel, less than the gap after it.

Counting each kernel’s work plus the gap after it (the serialized model used from here on), an eager step at batch 1 takes about 5,240 µs, three quarters of it gaps. A CPU running ahead of the GPU hides its own issue cost, not the GPU’s gap: with nn kernels of GPU time tit_i (each with its gap) and cc µs to issue each, the step lies between max⁡(nc,∑iti)\max(nc, \sum_i t_i) and nc+∑itinc + \sum_i t_i.

A CUDA graph records a sequence of kernels with their arguments once and replays it with a single launch from the CPU, which removes the CPU’s issue cost and shortens the GPU’s gap. Everything is frozen at capture: tensor shapes, memory addresses, the number of thread blocks. Engines therefore capture one graph per batch size (vLLM’s older default: 1, 2, 4, then every 8 up to 256 and every 16 up to 512) and pad each batch to the nearest captured size. Padding is free for the dense products, and the engines keep pad tokens out of the experts: vLLM sets their expert ids to −1 so supporting MoE kernels drop them (VLLM_MOE_SKIP_PADDING, on by default), and SGLang’s MoE routing skips “much of the padded region”. Graphs also hold memory: for GLM-5.2, SGLang measured 2.4 GB for 42 prefill shapes, yet the total came out 1.10 GB below running without graphs, because capturing up to the chunk size removes a per-request activation peak.

  • CPU launching a kernel
  • GPU idle between kernels
  • dense products
  • routed experts
  • sparse attention
  • indexer
One average layer of a decode step, B = 1 (modelled)eagerinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work66.0 µspiecewise graphinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work49.2 µsfull graphinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work47.6 µsCPU idle: the step is one graph launchfewer kernels (13 per layer)RMSNorm and quantization: 0.00 µs of workq_a and kv_a projections: 1.63 µs of worklatent norms, rope, cache write: 0.00 µs of workq_b and key absorption: 1.63 µs of workindexer projections: 1.63 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption and o_proj: 1.63 µs of workall-reduce, residual, RMSNorm: 0.00 µs of workrouter and shared expert gate/up: 1.63 µs of workshared expert down: 1.63 µs of workrouted experts: 7.56 µs of workall-reduce: 0.00 µs of work34.6 µsCPU idle: the step is one graph launch020406080µs (linear, from 0)One average layer of a decode step, B = 1 (modelled)eagerinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work66.0 µspiecewise graphinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work49.2 µsfull graphinput RMSNorm: 0.00 µs of workq_a and kv_a projections: 1.09 µs of workquery latent norm: 0.00 µs of workkey-value latent norm: 0.00 µs of workq_b projection: 1.09 µs of workkey absorption (W_UK): 1.09 µs of workrotary embedding: 0.00 µs of workKV cache write: 0.00 µs of workindexer projections: 1.09 µs of workFP8 query quantization: 0.00 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption (W_UV): 1.09 µs of worko_proj: 1.09 µs of workall-reduce: 0.00 µs of workresidual add and RMSNorm: 0.00 µs of workrouter: 1.09 µs of workrouting top-k: 0.00 µs of workshared expert gate/up: 1.09 µs of workshared expert down: 1.09 µs of workrouted experts gate/up: 3.78 µs of workrouted experts down: 3.78 µs of workall-reduce: 0.00 µs of work47.6 µsCPU idle: the step is one graph launchfewer kernels (13 per layer)RMSNorm and quantization: 0.00 µs of workq_a and kv_a projections: 1.63 µs of worklatent norms, rope, cache write: 0.00 µs of workq_b and key absorption: 1.63 µs of workindexer projections: 1.63 µs of workindexer scoring and top-k: 0.06 µs of worksparse latent attention: 0.29 µs of workvalue absorption and o_proj: 1.63 µs of workall-reduce, residual, RMSNorm: 0.00 µs of workrouter and shared expert gate/up: 1.63 µs of workshared expert down: 1.63 µs of workrouted experts: 7.56 µs of workall-reduce: 0.00 µs of work34.6 µsCPU idle: the step is one graph launch020406080µs (linear, from 0)
whole step, B = 1µsbelow eager
eager5,239–
piecewise graph3,89625.6%
full graph3,76928.1%
fewer kernels (13 per layer)2,74247.7%

GPU work alone (per-operation roofline): 1,381 µs.

A model with stated inputs, not a trace. Each kernel does its roofline work and is followed by a gap in which the GPU does nothing useful: 2.1 µs launched from a CUDA stream, 1.3 µs inside a CUDA graph (Hazy Research's H100 measurement); 5 µs is Hazy's estimate of a typical stall. Kernels per step: 23 per layer (vLLM's count for Kimi K3, assumed for GLM-5.3) × 79 layers + 20 = 1,837; with 13 per layer, 1,047. The piecewise graph runs the indexer and attention kernels of every layer outside the graph (an assumption). The kernel list and the split of each layer's work among its kernels are ours; kernels that only move activations (norms, rope, all-reduce) carry no work in the roofline and show as slivers. The CPU is assumed not to run ahead of the GPU.

Figure 4.One layer of a GLM-5.3 decode step, modelled: each kernel is followed by an idle gap on the GPU. A graph shortens the gaps, and only fewer kernels remove them; their share falls as the batch makes each kernel longer. A model with stated inputs, not a trace; the CPU's own issue cost in eager mode comes on top.

Not every kernel can be frozen. Attention kernels often plan their work on the CPU from the current sequence lengths, which change every step. Engines have three answers. A full graph captures the attention too, which needs a kernel whose launch configuration doesn’t depend on the lengths; FlashInfer, for instance, copies each step’s plan into a fixed buffer read by a kernel that stays resident with a fixed number of thread blocks. A piecewise graph leaves attention out: vLLM’s compiler (next section) splits the model at every attention call, captures the pieces between, and runs attention eagerly, so a layer costs one graph replay plus the eager launches of its attention, cache write and indexer. A breakable graph, introduced in SGLang in February 2026 and adopted by vLLM, captures the whole forward in one pass, ending the capture at each incompatible operation, running it eagerly and resuming, with no compiler involved; SGLang uses it for prefill by default and measured prefill-only steps 1.60× faster than eager on GLM-5.2.

vLLM’s default mode, FULL_AND_PIECEWISE, uses full graphs for uniform decode batches (every sequence with the same query length) and piecewise graphs for anything with prefill in it. FULL_DECODE_ONLY drops the piecewise graphs to save their memory and leaves prefill and mixed steps eager; DeepSeek-V4-Pro’s vLLM recipe uses it even for aggregated serving on 8×H200. Speculative decoding changes the shape: a step that verifies γ\gamma drafted tokens has γ+1\gamma+1 query positions per sequence, so the V4-Pro recipe’s GB200/GB300 section asks for “explicit capture sizes n*(K+1) for n=1..S” (its KK is our γ\gamma, SS the sequences per engine); vLLM’s newer model runner derives these sizes itself. Hybrid models add state: each sequence in a full decode graph needs its own block of recurrent (Mamba) state, so the state blocks cap the batch a graph can hold, and current vLLM refuses to start when the capture size exceeds them. For GLM-5.3, current vLLM (commit of 2026-11-11) uses breakable graphs by default, and its sparse MLA backend supports full graphs for uniform decode batches, speculative ones included, so a decode step can replay as one graph.

A graph shortens the gaps but keeps one per kernel. In the serialized model a full graph cuts the batch-1 eager step from 5,239 to 3,769 µs (28%), and the 2,388 µs of gaps left still exceed the 1,381 µs of work; at batch 64 it saves 6.5% (compute.py §H5). Only fewer kernels remove gaps: at 13 kernels per layer the batch-1 graph step falls to 2,742 µs. Async scheduling (vLLM) and the overlap scheduler (SGLang), both on by default, take the CPU’s share off the path by preparing the next batch while the GPU runs the current one. A persistent kernel goes furthest: it stays resident and pulls work from a queue instead of being launched per operation. Hazy Research’s megakernel runs a whole forward pass of a 1B model as one kernel, written for that model and that GPU. vLLM’s specialized decode for Kimi K3 cuts a layer from 23 kernels to 13 with fusion, small-matrix kernels and overlap with communication, from 130 µs to 69 µs per layer on a GB300 (the 10 fewer gaps are some 13 to 21 µs of that, our estimate): 1.58× the tokens per second at batch 1 on eight GPUs, 1.31× at batch 64.

Compilation

Hand-written kernels cover the large operations. For the rest, vLLM compiles the model with torch.compile: Dynamo traces the forward pass into a graph, with attention wrapped as one opaque custom operation, and Inductor generates fused Triton kernels for the pieces between attention calls. vLLM’s own CUDA versions of RMSNorm, activations and the like are switched off where Inductor’s are faster (on NVIDIA “Inductor actually generates a faster fused kernel than our custom CUDA kernel” for RMSNorm with FP8 rounding), and pattern-matching passes fuse across the custom kernels that remain and into collectives, as with the all-reduce and the following RMSNorm; “any fusion involving custom ops (attention, collectives, sub-byte quantization) continues to require custom passes”. The compiled graph is also what piecewise CUDA graphs are cut from.

Compilation runs at startup and is cached on disk; SGLang measured torch.compile at 78–86% of the time spent preparing prefill graphs, 158 seconds on GLM-5.2. Kernel libraries compile just in time too: a cold start of NVIDIA’s RL stack on more than 1,000 GPUs spent 38.8 minutes compiling kernels, 28 of them FlashInfer’s; with those baked into the container image and the other caches warm, that phase took 0.4 minutes.

On NVIDIA GPUs vLLM doesn’t compile several of the newest architectures at all. Breakable graphs set the compilation mode to none, and the architectures using them by default include GLM-5.3, DeepSeek-V4 and V4.1, Kimi K3, MiniMax-M3 and Step-5. In the worked example, then, every elementwise operation runs as vLLM’s own custom kernel or a plain PyTorch kernel, unfused, and the fusions that matter are written by hand into the model’s kernels, as DeepSeek’s fused attention kernel and Mega-mHC are. On AMD GPUs the default list is shorter (“Breakable CUDA graphs currently regress performance on ROCm”), and the recipes compile the same models, except DeepSeek-V4.1, which “does not support torch.compile”, and run AITER, AMD’s kernel library. SGLang leaves torch.compile off by default.

Determinism: the same token whatever else is in the batch

Run the same prompt at temperature 0 twice on a busy server and the outputs can differ. Thinking Machines sampled Qwen3-235B 1,000 times at temperature 0 and got 80 different completions. Their explanation: most inference kernels use no atomic additions, so a run on the same batch gives the same bits, but a kernel processes a request differently depending on the batch around it. RMSNorm, matrix products and attention all reduce (sum) over a dimension, and at small batch a fast kernel splits that sum across more SMs (split-K for products, split-KV for attention). Floating-point addition isn’t associative, so a different batch, through a different split, gives a slightly different sum. What’s missing is batch invariance: the output for a request is bitwise the same in every batch that contains it. Across GPUs the all-reduce and the expert combine are sums too, and kernels that accumulate with atomics, such as some expert combines, aren’t even repeatable on one batch.

The fix is a reduction order that doesn’t depend on the batch: no split-K, a fixed tiling along the summed dimension, and attention split into chunks of fixed size rather than a fixed number of chunks; one kernel configuration for all shapes is the simplest way. Compensated (Kahan) or pairwise summation makes the sum more accurate but still order-dependent, so it doesn’t help. vLLM’s batch-invariant mode also keeps its own all-reduce “with a fixed reduction order” and turns off the faster all-reduce kernels. It costs speed: Thinking Machines’ batch-invariant matrix product was about 20% slower than cuBLAS, and a Qwen3-8B run took 42 s instead of 26 s; SGLang’s deterministic mode measured slowdowns of 25% to 45% (average 34%) on Qwen3-8B on an H200.

A serving stack pays this for RL (S5): if the rollout engine and the trainer compute bitwise the same probabilities, the policy (the model being trained) that samples and the one trained are the same function, which needs batch-invariant kernels in the engine and the same reduction orders in the trainer. Of the 14 labs we searched (13 through their own documents, Tencent only through a vLLM recipe), one states batch-invariant inference kernels. DeepSeek-V4 replaced cuBLAS “end-to-end” with DeepGEMM, abandoned split-K “in most scenarios”, and gave up split-KV in decode attention, which leaves the last partly filled wave of sequences on few SMs. Its answer is a pair of decode kernels: one computes a whole sequence on one SM, the other spreads the last wave’s sequences across several SMs with “the same” accumulation order, so both give identical bits; the report calls the overhead “negligible” without a number. Aleph Alpha states the opposite for Kolibri’s RL: “none of the kernels are batch-invariant”. DeepGEMM’s kernels were “fully deterministic and batch-invariant” until September 2026, when a switch made that opt-in “so some kernels can run faster”.

The kernel families side by side

Kernel familyCost term it movesPhase where it paysWhat it is cut forExamples
Fusion of elementwise chainsHBM writes and rereads of intermediates; launchesprefill (bytes), decode (launches)one chain of operations, layout, formatRMSNorm + FP8 rounding; Mega-mHC
FlashAttention familythe T×TT \times T score trafficprefill and traininghead width, GPU generation (FA3 Hopper, FA4 Blackwell)FlashAttention 2, 3, 4
Latent and sparse decode attentioncache bytes per step, conversion cycles, or padded-head FLOPs (TP, speculation)decodecache layout (656, 584, 528, 288 B per token), heads per GPUFlashMLA, FlashInfer MLA, TOKENSPEED_MLA
Grouped expert productsweight bandwidth at decode, tensor-core use at prefillboth, differentlyexpert shape, format, GPU, rows per expertDeepGEMM, Marlin, Triton MoE, token-centric decode
All-to-all kernelsexchange time, SMs taken from computeEP servinginterconnect (NVLink or RDMA), buffer sizesDeepEP, NVLink one-sided
Exchange fused into expertsexposed exchange timeEP prefill and decodeNVLink domain, symmetric memory, tokens per rankMegaMoE (DeepGEMM, FlashInfer)
CUDA graphs, persistent kernelsCPU issue time (graphs); GPU gaps per kernel (shorter in graphs, fewer with fused or persistent kernels)small-batch decodebatch sizes, memory addresses; megakernels one modelfull, piecewise, breakable graphs; megakernels
Batch-invariant kernelsnone; costs speedRL rollouts, debuggingfixed reduction orderDeepSeek-V4, SGLang and vLLM modes

Where the models in the map sit

Kernel work named in each lab’s own documents, and the kernel flags its engine recipes set (vendor recipes marked as such). A blank means we searched and found nothing.

ModelLab’s own documentsRecipe flags (vendor)
DeepSeek-V4 / V4.1MegaMoE; TileLang kernels replacing “the vast majority” of PyTorch operators; batch-invariant kernels; V4.1: fused attention kernel, Mega-mHC, 11 kernels per decode layervLLM V4-Pro on 8×H200: "cudagraph_mode": "FULL_DECODE_ONLY", --block-size 256; Blackwell with EP: deep_gemm_amxf4_mega_moe (V4-Pro), deep_gemm_mega_moe (V4.1); SGLang Hopper: Marlin W4A16 or FlashInfer W4A8 experts
Kimi K3FlashKDA (training and prefill); fused KDA decode kernel replaying inputs for speculative verification; AttnRes merge fused into the all-reduce; token-centric expert decode kernelvLLM Blackwell: --attention-backend TOKENSPEED_MLA; NVLink EP: flashinfer_nvlink_one_sided all-to-all; SGLang Hopper: Marlin experts, FlashMLA decode
Mistral Large 4context-parallel collectives with in-kernel overlap: one index-producing layer’s decode step 240 to 153 µs on GB300vLLM: dedicated image; breakable graphs and FlashMLA sparse by default; MegaMoE (flashinfer_moe_ep_cutedsl) on EP prefill
Nemotron 3 Ultra / 3.5NVLink one-sided all-to-all (about 5%); weights padded at load for kernel alignment; 3.5: Mamba state replay for verification, multi-wave KV splitUltra model card (NVIDIA’s own): Blackwell only for now; SGLang --disable-piecewise-cuda-graph
GLM-5.3 (GLM-5 report)one quantization kernel bitwise identical in training and inference; deterministic top-k in RLvLLM: breakable graphs, no compilation (engine default); SGLang: --cuda-graph-max-bs-decode twice the concurrency
MiniMax-M3sparse-attention kernels: a top-k without exponentials, a persistent grid over key blocksvLLM: --block-size 128 “mandatory”
Step-5-Previewengine commands in StepFun’s README: SGLang FlashAttention-4 and MegaMoE on B300, FlashAttention-3 and Triton MoE on H200; MegaMoE needs expert parallelismvLLM recipe: the same MegaMoE split
gpt-oss-120bTriton MoE kernel with MXFP4 (a reference implementation, “not expected to be run in production”)vLLM: Marlin MXFP4 on A100, FlashInfer TRT-LLM MXFP4 on B200

Where one model’s documents cover both GPU generations, the kernels split by generation: for DeepSeek-V4, Kimi K3 (in SGLang) and Step-5-Preview, Blackwell gets MegaMoE, FlashAttention-4 or TRT-LLM MLA and Hopper gets Triton, Marlin, FlashAttention-3 or FlashMLA, and gpt-oss makes the same split between B200 and A100. They split by vendor too: the AMD cells of the same recipes run AITER kernels and, except for DeepSeek-V4.1, compile the model. MegaMoE requires expert parallelism in every recipe that explains it.

Easy to get wrong

  • “Decode attention is memory-bound.” With MLA it depends on how many query heads share each cached entry on a GPU, how many positions are verified at once and how the kernel pads heads. Under TP8 GLM-5.3’s useful work is 26.5 FLOPs per byte, but vLLM’s kernel computes 64 padded heads and sits at the BF16 ridge, and DeepSeek’s decode with 128 heads is compute-bound by FlashMLA’s own analysis.
  • “Few rows per expert waste the tensor cores, so expert kernels are slow at decode.” The tensor cores are mostly idle, but the time is set by reading the weights. A decode expert kernel should be compared with the bandwidth roof, not with peak FLOP/s.
  • “CUDA graphs remove launch overhead.” They remove the CPU’s issue cost and shorten the GPU’s gap between kernels (2.1 to 1.3 µs in Hazy Research’s measurement), and the kernels inside run as before; SGLang says of its breakable graphs that they do “not reduce model FLOPs or make compute-bound kernels cheaper”. At batch 1 the gaps left inside a graph still exceed GLM-5.3’s work; only fewer kernels remove them.
  • “Deterministic” and “batch-invariant” are the same thing. A kernel run twice on the same batch usually gives the same bits; what changes the output is a different batch around the request.
  • “vLLM compiles every model.” On NVIDIA GPUs it runs GLM-5.3, DeepSeek-V4, Kimi K3 and others with breakable graphs and compilation off; their fusions come from hand-written kernels.

Check yourself

  1. A decode step of GLM-5.3 at batch 8 under TP8 has a floor of 5,216 µs by the per-operation roofline, spread over about 1,840 kernels. In the serialized model, with a GPU gap of 2.1 µs after each kernel run eagerly and 1.3 µs inside a CUDA graph, how long are the eager and the full-graph steps? Why does a path with 13 kernels per layer instead of 23 still help inside a graph?
Answer

Eager: 5,216+1,840×2.1≈9,0805{,}216 + 1{,}840 \times 2.1 \approx 9{,}080 µs. Full graph: 5,216+1,840×1.3≈7,6105{,}216 + 1{,}840 \times 1.3 \approx 7{,}610 µs, about 16% less; the 2,390 µs of gaps left are almost half the work. The gaps scale with the number of kernels, so 13 per layer (about 1,050 kernels) leaves some 1,360 µs of gaps and a step of about 6,580 µs, 28% below eager (computed, compute.py §H5, with the article’s assumptions). The CPU’s issue cost comes on top in eager mode wherever it exceeds a kernel’s time.

  1. An expert product reads one FP8 expert slice of WW bytes and multiplies it by rr token rows. On an H200 (4.8 TB/s, 1,979 TFLOP/s FP8), at what rr does it stop waiting on memory, and why doesn’t padding r=3r = 3 rows to a 128-row tile slow it down?
Answer

It does 2r2r FLOPs per weight byte, so it reaches the ridge of 1,979/4.8≈4121{,}979 / 4.8 \approx 412 at r≈206r \approx 206. At r=3r = 3 padded to 128 rows it computes 2×128=2562 \times 128 = 256 FLOPs per byte, still below 412, so the padded arithmetic finishes before the weight bytes arrive. Time is W/4.8W / 4.8 TB/s either way.

  1. Why can’t a decode attention kernel that splits each sequence’s cache into “as many pieces as fill the GPU” be batch-invariant, and what does DeepSeek-V4 do instead?
Answer

The number of pieces depends on how many other sequences share the step, and each piece’s partial result is merged with the online-softmax rescaling, a floating-point sum whose order changes with the number of pieces. DeepSeek-V4 gives up the split: one kernel processes a whole sequence on one SM, and a second kernel, for the last partly filled wave, spreads a sequence over several SMs but accumulates in the same order, so both give the same bits.

In the map

Methods

Taxonomy nodes

Further reading

  • Dao et al., FlashAttention (2022): the IO analysis and the lower bound; FlashAttention-2 (2023) and -3 (2024), and Zadouri et al.’s FlashAttention-4 (2026), for how each GPU generation changed the kernel.
  • FlashMLA, docs/ on GitHub: the 2025-04-22 deep dive (why MLA decode is compute-bound) and the 2025-09-29 one (the FP8 sparse kernel and dequantization-bound decode).
  • DeepGEMM and DeepEP READMEs and release PRs: grouped-GEMM layouts, Mega MoE benchmarks, the two DeepEP generations.
  • DeepSeek-V4 report §3 (fused EP kernel, TileLang, batch-invariant kernels) and the V4.1 report §2.4.1 (Mega-mHC’s traffic count).
  • Kimi K3 report §5.4.2: decode kernels for KDA, attention residuals and latent MoE.
  • vLLM design docs cuda_graphs.md, torch_compile.md and fusions.md; SGLang’s “Advanced CUDA Graph Techniques” post (August 2026) for breakable graphs.
  • Horace He and Thinking Machines, “Defeating Nondeterminism in LLM Inference” (2025): batch invariance kernel by kernel.
  • Frantar et al., MARLIN (2024): why weight-only quantization pays up to a batch of about 32.
  • Hazy Research, “Look Ma, No Bubbles” (2025): launch overhead and a whole forward pass as one kernel.
↑