GLM-5.2 lets each new token attend to 2,048 of the tokens before it (index_topk: 2048), one in
64 at 128K tokens of context. A small scoring network, the indexer, picks them by reading a short
key for every one of the 131,072 cached tokens. Generating a token is limited by the bytes the GPU
reads, and in bytes that scan costs more than the attention it steers: a GLM-5.2 layer reads 17.8
MiB per generated token instead of 82 MiB, 4.6 times less, not 64 (computed below). Running the
indexer in 21 of 78 layers brings the model to 14 times. The cache still holds 6.58 GiB.
Sparse attention runs exact softmax attention over a subset of the past, chosen anew for every query. It keeps exact recall for what it selects and bounds the work per query while the memory keeps growing; T2‘s fixed-state mixers bound the memory and give up exact recall. The selection must cost less than the attention it saves, catch most of the mass dense attention would use, and read memory in shapes a GPU handles well.
Selection buys attention work and cache reads with selector overhead and recall risk, and only compression shrinks the cache.
Background
Prefill processes the prompt all at once; decode then generates one token at a time, each reading the KV cache of every layer. A kernel’s arithmetic intensity is the FLOPs it performs per byte it reads; an H800 does about 590 FP8 FLOPs per byte it reads (vendor peaks), so a kernel far below that is timed by its bytes, as decode attention with one query per sequence is (T1).
Multi-head latent attention (MLA, T1) caches one latent of numbers per token, plus a -number key that carries position; at decode all query heads read the same entry. FP8 and FP4 store a number in one byte or half a byte, with scales shared by a group of numbers.
Write for the set of past positions query may use (a set, not T2‘s state matrix), for its size and for the head width. Sparse attention computes the softmax over :
Work and bytes per query grow with , not . Let be the dense weights and the mass outside . Scores inside are exact, but each kept weight becomes . With the weighted mean value over the skipped positions, the dense output is , so
A selector is as good as the mass it misses: the top positions by true score minimize , the oracle every selector is judged against, and since softmax weights are all positive, whenever . is chosen per query, so a token query skips may be the one query needs, and every key and value stays cached.
fixed pattern sink 1–2 + recent window
missed mass ε 0.42
block top-k own block + best blocks of 4
missed mass ε 0.30
token top-k single positions
missed mass ε, true scores 0.15indexer 0.27
ranked bydense attention weightselectedquery rowcached (all kept)rows = queries, columns = past positions; click a row to make it the query
Query 28, budget 8 of 28 positions: the fixed pattern misses 42% of the attention mass, blocks 30%, single tokens ranked by the true scores 15% (the least any selector of 8 can miss), ranked by the noisy indexer 27%.
Fixed patterns
The oldest selectors ignore content. BigBird (Zaheer et al., 2020) combines a local window, a few global positions and random blocks. StreamingLLM (Xiao et al., 2023) found that a window alone fails because trained models park attention mass on the first tokens, an attention sink, and that “keeping the KV of initial tokens will largely recover the performance of window attention”; it claims stable perplexity, not retrieval of old facts.
A fixed pattern costs nothing to select, but a fact outside the window and anchors is out of reach. On RULER, synthetic long-context retrieval tasks scored out of 100, the GLM-5 report found that converting a 9B GQA model to alternating window and full layers by continued training causes “catastrophic degradation on long-context tasks (e.g., −30.35 on RULER@128K)”: a 30.35-point drop, from 75.28 to 44.93, with half the layers still full. Fixed choices live on inside the learned designs: MoBA always keeps the query’s own block, NSA the first and two local blocks, DeepSeek-V4 a 128-token window in every layer.
Content-based selection is older than learned indexers: Reformer (Kitaev et al., 2020) hashed queries and keys into buckets, the Routing Transformer (Roy et al., 2021) clustered them, Memorizing Transformers (Wu et al., 2022) attended to the nearest stored keys. Quest (arXiv:2406.10774), whose per-page minimum and maximum keys bound , and InfLLM (arXiv:2402.04617, “without any training”) select at inference in models trained dense.
Block selection
MoBA
Mixture of Block Attention (MoBA; Lu et al., 2025) splits the keys into blocks of tokens, represents each block by the mean of its keys, and lets each query take the top blocks by inner product with those means, plus its own block. It resembles a mixture-of-experts router, except that no gate value scales the chosen blocks, so the choice receives no gradient; the router has no parameters. Since , one token scoring above its neighbours lifts its block by only .
Its authors switched 29 of Llama 3.1 8B’s 32 layers to MoBA for 100B tokens (, top 12) and report RULER at 128K of 0.7818 against 0.7849 for full attention, with prefill “up to 6.5x” faster at 1M. At 128K those 12 blocks are 37.5% of the context, and generation ran dense.
Native Sparse Attention
NSA (Yuan et al., 2025) gives each query three attention branches, each with its own key and value projections, mixed by gates from an MLP and a sigmoid:
The compression branch attends to coarse tokens, each a small MLP of 32 consecutive keys taken every 16 positions, in effect a learned strided convolution. Its attention weights double as block scores (a 64-token block sums the scores of the coarse tokens overlapping it), and the selection branch attends to the original keys of the 16 best blocks, the first and two local ones by force. The window branch covers the last 512 tokens, because local patterns otherwise “can dominate the learning process”. All heads of a GQA group share one block choice, so a kernel loads each block once per group. Storage is a full cache plus coarse entries and the window (our reading).
Toy: the compression lane reads all keys through 15 coarse tokens (8 keys every 4); its scores pick 3 blocks of 8 (blocks 1, 4, 8; the first and the local one forced); the window reads the last 8. Hover a lane to see the keys it reads.
NSA was trained sparse from scratch in a 27B-parameter mixture-of-experts model with 3B active parameters (the weights one token uses). It reports beating full attention “on 7 out of 9 metrics” and a 9.0× forward speed-up at 64K against a Triton FlashAttention-2 kernel; its 11.6× decode figure is an “expected speedup” from 5,632 tokens loaded instead of 65,536. No model in the map ships NSA, but coarse scores steering a fine selection, a window beside it, and one choice per GQA group all recur.
MiniMax Sparse Attention
MSA (MiniMax, 2026) is block selection on grouped-query attention (GQA), where each key/value head
serves a group of query heads. Its Index Branch adds “one index query head for each GQA group and a
single index key head shared across groups”. Each index head scores every visible token, takes the
maximum inside each 128-token block (the other end of the pooling choice from MoBA’s mean), keeps
the query’s own block, and picks the top 16 blocks for its group; the Main Branch runs exact GQA
attention over them. In MiniMax-M3
(sparse_block_size: 128, sparse_topk_blocks: 16, sparse_num_index_heads: 4 equal to
num_key_value_heads: 4) each of four groups reads 2,048 tokens of its own choosing, in 57 of 60
layers; M3.1-Flash makes all 60 sparse.
The paper reports that “on a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4× at 1M context” (quality compared up to 128K), with kernel speed-ups of 14.2× in prefill and 7.6× in decode. Both follow from the indexer’s cost: in FLOPs it is 16 times the sparse attention at 1M, so dense ÷ (sparse + indexer) ≈ 512/17. In bytes, a dense layer reads 2,048 B per past token, the sparse one B of selected entries plus a 256 B index key per past token: 7.9× at 1M, against the measured 7.6× (computed, BF16). Nothing is compressed: M3 stores B per token, 16.8 GiB at 128K, so its card’s “dramatically reduces the attention compute and memory footprint” holds for compute only.
DeepSeek Sparse Attention
DeepSeek Sparse Attention (DSA), introduced in DeepSeek-V3.2, selects single tokens. A lightning indexer scores each past token for query as
where each of the indexer heads has its own query of width , all heads
share one key per token, and comes from a linear layer with no sign constraint, so
the score is a small ReLU network of the key rather than a similarity. The main attention takes the
top tokens. V3.2’s indexer has 64 heads of 128 dimensions (index_n_heads: 64,
index_head_dim: 128); GLM-5.2, GLM-5.3, Hy4-preview and Mistral Large 4 have 32 of 128. All five select (index_topk). By our reading the
indexer is cheap mostly through its single shared key, FP8 scoring and a ReLU chosen “for
throughput consideration”; its 32 or 64 heads are half the main attention’s 64 or 128.
Every token selector in the map selects from one shared entry per token, MLA’s latent or DeepSeek-V4’s shared key-value entry, so one gather of 2,048 latents (copying rows at scattered positions into one tile) serves all heads. V3.2 also chose MLA because it continued training from an MLA model.
At T = 128K (131,072 tokens), this layer reads per decode step 131,072 indexer keys × 132 B = 16.5 MiB and 2,048 latents × 656 B = 1.28 MiB. Stored per sequence, whole model: 131,072 × (78 × 656 + 21 × 132 = 53,940 B) = 6.58 GiB (FP8 layout).
Hover the indexer or the attention for its FLOPs per decoded token.
Entry sizes in DeepSeek's FP8 layout (vLLM fp8_ds_mla; external, assumed for GLM-5.2). The strip is a toy: 32 past positions, 8 picks standing for 2,048 of T.
Per layer and token the cache holds the latent, plus a 128-number indexer key in layers with an
indexer: 1,152 B plus 256 B in BF16. DeepSeek’s FP8 layout takes 656 B (512 FP8 numbers, four FP32
scales, 64 BF16 position numbers) and about 132 B; vLLM’s Mistral Large 4 recipe names it as its
default cache (fp8_ds_mla), and with it the V4 report’s cache ratios below come out right. For
GLM-5.2 that is 53,940 B per token, 6.58 GiB at 128K (computed).
Training the indexer
Top- returns positions that do not change when a score moves slightly, so its gradient with respect to the scores is zero almost everywhere and the language-modeling loss cannot teach a selector what to pick. MoBA has no selector parameters; NSA reuses the scores of a branch trained through its own output; DSA, MSA and QSA fit the indexer to the main attention with a separate loss. None uses a straight-through estimator or a relaxed top-.
The target sums the main attention over its heads and normalizes over positions, , and the indexer minimizes
the forward direction, which penalizes most where the main attention has mass and the indexer puts little: a penalty for misses, which is what recall needs. It is distillation of an attention map into a small student (and ListNet’s ranking loss, Cao et al., 2007). V3.2 trained it in two stages: a dense warm-up of 2.1B tokens with everything else frozen, then 943.7B tokens of joint training with top-2,048 selection, where and the softmax are taken over only, so a token the indexer failed to pick gets no correction. The indexer’s input is detached and the target held fixed, so the KL cannot reach the main model; MiniMax found that otherwise “the backbone can lower the KL loss by simplifying the Main Branch attention distribution”. Mistral Large 4 used the same two stages (15.1B and 198B tokens).
GLM-5’s ablation on GLM-4.7-Flash measured the recall cost: after the warm-up alone RULER at 128K fell from 79.21 to 71.35, and after 150B tokens of joint training it stood at 78.86, “only a 0.35-point deficit”. In reinforcement learning the rollout engine and the trainer must select the same tokens; GLM-5 found that non-deterministic top-k kernels, which can break ties differently, “caused drastic performance degradation during RL after only a few steps”, and froze the indexer.
The indexer’s own cost
At decode the attention reads latents, the indexer one key for every past token. In DeepSeek’s FP8 layout that is B against B per layer, equal at
Dense attention reads B, so in a layer with its own indexer the saving, , approaches : 3.8× at 32K, 4.6× at 128K, 4.9× at 1M, while the attention alone reads 16, 64 and 512 times less (computed; 4.5 in BF16). DeepSeek-V3.2, with an indexer in every layer, reads 4.6 times less cache per decode step at 128K. Bytes decide because the scan does 62.5 FLOPs per byte (), a tenth of the H800’s ~590.
FLOPs decide where work is compute-bound: prefill, and verifying several draft tokens at once. Per past token, absorbed dense MLA costs FLOPs in GLM-5.2 (score plus value sum, a multiply-add counted as two) and the indexer , which equals the sparse attention’s MFLOP at
and costs 30 times as much at 1M (computed; V3.2’s constants are both twice GLM-5.2’s). Counting FP8 work at half cost, as the V4.1 report does, moves the crossover to about 69,000.
In prefill, dense MLA does not run absorbed: “the MHA mode is used for training and prefilling”
(V3.2 report), at FLOPs per query-key pair in GLM-5.2 (from
qk_nope_head_dim: 192, qk_rope_head_dim: 64, v_head_dim: 256), only 7.9 times the indexer’s
(V3.2: 5.0), while sparse attention runs absorbed. DSA therefore beats dense prefill only for query
positions past about 5,000 tokens (V3.2: 8,700), and at 128K a whole prefill costs 5.2 times less
per indexer layer (V3.2: 3.3; computed). Below 2,048 tokens DSA selects everything anyway, and V3.2
runs short prompts in “a masked MHA mode to simulate DSA”.
Neither count includes top-k itself, a pass over all scores per query and layer. In nearest-neighbour terms DSA is retrieve-then-rerank, a linear scan with a cheap learned score; LSH or inverted files would need an inner-product score in a fixed space, which this score is not.
dense MLAsparse MLA attentionindexersparse + indexerGLM-5.2 config; per decoded token
- at
- 128K tokens
- dense
- 82 MiB
- sparse attention
- 1.28 MiB
- indexer
- 16.5 MiB
- sparse + indexer
- 17.8 MiB
- indexer : sparse
- 12.88 : 1
- dense ÷ (sparse + indexer)
- 4.61×
When the savings show up
A decode step reads the active weights once for the whole batch and the cache once per sequence. For GLM-5.2 at 128K, assuming 40 GB of FP8 active weights (GLM-5’s 40B active parameters), a step for one sequence reads 46.7 GB with dense MLA and 40.5 GB with DSA, 1.15 times less; for 32 sequences, 4.6 times less (computed; larger batches touch more experts, lowering both). Selection pays at large batch and long context, which is also where the full-size cache caps how many sequences fit: 64 sequences of 128K hold 421 GiB.
Speculative decoding drafts a few tokens cheaply and has the main model verify them in one pass (3 to 5 drafts in the recipes for GLM-5.3, GLM-5.3-Flash, Hy4-preview and Mistral Large 4). Dense attention reads each entry once for all queries of a step; DSA selects per query, so four verified queries gather up to four sets of latents, and a GLM-5.2 layer at 128K saves 3.8 times instead of 4.6 (computed, disjoint selections).
Tensor parallelism splits heads across GPUs, but with one latent and one indexer key per token for all heads, every rank holds the whole cache and runs the whole indexer (“the indexer is replicated across the TP ranks”, vLLM’s V4.1 code). Mistral cites this “KV replication” as its reason to shard the cache along the sequence instead, exchanging candidates before the global top-k.
Cheaper indexers
Sharing indices across layers
IndexCache (Bai et al., 2026) found that in a 30B DSA model “adjacent layers share 70-100% of their
selected tokens”. IndexCache keeps an indexer in a few Full layers and lets the Shared layers
between them reuse the nearest Full layer’s choice; Z.ai ships it as IndexShare.
GLM-5.2 and GLM-5.3 mark 21 of 78 layers “full” in indexer_types (0, 1, 2, then every fourth from
6); a shared layer builds no indexer (self.indexer = None) and caches no indexer keys. Hy4-preview
has the same 21/57 split at other offsets, Mistral Large 4 44 full and 28 shared.
With 21 indexers the model-level byte ceiling rises from 4.97 to : at 128K a decode step reads 446 MiB (346.5 MiB of indexer keys, 99.9 MiB of latents) against 6.25 GiB dense, 14.3 times less, and the indexer passes the attention in bytes at 37,800 tokens instead of 10,200 (computed). The 128K cache drops from 7.50 to 6.58 GiB.
Sharing loses recall when layers disagree: one indexer in four in a uniform pattern without
retraining lowered IndexCache’s long-context average from 50.2 to 43.0, a searched pattern recovered
49.9, and training each kept indexer on the attention of the layers it serves stayed “within 0.4%
of the baseline”. A separate flag, index_share_for_mtp_iteration, reuses the top-k of the multi-token-prediction draft layer across
its own draft steps; the main model still selects for every verified token.
A · across layers IndexShare in GLM-5.2's first 14 layers
- indexers
- 21 / 78 layers
- decode reads at 128K, dense ÷ DSA
- 14.33×
- indexer passes attention in bytes at
- 37,804 tokens
B · within a layer pooled indexer keys
pool 4: mean (QSA, Qwen3.8-Flash) or learned softmax (GLM-5.3-Flash)
262,144 entries scored per query at T = 1M
indexerindexer keypooled keycached (hatched)read
indexer FLOPs per past token: 8,256 (GLM-5.2), 1,028 (Step-5-preview, per 4 KV groups), 257 (Qwen3.8-Flash)
Pooled indexers
Qwen3.8-Flash shrinks the indexer itself. Its Qwen Sparse Attention (QSA) mean-pools the indexer’s
keys over micro-blocks of four tokens (indexer_compress_ratio: 4), normalizes the pooled key, and
scores it with four ReLU heads of 128 dimensions (indexer_n_heads: 4), summed without learned
weights. It picks 512 blocks, 2,048 tokens (indexer_budget: 2048), plus the current incomplete
block, whose uncompressed keys and values the main attention reads. The indexer costs 257 FLOPs per
past token, 32 times less than GLM-5.2’s, and SGLang caches one pooled key per four tokens. QSA fills
the 12 global layers (24 query heads, 2 key/value heads of width 256) of a Gated DeltaNet hybrid (T2).
The Qwen3.8-Next report motivates micro-blocks “so that the cost of indexing itself falls with sequence length” and states the gain as “indexer cost from to ”: a factor on a term that still grows quadratically. QSA’s indexer equals its sparse attention in FLOPs at about 196K tokens and costs 5.4 times it at 1M (computed). In a 35B-A3B ablation (3B of 35B parameters active), “QSA matches the full-attention baseline at a relative indexer latency of 0.25, whereas IndexShare remains below the baseline at 0.5”, with the layers sharing an index four apart.
GLM-5.3-Flash pools inside the DSA code path: index_kpool: 4 merges raw indexer keys four at a
time with a learned softmax gate, keeps 32 weighted heads, and selects 512 pools, in 11 sparse MLA
layers (without a position key) beside 34 Kimi Delta Attention layers. Pooling cuts its
indexer-to-attention FLOP ratio at 1M from 32 to 8.1 (computed, 64 query heads). Step-5-preview (config and serving
code only) merges indexer keys over regions of eight tokens with a learned softmax, stored in FP8. Its 16
indexer heads of width 256 form four groups, each picking 512 regions, up to 4,096 tokens, for its
own key/value head (num_provider_groups: 4): like MSA it selects per KV group, at 1,028 indexer
FLOPs per past token.
Compressed Sparse Attention and Heavily Compressed Attention
Selection and compression serve different heads. A top- set captures at most
of a head’s mass, hopeless for a head that spreads its weight over
thousands of positions, which a coarse average approximates well (our reading; the
sparse-plus-low-rank idea of Scatterbrain, Chen et al., 2021). NSA mixes both in one layer;
DeepSeek-V4 alternates layer types (compress_ratios).
Compressed Sparse Attention (CSA) turns every tokens into one cached entry, a per-channel
softmax-weighted sum over a window of eight tokens moved four at a time, with weights computed from
each token’s hidden state plus a learned bias per slot. This is learned strided pooling, as in
memory-compressed attention (Liu et al., 2018) and the Compressive Transformer (Rae et al., 2020).
An entry has 512 dimensions and is both key and value for all heads (num_key_value_heads: 1,
head_dim: 512). A lightning indexer with compressed keys of its own picks 512 entries in V4-Flash
and 1,024 in V4-Pro (index_topk), the equivalent of 2,048 or 4,096 tokens. Heavily Compressed
Attention (HCA) compresses every 128 tokens without overlap and attends to all its entries.
Both add an uncompressed 128-token window to the same softmax, because “a query cannot access information from other tokens within its own compressed block”, and a learned sink logit per head, T2‘s stand-in for StreamingLLM’s sink tokens. V4-Flash has 2 window-only, 21 CSA and 20 HCA layers; V4-Pro 30 CSA and 31 HCA.
- stored entry
- read by the query
- indexer
- sink logit
V4-Flash: 2 window-only + 21 CSA + 20 HCA layers (43; compress_ratios). At 128K = 131,072 tokens a CSA layer stores 131,072 / 4 = 32,768 entries and reads the top 512 (index_topk) plus the 128-token window; an HCA layer stores 131,072 / 128 = 1,024 and reads them all plus the window.
Toy strip: 298 tokens at the real ratios and window; the indexer picks 6 entries here. Each CSA entry pools the 8 tokens under its bracket (brackets alternate rows so the overlap shows); the stored entries and the window are both cache, drawn hatched.
Deployed, an entry keeps 448 dimensions in FP8 and 64 position dimensions in BF16 (576 B, scale bytes not stated) and an indexer key takes 68 B in 4-bit MXFP4. HCA layers store no indexer keys and the window is a fixed ring, so V4-Flash stores per token
437 MiB at 128K and 3.39 GiB at 1M, plus 3.0 MiB of window (computed). Against V3.2 at 1M in its FP8 layout that is 7.2%, and 10.3% for V4-Pro; the V4 report says 7% and 10%.
The mixing weights inside an entry are fixed when it is written, so no later query can reweight the tokens it merged. The indexer, scoring entries with 64 heads, costs 52 times its CSA layer’s attention (64 query heads over 512 entries plus the window) in FLOPs at 1M (13 with FP4 work weighted at a quarter; computed). V4 rounds index scores to BF16 for “a 2× speedup for the top-k selector, while preserving a 99.7% recall rate of KV entries”, recall meaning agreement with the top-k chosen at higher precision.
Compressed entries change serving. A prefix-cache hit reuses entries “until the last complete compression block” and recomputes the tail, and V4 treats the window and that tail “as a state-space model”, a per-sequence state like T2‘s hybrids. V4.1 keeps no window keys for cached prefixes and replays the last 128 tokens instead, taking its persistent cache to “roughly 1/8” of V4-Flash’s.
CSA2
DeepSeek-V4.1-Flash has two window-only layers (0, 1) and 38 CSA2 layers, a CSA without overlap,
at ratio 2 in layers 2 to 19 and ratio 1 (no compression) in 20 to 39. Full layers (2, 8, 14, 20;
kv_source_layer_ids) store entries and run an indexer; Reindex layers (24, 28, 32, 36) rescore an
earlier layer’s entries with their own indexer; the rest reuse both entries and top-k. Every layer
keeps its own queries and window. Layers 20 to 39 project their entries from layer 19’s output
(“Causal Encoder-Decoder”), so prefill runs only the lower half. With main entries in FP4 (288 B:
4-bit numbers, an 8-bit scale per 16 channels) and indexer keys assumed in MXFP4 (68 B, as in V4),
the report’s figure exactly, and 0.256 of V4-Flash’s 3,471 B (“roughly 1/4”). If every CSA2 layer stored its own entries in V4’s format, the lighter compression would cost 18,676 B, 5.4 times V4-Flash; reuse across layers then saves 11.6 times and FP4 1.8 times (computed). T5 covers what layers can share.
V4.1 also bounds the indexer with a two-stage cascade, the inverted-file idea: layer 20 scans every position and publishes its 2,048 best blocks of eight, and the Reindex layers score only those 16,384 candidates, which “changes the per-query cost of deeper indexers from linear in context length to constant”.
Native or retrofitted
A retrofit’s parity claim compares it with the dense model it came from; a model trained sparse from the start needs a dense twin. DeepSeek alone has done both: V3.2 is a retrofit, V4.1-Flash was trained sparse from scratch.
| Model or method | When sparsity entered | Source |
|---|---|---|
| NSA (27B test model) | from scratch | NSA paper: “Both Full Attention and sparse attention models are pretrained on 270B tokens” |
| MoBA (Llama 3.1 8B) | retrofit after 1M-context continued pretraining | “MoBA is activated for 100B tokens” |
| MSA paper (109B test model) | both: from scratch (MSA-PT) and from a GQA checkpoint (MSA-CPT, 400B tokens) | “supports both training from scratch and near-lossless conversion from pretrained GQA checkpoints” |
| MiniMax-M3, M3.1-Flash | not disclosed | none |
| DeepSeek-V3.2 (DSA) | retrofit by continued training of V3.1-Terminus | “through continued training” |
| GLM-5 (DSA) | retrofit at the end of mid-training | “introduced via Continued Pre-Training from a dense base model” |
| GLM-5.2, GLM-5.3, Hy4-preview | not disclosed | none |
| Mistral Large 4 (DSA) | retrofit after converting a sliding-window and KV-sharing design to full attention: 15.1B-token warm-up, 198B sparse | “we adapted ML4 to DSA” |
| DeepSeek-V4 (CSA, HCA) | during pretraining from 64K sequence length: V4-Flash after 1T dense tokens, V4-Pro after “a longer stage of dense attention” | “introduce sparse attention at the sequence length of 64K” |
| DeepSeek-V4.1-Flash (CSA2) | from scratch; hierarchical indexer in post-training | “We train the model from scratch with sparse attention” |
| Qwen3.8-Flash (QSA) | retrofit at continued pretraining, 256K sequences | “introduced during the continued pretraining (CPT) stage” |
| IndexCache | on an existing DSA model, with or without training | IndexCache abstract |
Only the MSA paper reports both routes on one model: on RULER at 8K, 84.2 from scratch, 77.2 converted, 79.8 for full attention. Mistral’s first design cut the cache 16-fold with sliding windows and cross-layer sharing; agentic long-context scores regressed until it moved back to full attention and then to DSA (report §3).
Comparison
| Fixed pattern | MoBA | NSA | MSA | DSA | Pooled indexer | CSA | HCA | CSA2 | |
|---|---|---|---|---|---|---|---|---|---|
| Unit selected | fixed positions | block (4,096 in the paper’s run) | block of 64, plus coarse tokens and window | block of 128, per GQA group | token, per query | 4 tokens per query (QSA, GLM-5.3-Flash); 8 tokens per KV group (Step-5) | compressed entry (1 per 4 tokens) | none (reads all entries) | entry (1 per 2 or 1 tokens), shared across layers |
| Selector | none | mean-pooled keys, no parameters | compression-branch scores | index branch, max-pooled | lightning indexer, weighted ReLU | indexer on pooled keys | lightning indexer on compressed keys | none | indexer in Full and Reindex layers |
| Per query | window + anchors | top-k blocks + own block | three gated branches | 2,048 tokens per group | 2,048 tokens | 2,048 (Qwen, GLM) or 4,096 per group (Step-5) tokens | 512 or 1,024 entries + 128 window | entries + 128 window | 512 entries + 128 window |
| KV cache | bounded (StreamingLLM) or full | full | full, plus coarse tokens | full | full | full | 1/4, mixed FP8/BF16 | 1/128 | 890 B/token, FP4 |
| Indexer FLOPs per past token | none | one dot product per block | coarse tokens only | 1,024 (M3) | 8,256 (GLM-5.2) to 16,512 (V3.2) | 257 (Qwen3.8), 1,028 (Step-5) | 16,512 per 4 tokens (V4-Flash) | none | 8,256 per entry: layers 2, 8, 14 over , layer 20 over , layers 24–36 over 16,384 candidates |
| Main cost or risk | content-blind | coarse blocks dilute single tokens | three caches and kernels | indexer reads every token | indexer reads dominate past ~10K; gathers | ÷4 or ÷8 on a still quadratic term | averaged tokens; indexer at 1M | very coarse | reuse depends on layer similarity |
| In the map | windows of hybrids (T2) | none | none | MiniMax-M3, M3.1-Flash | V3.2, GLM-5.2, GLM-5.3, Hy4-preview, Mistral Large 4 | Qwen3.8-Flash, GLM-5.3-Flash, Step-5-preview | 5 DeepSeek-V4 models | the same 5 | DeepSeek-V4.1-Flash and its dated -1017 checkpoint |
Where the models in the map sit
The site’s code check lightning-indexer-sparse-attention (“a lightweight indexer scores past
tokens and each query attends only to the top-k tokens it selects”) is present for 17 of 35 models
from seven labs; block selectors satisfy the wording too. Five run DSA over uncompressed tokens (V3.2,
GLM-5.2, GLM-5.3, Hy4-preview, Mistral Large 4; four labs); seven DeepSeek-V4 and V4.1 variants run
it over compressed entries; GLM-5.3-Flash and Qwen3.8-Flash pool per query; MiniMax-M3, M3.1-Flash
and Step-5-preview select per KV group. We reread each verdict against the code and counted
Step-3.7-Flash’s “unclear” as absent (its config enables no sparse attention). Mistral Large 4 also
shares one latent among layers 35, 39, …, 71 (kv_source_layers), a T5 topic.
The table gives every cache in BF16, with the deployed size where a source documents it, how many
times fewer cache bytes a decode step reads than with its sparse layers dense (BF16), and the
indexer’s FLOPs against its own layer’s attention at 1M (computed from each config.json; † beyond
the configured context).
| Model | Unit selected | κ | B/token, BF16 | B/token, deployed (format) | Fixed part, BF16 | 128K, BF16 | 1M, BF16 | Decode reads ÷, 128K | Indexer : attention FLOPs, 1M |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V3.2 | token, indexer in all 61 layers | 2,048 | 85,888 | 48,068 (FP8 layout) | none | 10.5 GiB | 83.9 GiB † | 4.2× | 30.4 |
| GLM-5.2, GLM-5.3 | token, 21 indexers in 78 layers | 2,048 | 95,232 | 53,940 (FP8 layout; FP8 cache in vLLM’s GLM-5.3 recipe) | none | 11.6 GiB | 93.0 GiB | 13.3× | 30.4 |
| Hy4-preview | token, 21 indexers in 78 layers | 2,048 | 95,232 | not stated | none | 11.6 GiB | 93.0 GiB | 13.3× | 30.4 |
| Mistral Large 4 | token, 44 indexers in 72 layers | 2,048 | 82,688 | 45,984 (fp8_ds_mla, vLLM recipe) | none | 10.1 GiB | 80.8 GiB | 6.4× | 30.4 (32.3 without position key) |
| GLM-5.3-Flash | pool of 4, 11 sparse layers | 512 pools | 11,968 (pooled keys; HF code 16,918) | not stated | 140.8 MiB (KDA) | 1.60 GiB | 11.8 GiB | 6.2× | 8.1 |
| Qwen3.8-Flash | block of 4, 12 QSA layers | 512 blocks | 25,344 (pooled keys, as SGLang; HF code 27,648) | not stated | 110.1 MiB (GDN) | 3.20 GiB | 24.9 GiB † | 12.5× | 5.4 |
| Step-5-preview | region of 8 per KV group, 23 layers | 512 regions | 72,128 | 71,392 (FP8 region keys, serving code) | 103.5 MiB (window) | 8.91 GiB | 70.5 GiB | 18.9× | 5.3 |
| MiniMax-M3 | block of 128 per KV group, 57 layers | 16 blocks | 137,472 | not stated | none | 16.8 GiB | 134.3 GiB | 5.5× | 16.0 |
| MiniMax-M3.1-Flash | block of 128 per KV group, 60 layers | 16 blocks | 138,240 | not stated | none | 16.9 GiB | 135.0 GiB | 7.1× | 16.0 |
| DeepSeek-V4-Flash (3 variants) | entry of 4 (CSA) / all of 128 (HCA) | 512 | 6,880 | 3,471 (FP8/BF16 + MXFP4, V4 report) | 5.4 MiB (window) | 865 MiB | 6.72 GiB | 3.4× | 51.6 |
| DeepSeek-V4-Pro (2 variants) | entry of 4 (CSA) / all of 128 (HCA) | 1,024 | 9,848 | 4,969.5 (FP8/BF16 + MXFP4, V4 report) | 7.6 MiB (window) | 1.21 GiB | 9.62 GiB | 3.2× | 14.3 |
| DeepSeek-V4.1-Flash (2 variants) | entry of 2 or 1, shared | 512 | 3,200 | 890 (FP4 + MXFP4, V4.1 report) | 5.0 MiB (window) | 405 MiB | 3.13 GiB | 31.0× | n/a |
Notes: “FP8 layout” is DeepSeek’s 656 + 132 B format (Mistral’s entries without a position key counted at 528 B). “Pooled keys” caches one indexer key per pool, as SGLang does for QSA. The fixed part is one sequence’s in-memory window or recurrent state. Multi-token-prediction layers are left out (+1.5% for GLM-5.2). “Decode reads ÷” compares each layer with reading every entry it could select from, which for V4.1 includes the uncompressed entries of layers 20 to 39. Qwen3.8-Flash’s config stops at 262,144 tokens, though its report evaluates to 1M.
Token and block selection leave the cache at dense size, 12 to 138 KB per token in BF16, lowest in the two linear-attention hybrids, where three quarters of the layers hold a fixed state (T2). In deployed formats V4-Flash stores 15.5 times less than GLM-5.2 per token, V4.1-Flash 61 times less (computed).
- BF16
- documented deployed format
- label: BF16 GiB per sequence incl. fixed part · † beyond configured context
Selection keeps 11,968 B to 138,240 B per token in BF16; hover or focus a row for its formula and config fields.
Easy to get wrong
- 64 times fewer attention FLOPs is not 64 times faster decode. With an indexer in every layer, decode reads fall by at most about 5 times, and the indexer reads more than the attention from about 10,000 tokens, not 34,500.
- The savings depend on batch size. At batch 1 the weights dominate the bytes read, and GLM-5.2 at 128K reads about 13% less per step.
- Below κ tokens DSA is dense attention: every prompt shorter than 2,048 tokens selects everything.
- GLM-5’s “−30.35 on RULER@128K” for sliding windows is a drop from 75.28 to 44.93, not a score. Its “lossless by construction” for DSA cannot hold exactly, since whenever ; its own ablation shows a 0.35-point gap.
- Not every learned selector selects per query: MSA and Step-5-preview choose per KV group.
- MSA’s “on par with GQA” is a 109B test model evaluated up to 128K, and MiniMax-M3’s “memory footprint” claim goes beyond the paper: M3 stores a full cache.
Check yourself
- Hy4-preview has 64 attention heads,
kv_lora_rank: 512,qk_rope_head_dim: 64,index_n_heads: 32,index_head_dim: 128andindex_topk: 2048. In DeepSeek’s FP8 layout, where does one indexer layer read as many bytes per decoded token as its sparse attention, and where would that move with keys pooled by four? Where is the FLOP crossover?Answer
Bytes: tokens. Pooled by four, the indexer reads 33 B per past token, so about 40,700. FLOPs: , or about 138,000 pooled. Both keep growing linearly beyond. - Why can’t the language-modeling loss train a DSA indexer, what trains it instead, and which
tokens does that loss never correct in the sparse stage?
Answer
Top-k is piecewise constant in the scores, so no gradient reaches them. A KL divergence from the head-summed main attention to the softmax of the indexer scores trains it. In the sparse stage both are taken over the selected set, so unselected tokens get no correction; only the dense warm-up covers every position.
In the map
Methods
- DeepSeek Sparse Attention 16 sources
- Native Sparse Attention 1 source
- Mixture of Block Attention 1 source
- MiniMax Sparse Attention 7 sources
- Qwen Sparse Attention 6 sources
- Compressed Sparse Attention 6 sources
- Heavily Compressed Attention 5 sources
- Compressed Sparse Attention 2 2 sources
- IndexShare 3 sources
- IndexCache 3 sources
- Fine-grained token selection mechanism 2 sources
- Compressed lightweight indexer 1 source
Taxonomy nodes
- sparse attention 24 methods
- fixed-pattern sparse attention 3 methods
- block selection 9 methods
- sparse attention indexer 24 methods
Further reading
- Zaheer et al., BigBird (2020), arXiv:2007.14062; Xiao et al., StreamingLLM (2023), arXiv:2309.17453. Fixed patterns and attention sinks.
- Kitaev et al., Reformer (2020), arXiv:2001.04451; Roy et al., Routing Transformer (2021), arXiv:2003.05997; Wu et al., Memorizing Transformers (2022), arXiv:2203.08913. Earlier content-based selection.
- Lu et al., MoBA (2025), arXiv:2502.13189; Yuan et al., Native Sparse Attention (2025), arXiv:2502.11089. Block selection; NSA’s kernel for GQA groups.
- MiniMax, MiniMax Sparse Attention (2026), arXiv:2606.13392. Detached-KL training, kernels, both training routes.
- DeepSeek-V3.2 report, §2 and App. A, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/resolve/main/assets/paper.pdf. The lightning indexer and the MHA and MQA modes of MLA.
- GLM-5 report, §2.1, arXiv:2602.15763. DSA by continued pretraining; deterministic top-k.
- Mistral Large 4 technical report, §3 and §6, https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/assets/technical_report.pdf. From a compressed cache back to full attention, then DSA; context-parallel serving.
- DeepSeek-V4 report, §2.3 and §3.5, https://arxiv.org/html/2606.19348; DeepSeek-V4.1-Flash report, §2.2 to §2.4, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/resolve/main/DeepSeek_V41_Tech_Report.pdf. CSA, HCA, CSA2, prefix caching with compressed entries.
- Qwen3.8-Next report, §2.1, https://arxiv.org/html/2608.30320; Bai et al., IndexCache (2026), arXiv:2603.12201. Pooled and shared indexers.
- Liu et al. (2018), arXiv:1801.10198; Rae et al., Compressive Transformer (2020), arXiv:1911.05507; Chen et al., Scatterbrain (2021), arXiv:2110.15343. Compressed keys and values; sparse-plus-low-rank attention.
