Open Models Almanac

Token mixer · T3

Sparse attention and indexing

Selecting a few past entries per query cuts attention work, but the selector still reads every entry, misses some of what dense attention would use, and leaves the KV cache full unless entries are compressed

Illustration of a very long wooden bookshelf receding into the distance, every slot filled with plain ink-blue and off-white books. In front of it stands a small wooden card-catalogue cabinet with one drawer open and a brass lamp on top. Teal threads run from the raised cards to six books at scattered places along the shelf, each pulled a little forward and glowing coral; the rest of the shelf stays full.

GLM-5.2 lets each new token attend to 2,048 of the tokens before it (index_topk: 2048), one in 64 at 128K tokens of context. A small scoring network, the indexer, picks them by reading a short key for every one of the 131,072 cached tokens. Generating a token is limited by the bytes the GPU reads, and in bytes that scan costs more than the attention it steers: a GLM-5.2 layer reads 17.8 MiB per generated token instead of 82 MiB, 4.6 times less, not 64 (computed below). Running the indexer in 21 of 78 layers brings the model to 14 times. The cache still holds 6.58 GiB.

Sparse attention runs exact softmax attention over a subset of the past, chosen anew for every query. It keeps exact recall for what it selects and bounds the work per query while the memory keeps growing; T2‘s fixed-state mixers bound the memory and give up exact recall. The selection must cost less than the attention it saves, catch most of the mass dense attention would use, and read memory in shapes a GPU handles well.

Selection buys attention work and cache reads with selector overhead and recall risk, and only compression shrinks the cache.

Background

Prefill processes the prompt all at once; decode then generates one token at a time, each reading the KV cache of every layer. A kernel’s arithmetic intensity is the FLOPs it performs per byte it reads; an H800 does about 590 FP8 FLOPs per byte it reads (vendor peaks), so a kernel far below that is timed by its bytes, as decode attention with one query per sequence is (T1).

Multi-head latent attention (MLA, T1) caches one latent of dc=512d_c = 512 numbers per token, plus a dhR=64d_h^R = 64-number key that carries position; at decode all hh query heads read the same entry. FP8 and FP4 store a number in one byte or half a byte, with scales shared by a group of numbers.

Write StS_t for the set of past positions query tt may use (a set, not T2‘s state matrix), κ\kappa for its size and dhd_h for the head width. Sparse attention computes the softmax over StS_t:

otS=∑s∈Stexp⁡ ⁣(qt⊤ks/dh)∑r∈Stexp⁡ ⁣(qt⊤kr/dh) vs,∣St∣=κ≪T.o^S_t = \sum_{s \in S_t} \frac{\exp\!\big(q_t^\top k_s/\sqrt{d_h}\big)}{\sum_{r \in S_t} \exp\!\big(q_t^\top k_r/\sqrt{d_h}\big)}\, v_s, \qquad |S_t| = \kappa \ll T.

Work and bytes per query grow with κ\kappa, not TT. Let at,sa_{t,s} be the dense weights and εt=∑s∉Stat,s\varepsilon_t = \sum_{s \notin S_t} a_{t,s} the mass outside StS_t. Scores inside StS_t are exact, but each kept weight becomes at,s/(1−εt)a_{t,s}/(1-\varepsilon_t). With oˉt out\bar o^{\,\mathrm{out}}_t the weighted mean value over the skipped positions, the dense output is ot=(1−εt) otS+εt oˉt outo_t = (1-\varepsilon_t)\,o^S_t + \varepsilon_t\,\bar o^{\,\mathrm{out}}_t, so

ot−otS=εt (oˉt out−otS),∥ot−otS∥≤2 εtmax⁡s∥vs∥.o_t - o^S_t = \varepsilon_t\,\big(\bar o^{\,\mathrm{out}}_t - o^S_t\big), \qquad \lVert o_t - o^S_t\rVert \le 2\,\varepsilon_t \max_s \lVert v_s\rVert.

A selector is as good as the mass it misses: the top κ\kappa positions by true score minimize εt\varepsilon_t, the oracle every selector is judged against, and since softmax weights are all positive, εt>0\varepsilon_t > 0 whenever κ<t\kappa < t. StS_t is chosen per query, so a token query tt skips may be the one query t+1000t + 1000 needs, and every key and value stays cached.

budget

fixed pattern sink 1–2 + recent window

missed mass ε 0.42

block top-k own block + best blocks of 4

missed mass ε 0.30

token top-k single positions

missed mass ε, true scores 0.15indexer 0.27

ranked by

dense attention weightselectedquery rowcached (all kept)rows = queries, columns = past positions; click a row to make it the query

Query 28, budget 8 of 28 positions: the fixed pattern misses 42% of the attention mass, blocks 30%, single tokens ranked by the true scores 15% (the least any selector of 8 can miss), ranked by the noisy indexer 27%.

Figure 1.Three ways to spend a budget of eight positions. A fixed pattern misses distant spots; blocks reach them but carry neighbours along; single tokens hit them exactly when ranked by the true scores, the best any selector can do. A learned indexer ranks by an estimate and misses more. The stored cache is the same in all three.

Fixed patterns

The oldest selectors ignore content. BigBird (Zaheer et al., 2020) combines a local window, a few global positions and random blocks. StreamingLLM (Xiao et al., 2023) found that a window alone fails because trained models park attention mass on the first tokens, an attention sink, and that “keeping the KV of initial tokens will largely recover the performance of window attention”; it claims stable perplexity, not retrieval of old facts.

A fixed pattern costs nothing to select, but a fact outside the window and anchors is out of reach. On RULER, synthetic long-context retrieval tasks scored out of 100, the GLM-5 report found that converting a 9B GQA model to alternating window and full layers by continued training causes “catastrophic degradation on long-context tasks (e.g., −30.35 on RULER@128K)”: a 30.35-point drop, from 75.28 to 44.93, with half the layers still full. Fixed choices live on inside the learned designs: MoBA always keeps the query’s own block, NSA the first and two local blocks, DeepSeek-V4 a 128-token window in every layer.

Content-based selection is older than learned indexers: Reformer (Kitaev et al., 2020) hashed queries and keys into buckets, the Routing Transformer (Roy et al., 2021) clustered them, Memorizing Transformers (Wu et al., 2022) attended to the nearest stored keys. Quest (arXiv:2406.10774), whose per-page minimum and maximum keys bound q⊤kq^\top k, and InfLLM (arXiv:2402.04617, “without any training”) select at inference in models trained dense.

Block selection

MoBA

Mixture of Block Attention (MoBA; Lu et al., 2025) splits the keys into blocks of BB tokens, represents each block by the mean of its keys, and lets each query take the top blocks by inner product with those means, plus its own block. It resembles a mixture-of-experts router, except that no gate value scales the chosen blocks, so the choice receives no gradient; the router has no parameters. Since q⊤kˉB=1B∑j∈Bq⊤kjq^\top \bar k_B = \frac1B \sum_{j \in B} q^\top k_j, one token scoring Δ\Delta above its neighbours lifts its block by only Δ/B\Delta/B.

Its authors switched 29 of Llama 3.1 8B’s 32 layers to MoBA for 100B tokens (B=4,096B = 4{,}096, top 12) and report RULER at 128K of 0.7818 against 0.7849 for full attention, with prefill “up to 6.5x” faster at 1M. At 128K those 12 blocks are 37.5% of the context, and generation ran dense.

Native Sparse Attention

NSA (Yuan et al., 2025) gives each query three attention branches, each with its own key and value projections, mixed by gates γtc∈[0,1]\gamma_t^{c} \in [0, 1] from an MLP and a sigmoid:

ot=γtcmp Attn(qt,K~cmp,V~cmp)+γtslc Attn(qt,KStslc,VStslc)+γtwin Attn(qt,Kwin,Vwin).o_t = \gamma_t^{\mathrm{cmp}}\, \mathrm{Attn}\big(q_t, \tilde K^{\mathrm{cmp}}, \tilde V^{\mathrm{cmp}}\big) + \gamma_t^{\mathrm{slc}}\, \mathrm{Attn}\big(q_t, K^{\mathrm{slc}}_{S_t}, V^{\mathrm{slc}}_{S_t}\big) + \gamma_t^{\mathrm{win}}\, \mathrm{Attn}\big(q_t, K^{\mathrm{win}}, V^{\mathrm{win}}\big).

The compression branch attends to coarse tokens, each a small MLP of 32 consecutive keys taken every 16 positions, in effect a learned strided convolution. Its attention weights double as block scores (a 64-token block sums the scores of the coarse tokens overlapping it), and the selection branch attends to the original keys of the 16 best blocks, the first and two local ones by force. The window branch covers the last 512 tokens, because local patterns otherwise “can dominate the learning process”. All heads of a GQA group share one block choice, so a kernel loads each block once per group. Storage is a full cache plus T/16T/16 coarse entries and the window (our reading).

1 · compression — MLP of 8 keys every 4 → 15 coarse tokens— l = 32, stride 164,096 coarse tokensscores2 · selection — top 3 blocks of 8 by those scores— 16 blocks of 64forcedforced16 blocks × 64 = 1,024 tokens (1 initial, 2 local forced)3 · window — last 8 positions— last 512512 tokensAttnγAttnγAttnγ+ot123456780K16K32K48K64Kqueryown K, V per branch (cached)selection: 64selection: 65,536compression: 15compression: 4,096window: 8window: 512Paper (§4.1): l = 32, stride 16, l′ = 64, n = 16 blocks, w = 512.Toy: 8 keys every 4, blocks of 8, 3 picked, w = 8.

Toy: the compression lane reads all keys through 15 coarse tokens (8 keys every 4); its scores pick 3 blocks of 8 (blocks 1, 4, 8; the first and the local one forced); the window reads the last 8. Hover a lane to see the keys it reads.

Figure 2.Native Sparse Attention's three branches. The coarse branch sees everything at low resolution and tells the selection branch where to look; the window keeps the recent past exact.

NSA was trained sparse from scratch in a 27B-parameter mixture-of-experts model with 3B active parameters (the weights one token uses). It reports beating full attention “on 7 out of 9 metrics” and a 9.0× forward speed-up at 64K against a Triton FlashAttention-2 kernel; its 11.6× decode figure is an “expected speedup” from 5,632 tokens loaded instead of 65,536. No model in the map ships NSA, but coarse scores steering a fine selection, a window beside it, and one choice per GQA group all recur.

MiniMax Sparse Attention

MSA (MiniMax, 2026) is block selection on grouped-query attention (GQA), where each key/value head serves a group of query heads. Its Index Branch adds “one index query head for each GQA group and a single index key head shared across groups”. Each index head scores every visible token, takes the maximum inside each 128-token block (the other end of the pooling choice from MoBA’s mean), keeps the query’s own block, and picks the top 16 blocks for its group; the Main Branch runs exact GQA attention over them. In MiniMax-M3 (sparse_block_size: 128, sparse_topk_blocks: 16, sparse_num_index_heads: 4 equal to num_key_value_heads: 4) each of four groups reads 2,048 tokens of its own choosing, in 57 of 60 layers; M3.1-Flash makes all 60 sparse.

The paper reports that “on a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4× at 1M context” (quality compared up to 128K), with kernel speed-ups of 14.2× in prefill and 7.6× in decode. Both follow from the indexer’s cost: in FLOPs it is 16 times the sparse attention at 1M, so dense ÷ (sparse + indexer) ≈ 512/17. In bytes, a dense layer reads 2,048 B per past token, the sparse one 4⋅2,048⋅5124 \cdot 2{,}048 \cdot 512 B of selected entries plus a 256 B index key per past token: 7.9× at 1M, against the measured 7.6× (computed, BF16). Nothing is compressed: M3 stores 60⋅2,048+57⋅256=137,47260 \cdot 2{,}048 + 57 \cdot 256 = 137{,}472 B per token, 16.8 GiB at 128K, so its card’s “dramatically reduces the attention compute and memory footprint” holds for compute only.

DeepSeek Sparse Attention

DeepSeek Sparse Attention (DSA), introduced in DeepSeek-V3.2, selects single tokens. A lightning indexer scores each past token ss for query tt as

It,s=∑j=1hIwt,j ReLU(qt,jI⋅ksI),I_{t,s} = \sum_{j=1}^{h^I} w_{t,j}\, \mathrm{ReLU}\big(q^I_{t,j} \cdot k^I_s\big),

where each of the hIh^I indexer heads has its own query qt,jIq^I_{t,j} of width dhId_h^I, all heads share one key ksIk^I_s per token, and wt,jw_{t,j} comes from a linear layer with no sign constraint, so the score is a small ReLU network of the key rather than a similarity. The main attention takes the top κ\kappa tokens. V3.2’s indexer has 64 heads of 128 dimensions (index_n_heads: 64, index_head_dim: 128); GLM-5.2, GLM-5.3, Hy4-preview and Mistral Large 4 have 32 of 128. All five select κ=2,048\kappa = 2{,}048 (index_topk). By our reading the indexer is cheap mostly through its single shared key, FP8 scoring and a ReLU chosen “for throughput consideration”; its 32 or 64 heads are half the main attention’s 64 or 128.

Every token selector in the map selects from one shared entry per token, MLA’s latent or DeepSeek-V4’s shared key-value entry, so one gather of 2,048 latents (copying rows at scattered positions into one tile) serves all heads. V3.2 also chose MLA because it continued training from an MLA model.

xtcQtquery latent (2,048): feeds both query pathsLIGHTNING INDEXERHadamard + FP8qI: 32 × 128indexer keys: 128 numbers per token · FP8, 132 B131,072 keys read16.5 MiB per stepT entries stored, all scannedReLU(qIt,j · kIs) weighted by wt,j, summed over 32 heads → It,swt,j: 32, from xtscore per past positiontop-2,048no LM-loss gradient(top-k); trained by KLto the main attentionindex list,2,048 positionsMLA IN MQA MODE · 64 HEADS2,048 latents read · 1.28 MiB2,048 × 576T entries stored: latent cs 512 + position key 64656 B per token, FP8 layoutsoftmax over2,048 · Σ values64 heads → 576 eachqueries from cQt, WUK absorbedot

At T = 128K (131,072 tokens), this layer reads per decode step 131,072 indexer keys × 132 B = 16.5 MiB and 2,048 latents × 656 B = 1.28 MiB. Stored per sequence, whole model: 131,072 × (78 × 656 + 21 × 132 = 53,940 B) = 6.58 GiB (FP8 layout).

Hover the indexer or the attention for its FLOPs per decoded token.

Entry sizes in DeepSeek's FP8 layout (vLLM fp8_ds_mla; external, assumed for GLM-5.2). The strip is a toy: 32 past positions, 8 picks standing for 2,048 of T.

Figure 3.One DSA layer at decode. The indexer scores every past token with a single shared 128-number key; the attention then gathers only the 2,048 chosen latents, which all 64 heads share.

Per layer and token the cache holds the latent, plus a 128-number indexer key in layers with an indexer: 1,152 B plus 256 B in BF16. DeepSeek’s FP8 layout takes 656 B (512 FP8 numbers, four FP32 scales, 64 BF16 position numbers) and about 132 B; vLLM’s Mistral Large 4 recipe names it as its default cache (fp8_ds_mla), and with it the V4 report’s cache ratios below come out right. For GLM-5.2 that is 53,940 B per token, 6.58 GiB at 128K (computed).

Training the indexer

Top-kk returns positions that do not change when a score moves slightly, so its gradient with respect to the scores is zero almost everywhere and the language-modeling loss cannot teach a selector what to pick. MoBA has no selector parameters; NSA reuses the scores of a branch trained through its own output; DSA, MSA and QSA fit the indexer to the main attention with a separate loss. None uses a straight-through estimator or a relaxed top-kk.

The target sums the main attention over its hh heads and normalizes over positions, pt,s∝∑i=1hat,s(i)p_{t,s} \propto \sum_{i=1}^{h} a^{(i)}_{t,s}, and the indexer minimizes

LI=∑tDKL(pt,: ∥ softmaxs(It,s)),\mathcal L^I = \sum_t D_{\mathrm{KL}}\Big(p_{t,:}\ \Big\|\ \mathrm{softmax}_s\big(I_{t,s}\big)\Big),

the forward direction, which penalizes most where the main attention has mass and the indexer puts little: a penalty for misses, which is what recall needs. It is distillation of an attention map into a small student (and ListNet’s ranking loss, Cao et al., 2007). V3.2 trained it in two stages: a dense warm-up of 2.1B tokens with everything else frozen, then 943.7B tokens of joint training with top-2,048 selection, where pp and the softmax are taken over StS_t only, so a token the indexer failed to pick gets no correction. The indexer’s input is detached and the target held fixed, so the KL cannot reach the main model; MiniMax found that otherwise “the backbone can lower the KL loss by simplifying the Main Branch attention distribution”. Mistral Large 4 used the same two stages (15.1B and 198B tokens).

GLM-5’s ablation on GLM-4.7-Flash measured the recall cost: after the warm-up alone RULER at 128K fell from 79.21 to 71.35, and after 150B tokens of joint training it stood at 78.86, “only a 0.35-point deficit”. In reinforcement learning the rollout engine and the trainer must select the same tokens; GLM-5 found that non-deterministic top-k kernels, which can break ties differently, “caused drastic performance degradation during RL after only a few steps”, and froze the indexer.

The indexer’s own cost

At decode the attention reads κ\kappa latents, the indexer one key for every past token. In DeepSeek’s FP8 layout that is 656⋅2,048656 \cdot 2{,}048 B against 132⋅T132 \cdot T B per layer, equal at

T=656⋅2048132≈10,200 tokens.T = \frac{656 \cdot 2048}{132} \approx 10{,}200 \text{ tokens}.

Dense attention reads 656 T656\,T B, so in a layer with its own indexer the saving, 656T/(656⋅2048+132T)656T/(656 \cdot 2048 + 132T), approaches 656/132=4.97656/132 = 4.97: 3.8× at 32K, 4.6× at 128K, 4.9× at 1M, while the attention alone reads 16, 64 and 512 times less (computed; 4.5 in BF16). DeepSeek-V3.2, with an indexer in every layer, reads 4.6 times less cache per decode step at 128K. Bytes decide because the scan does 62.5 FLOPs per byte (8,256/1328{,}256/132), a tenth of the H800’s ~590.

FLOPs decide where work is compute-bound: prefill, and verifying several draft tokens at once. Per past token, absorbed dense MLA costs 2h(2dc+dhR)=139,2642h(2d_c + d_h^R) = 139{,}264 FLOPs in GLM-5.2 (score plus value sum, a multiply-add counted as two) and the indexer 2hIdhI+2hI=8,2562h^I d_h^I + 2h^I = 8{,}256, which equals the sparse attention’s 139,264⋅2,048=285139{,}264 \cdot 2{,}048 = 285 MFLOP at

T∗=139,264⋅20488,256≈34,500 tokens,T^* = \frac{139{,}264 \cdot 2048}{8{,}256} \approx 34{,}500 \text{ tokens},

and costs 30 times as much at 1M (computed; V3.2’s constants are both twice GLM-5.2’s). Counting FP8 work at half cost, as the V4.1 report does, moves the crossover to about 69,000.

In prefill, dense MLA does not run absorbed: “the MHA mode is used for training and prefilling” (V3.2 report), at 64⋅1,024=65,53664 \cdot 1{,}024 = 65{,}536 FLOPs per query-key pair in GLM-5.2 (from qk_nope_head_dim: 192, qk_rope_head_dim: 64, v_head_dim: 256), only 7.9 times the indexer’s (V3.2: 5.0), while sparse attention runs absorbed. DSA therefore beats dense prefill only for query positions past about 5,000 tokens (V3.2: 8,700), and at 128K a whole prefill costs 5.2 times less per indexer layer (V3.2: 3.3; computed). Below 2,048 tokens DSA selects everything anyway, and V3.2 runs short prompts in “a masked MHA mode to simulate DSA”.

Neither count includes top-k itself, a pass over all TT scores per query and layer. In nearest-neighbour terms DSA is retrieve-then-rerank, a linear scan with a cheap learned score; LSH or inverted files would need an inner-product score in a fixed space, which this score is not.

dense MLAsparse MLA attentionindexersparse + indexerGLM-5.2 config; per decoded token

at
128K tokens
dense
82 MiB
sparse attention
1.28 MiB
indexer
16.5 MiB
sparse + indexer
17.8 MiB
indexer : sparse
12.88 : 1
dense ÷ (sparse + indexer)
4.61×
Figure 4.Work per generated token in one GLM-5.2 layer. Selection makes the attention flat in context length; the indexer that selects keeps growing, and in bytes it becomes the larger cost past about 10,200 tokens.

When the savings show up

A decode step reads the active weights once for the whole batch and the cache once per sequence. For GLM-5.2 at 128K, assuming 40 GB of FP8 active weights (GLM-5’s 40B active parameters), a step for one sequence reads 46.7 GB with dense MLA and 40.5 GB with DSA, 1.15 times less; for 32 sequences, 4.6 times less (computed; larger batches touch more experts, lowering both). Selection pays at large batch and long context, which is also where the full-size cache caps how many sequences fit: 64 sequences of 128K hold 421 GiB.

Speculative decoding drafts a few tokens cheaply and has the main model verify them in one pass (3 to 5 drafts in the recipes for GLM-5.3, GLM-5.3-Flash, Hy4-preview and Mistral Large 4). Dense attention reads each entry once for all queries of a step; DSA selects per query, so four verified queries gather up to four sets of latents, and a GLM-5.2 layer at 128K saves 3.8 times instead of 4.6 (computed, disjoint selections).

Tensor parallelism splits heads across GPUs, but with one latent and one indexer key per token for all heads, every rank holds the whole cache and runs the whole indexer (“the indexer is replicated across the TP ranks”, vLLM’s V4.1 code). Mistral cites this “KV replication” as its reason to shard the cache along the sequence instead, exchanging candidates before the global top-k.

Cheaper indexers

Sharing indices across layers

IndexCache (Bai et al., 2026) found that in a 30B DSA model “adjacent layers share 70-100% of their selected tokens”. IndexCache keeps an indexer in a few Full layers and lets the Shared layers between them reuse the nearest Full layer’s choice; Z.ai ships it as IndexShare. GLM-5.2 and GLM-5.3 mark 21 of 78 layers “full” in indexer_types (0, 1, 2, then every fourth from 6); a shared layer builds no indexer (self.indexer = None) and caches no indexer keys. Hy4-preview has the same 21/57 split at other offsets, Mistral Large 4 44 full and 28 shared.

With 21 indexers the model-level byte ceiling rises from 4.97 to 78⋅656/(21⋅132)=18.578 \cdot 656/(21 \cdot 132) = 18.5: at 128K a decode step reads 446 MiB (346.5 MiB of indexer keys, 99.9 MiB of latents) against 6.25 GiB dense, 14.3 times less, and the indexer passes the attention in bytes at 37,800 tokens instead of 10,200 (computed). The 128K cache drops from 7.50 to 6.58 GiB.

Sharing loses recall when layers disagree: one indexer in four in a uniform pattern without retraining lowered IndexCache’s long-context average from 50.2 to 43.0, a searched pattern recovered 49.9, and training each kept indexer on the attention of the layers it serves stayed “within 0.4% of the baseline”. A separate flag, index_share_for_mtp_iteration, reuses the top-k of the multi-token-prediction draft layer across its own draft steps; the main model still selects for every verified token.

A · across layers IndexShare in GLM-5.2's first 14 layers

indexers
21 / 78 layers
decode reads at 128K, dense ÷ DSA
14.33×
indexer passes attention in bytes at
37,804 tokens

B · within a layer pooled indexer keys

pool size

pool 4: mean (QSA, Qwen3.8-Flash) or learned softmax (GLM-5.3-Flash)
262,144 entries scored per query at T = 1M

indexerindexer keypooled keycached (hatched)read

indexer FLOPs per past token: 8,256 (GLM-5.2), 1,028 (Step-5-preview, per 4 KV groups), 257 (Qwen3.8-Flash)

Figure 5.Two ways to make the indexer cheaper. Sharing runs it in a few layers and hands the choice down; pooling makes each indexer score one entry per group of tokens.

Pooled indexers

Qwen3.8-Flash shrinks the indexer itself. Its Qwen Sparse Attention (QSA) mean-pools the indexer’s keys over micro-blocks of four tokens (indexer_compress_ratio: 4), normalizes the pooled key, and scores it with four ReLU heads of 128 dimensions (indexer_n_heads: 4), summed without learned weights. It picks 512 blocks, 2,048 tokens (indexer_budget: 2048), plus the current incomplete block, whose uncompressed keys and values the main attention reads. The indexer costs 257 FLOPs per past token, 32 times less than GLM-5.2’s, and SGLang caches one pooled key per four tokens. QSA fills the 12 global layers (24 query heads, 2 key/value heads of width 256) of a Gated DeltaNet hybrid (T2).

The Qwen3.8-Next report motivates micro-blocks “so that the cost of indexing itself falls with sequence length” and states the gain as “indexer cost from O(n2)O(n^2) to O(n2/r)O(n^2/r)”: a factor r=4r = 4 on a term that still grows quadratically. QSA’s indexer equals its sparse attention in FLOPs at about 196K tokens and costs 5.4 times it at 1M (computed). In a 35B-A3B ablation (3B of 35B parameters active), “QSA matches the full-attention baseline at a relative indexer latency of 0.25, whereas IndexShare remains below the baseline at 0.5”, with the layers sharing an index four apart.

GLM-5.3-Flash pools inside the DSA code path: index_kpool: 4 merges raw indexer keys four at a time with a learned softmax gate, keeps 32 weighted heads, and selects 512 pools, in 11 sparse MLA layers (without a position key) beside 34 Kimi Delta Attention layers. Pooling cuts its indexer-to-attention FLOP ratio at 1M from 32 to 8.1 (computed, 64 query heads). Step-5-preview (config and serving code only) merges indexer keys over regions of eight tokens with a learned softmax, stored in FP8. Its 16 indexer heads of width 256 form four groups, each picking 512 regions, up to 4,096 tokens, for its own key/value head (num_provider_groups: 4): like MSA it selects per KV group, at 1,028 indexer FLOPs per past token.

Compressed Sparse Attention and Heavily Compressed Attention

Selection and compression serve different heads. A top-κ\kappa set captures at most κmax⁡sat,s\kappa \max_s a_{t,s} of a head’s mass, hopeless for a head that spreads its weight over thousands of positions, which a coarse average approximates well (our reading; the sparse-plus-low-rank idea of Scatterbrain, Chen et al., 2021). NSA mixes both in one layer; DeepSeek-V4 alternates layer types (compress_ratios).

Compressed Sparse Attention (CSA) turns every m=4m = 4 tokens into one cached entry, a per-channel softmax-weighted sum over a window of eight tokens moved four at a time, with weights computed from each token’s hidden state plus a learned bias per slot. This is learned strided pooling, as in memory-compressed attention (Liu et al., 2018) and the Compressive Transformer (Rae et al., 2020). An entry has 512 dimensions and is both key and value for all heads (num_key_value_heads: 1, head_dim: 512). A lightning indexer with compressed keys of its own picks 512 entries in V4-Flash and 1,024 in V4-Pro (index_topk), the equivalent of 2,048 or 4,096 tokens. Heavily Compressed Attention (HCA) compresses every 128 tokens without overlap and attends to all its entries.

Both add an uncompressed 128-token window to the same softmax, because “a query cannot access information from other tokens within its own compressed block”, and a learned sink logit per head, T2‘s stand-in for StreamingLLM’s sink tokens. V4-Flash has 2 window-only, 21 CSA and 20 HCA layers; V4-Pro 30 CSA and 31 HCA.

  • stored entry
  • read by the query
  • indexer
  • sink logit
CSA layer · 1 entry per 4 tokens, pooled over 832,768 entries · reads 512 + 128indexer: top-kown group: not yet an entry ↘window, uncompressed (128)querywindow, uncompressed (128)own group (42 tokens): not yet an entry ↑HCA layer · 1 entry per 128 tokens, no overlap, all read1,024 entries · reads all + 128softmaxsoftmax

V4-Flash: 2 window-only + 21 CSA + 20 HCA layers (43; compress_ratios). At 128K = 131,072 tokens a CSA layer stores 131,072 / 4 = 32,768 entries and reads the top 512 (index_topk) plus the 128-token window; an HCA layer stores 131,072 / 128 = 1,024 and reads them all plus the window.

Toy strip: 298 tokens at the real ratios and window; the indexer picks 6 entries here. Each CSA entry pools the 8 tokens under its bracket (brackets alternate rows so the overlap shows); the stored entries and the window are both cache, drawn hatched.

Figure 6.DeepSeek-V4's two compressed layers. CSA compresses lightly and selects; HCA compresses heavily and reads everything; a 128-token window in both covers the tokens not yet compressed.

Deployed, an entry keeps 448 dimensions in FP8 and 64 position dimensions in BF16 (576 B, scale bytes not stated) and an indexer key takes 68 B in 4-bit MXFP4. HCA layers store no indexer keys and the window is a fixed ring, so V4-Flash stores per token

21⋅576+684+20⋅576128=3,471 B,21 \cdot \frac{576 + 68}{4} + 20 \cdot \frac{576}{128} = 3{,}471 \text{ B},

437 MiB at 128K and 3.39 GiB at 1M, plus 3.0 MiB of window (computed). Against V3.2 at 1M in its FP8 layout that is 7.2%, and 10.3% for V4-Pro; the V4 report says 7% and 10%.

The mixing weights inside an entry are fixed when it is written, so no later query can reweight the tokens it merged. The indexer, scoring T/4T/4 entries with 64 heads, costs 52 times its CSA layer’s attention (64 query heads over 512 entries plus the window) in FLOPs at 1M (13 with FP4 work weighted at a quarter; computed). V4 rounds index scores to BF16 for “a 2× speedup for the top-k selector, while preserving a 99.7% recall rate of KV entries”, recall meaning agreement with the top-k chosen at higher precision.

Compressed entries change serving. A prefix-cache hit reuses entries “until the last complete compression block” and recomputes the tail, and V4 treats the window and that tail “as a state-space model”, a per-sequence state like T2‘s hybrids. V4.1 keeps no window keys for cached prefixes and replays the last 128 tokens instead, taking its persistent cache to “roughly 1/8” of V4-Flash’s.

CSA2

DeepSeek-V4.1-Flash has two window-only layers (0, 1) and 38 CSA2 layers, a CSA without overlap, at ratio 2 in layers 2 to 19 and ratio 1 (no compression) in 20 to 39. Full layers (2, 8, 14, 20; kv_source_layer_ids) store entries and run an indexer; Reindex layers (24, 28, 32, 36) rescore an earlier layer’s entries with their own indexer; the rest reuse both entries and top-k. Every layer keeps its own queries and window. Layers 20 to 39 project their entries from layer 19’s output (“Causal Encoder-Decoder”), so prefill runs only the lower half. With main entries in FP4 (288 B: 4-bit numbers, an 8-bit scale per 16 channels) and indexer keys assumed in MXFP4 (68 B, as in V4),

3⋅288+682+288+681=890 B per token,3 \cdot \frac{288 + 68}{2} + \frac{288 + 68}{1} = 890 \text{ B per token},

the report’s figure exactly, and 0.256 of V4-Flash’s 3,471 B (“roughly 1/4”). If every CSA2 layer stored its own entries in V4’s format, the lighter compression would cost 18,676 B, 5.4 times V4-Flash; reuse across layers then saves 11.6 times and FP4 1.8 times (computed). T5 covers what layers can share.

V4.1 also bounds the indexer with a two-stage cascade, the inverted-file idea: layer 20 scans every position and publishes its 2,048 best blocks of eight, and the Reindex layers score only those 16,384 candidates, which “changes the per-query cost of deeper indexers from linear in context length to constant”.

Native or retrofitted

A retrofit’s parity claim compares it with the dense model it came from; a model trained sparse from the start needs a dense twin. DeepSeek alone has done both: V3.2 is a retrofit, V4.1-Flash was trained sparse from scratch.

Model or methodWhen sparsity enteredSource
NSA (27B test model)from scratchNSA paper: “Both Full Attention and sparse attention models are pretrained on 270B tokens”
MoBA (Llama 3.1 8B)retrofit after 1M-context continued pretraining“MoBA is activated for 100B tokens”
MSA paper (109B test model)both: from scratch (MSA-PT) and from a GQA checkpoint (MSA-CPT, 400B tokens)“supports both training from scratch and near-lossless conversion from pretrained GQA checkpoints”
MiniMax-M3, M3.1-Flashnot disclosednone
DeepSeek-V3.2 (DSA)retrofit by continued training of V3.1-Terminus“through continued training”
GLM-5 (DSA)retrofit at the end of mid-training“introduced via Continued Pre-Training from a dense base model”
GLM-5.2, GLM-5.3, Hy4-previewnot disclosednone
Mistral Large 4 (DSA)retrofit after converting a sliding-window and KV-sharing design to full attention: 15.1B-token warm-up, 198B sparse“we adapted ML4 to DSA”
DeepSeek-V4 (CSA, HCA)during pretraining from 64K sequence length: V4-Flash after 1T dense tokens, V4-Pro after “a longer stage of dense attention”“introduce sparse attention at the sequence length of 64K”
DeepSeek-V4.1-Flash (CSA2)from scratch; hierarchical indexer in post-training“We train the model from scratch with sparse attention”
Qwen3.8-Flash (QSA)retrofit at continued pretraining, 256K sequences“introduced during the continued pretraining (CPT) stage”
IndexCacheon an existing DSA model, with or without trainingIndexCache abstract

Only the MSA paper reports both routes on one model: on RULER at 8K, 84.2 from scratch, 77.2 converted, 79.8 for full attention. Mistral’s first design cut the cache 16-fold with sliding windows and cross-layer sharing; agentic long-context scores regressed until it moved back to full attention and then to DSA (report §3).

Comparison

Fixed patternMoBANSAMSADSAPooled indexerCSAHCACSA2
Unit selectedfixed positionsblock (4,096 in the paper’s run)block of 64, plus coarse tokens and windowblock of 128, per GQA grouptoken, per query4 tokens per query (QSA, GLM-5.3-Flash); 8 tokens per KV group (Step-5)compressed entry (1 per 4 tokens)none (reads all entries)entry (1 per 2 or 1 tokens), shared across layers
Selectornonemean-pooled keys, no parameterscompression-branch scoresindex branch, max-pooledlightning indexer, weighted ReLUindexer on pooled keyslightning indexer on compressed keysnoneindexer in Full and Reindex layers
Per querywindow + anchorstop-k blocks + own blockthree gated branches2,048 tokens per group2,048 tokens2,048 (Qwen, GLM) or 4,096 per group (Step-5) tokens512 or 1,024 entries + 128 windowT/128T/128 entries + 128 window512 entries + 128 window
KV cachebounded (StreamingLLM) or fullfullfull, plus coarse tokensfullfullfull1/4, mixed FP8/BF161/128890 B/token, FP4
Indexer FLOPs per past tokennoneone dot product per blockcoarse tokens only1,024 (M3)8,256 (GLM-5.2) to 16,512 (V3.2)257 (Qwen3.8), 1,028 (Step-5)16,512 per 4 tokens (V4-Flash)none8,256 per entry: layers 2, 8, 14 over T/2T/2, layer 20 over TT, layers 24–36 over 16,384 candidates
Main cost or riskcontent-blindcoarse blocks dilute single tokensthree caches and kernelsindexer reads every tokenindexer reads dominate past ~10K; gathers÷4 or ÷8 on a still quadratic termaveraged tokens; indexer at 1Mvery coarsereuse depends on layer similarity
In the mapwindows of hybrids (T2)nonenoneMiniMax-M3, M3.1-FlashV3.2, GLM-5.2, GLM-5.3, Hy4-preview, Mistral Large 4Qwen3.8-Flash, GLM-5.3-Flash, Step-5-preview5 DeepSeek-V4 modelsthe same 5DeepSeek-V4.1-Flash and its dated -1017 checkpoint

Where the models in the map sit

The site’s code check lightning-indexer-sparse-attention (“a lightweight indexer scores past tokens and each query attends only to the top-k tokens it selects”) is present for 17 of 35 models from seven labs; block selectors satisfy the wording too. Five run DSA over uncompressed tokens (V3.2, GLM-5.2, GLM-5.3, Hy4-preview, Mistral Large 4; four labs); seven DeepSeek-V4 and V4.1 variants run it over compressed entries; GLM-5.3-Flash and Qwen3.8-Flash pool per query; MiniMax-M3, M3.1-Flash and Step-5-preview select per KV group. We reread each verdict against the code and counted Step-3.7-Flash’s “unclear” as absent (its config enables no sparse attention). Mistral Large 4 also shares one latent among layers 35, 39, …, 71 (kv_source_layers), a T5 topic.

The table gives every cache in BF16, with the deployed size where a source documents it, how many times fewer cache bytes a decode step reads than with its sparse layers dense (BF16), and the indexer’s FLOPs against its own layer’s attention at 1M (computed from each config.json; † beyond the configured context).

ModelUnit selectedκB/token, BF16B/token, deployed (format)Fixed part, BF16128K, BF161M, BF16Decode reads ÷, 128KIndexer : attention FLOPs, 1M
DeepSeek-V3.2token, indexer in all 61 layers2,04885,88848,068 (FP8 layout)none10.5 GiB83.9 GiB †4.2×30.4
GLM-5.2, GLM-5.3token, 21 indexers in 78 layers2,04895,23253,940 (FP8 layout; FP8 cache in vLLM’s GLM-5.3 recipe)none11.6 GiB93.0 GiB13.3×30.4
Hy4-previewtoken, 21 indexers in 78 layers2,04895,232not statednone11.6 GiB93.0 GiB13.3×30.4
Mistral Large 4token, 44 indexers in 72 layers2,04882,68845,984 (fp8_ds_mla, vLLM recipe)none10.1 GiB80.8 GiB6.4×30.4 (32.3 without position key)
GLM-5.3-Flashpool of 4, 11 sparse layers512 pools11,968 (pooled keys; HF code 16,918)not stated140.8 MiB (KDA)1.60 GiB11.8 GiB6.2×8.1
Qwen3.8-Flashblock of 4, 12 QSA layers512 blocks25,344 (pooled keys, as SGLang; HF code 27,648)not stated110.1 MiB (GDN)3.20 GiB24.9 GiB †12.5×5.4
Step-5-previewregion of 8 per KV group, 23 layers512 regions72,12871,392 (FP8 region keys, serving code)103.5 MiB (window)8.91 GiB70.5 GiB18.9×5.3
MiniMax-M3block of 128 per KV group, 57 layers16 blocks137,472not statednone16.8 GiB134.3 GiB5.5×16.0
MiniMax-M3.1-Flashblock of 128 per KV group, 60 layers16 blocks138,240not statednone16.9 GiB135.0 GiB7.1×16.0
DeepSeek-V4-Flash (3 variants)entry of 4 (CSA) / all of 128 (HCA)5126,8803,471 (FP8/BF16 + MXFP4, V4 report)5.4 MiB (window)865 MiB6.72 GiB3.4×51.6
DeepSeek-V4-Pro (2 variants)entry of 4 (CSA) / all of 128 (HCA)1,0249,8484,969.5 (FP8/BF16 + MXFP4, V4 report)7.6 MiB (window)1.21 GiB9.62 GiB3.2×14.3
DeepSeek-V4.1-Flash (2 variants)entry of 2 or 1, shared5123,200890 (FP4 + MXFP4, V4.1 report)5.0 MiB (window)405 MiB3.13 GiB31.0×n/a

Notes: “FP8 layout” is DeepSeek’s 656 + 132 B format (Mistral’s entries without a position key counted at 528 B). “Pooled keys” caches one indexer key per pool, as SGLang does for QSA. The fixed part is one sequence’s in-memory window or recurrent state. Multi-token-prediction layers are left out (+1.5% for GLM-5.2). “Decode reads ÷” compares each layer with reading every entry it could select from, which for V4.1 includes the uncompressed entries of layers 20 to 39. Qwen3.8-Flash’s config stops at 262,144 tokens, though its report evaluates to 1M.

Token and block selection leave the cache at dense size, 12 to 138 KB per token in BF16, lowest in the two linear-attention hybrids, where three quarters of the layers hold a fixed state (T2). In deployed formats V4-Flash stores 15.5 times less than GLM-5.2 per token, V4.1-Flash 61 times less (computed).

per-sequence total at
  • BF16
  • documented deployed format
  • label: BF16 GiB per sequence incl. fixed part · † beyond configured context
1 kB10 kB100 kBtoken selection (DSA)DeepSeek-V3.2: BF16 85,888 B per token — 61 layers × latent (512 + 64) × 2 B + 61 indexer layers × 128 × 2 B = 85,888 B (kv_lora_rank, qk_rope_head_dim, index_head_dim, indexer_types). Fixed part: no fixed part. Deployed (DeepSeek's FP8 layout (FlashMLA)): 61 × 656 B (512 FP8 + 4 FP32 scales + 64 BF16) + 61 × 132 B (FP8 key + FP32 scale) = 48,068 B. At 128K: 131,072 × 85,888 B = 10.5 GiB in BF16.DeepSeek-V3.210.5 GiBGLM-5.2, GLM-5.3: BF16 95,232 B per token — 78 layers × latent (512 + 64) × 2 B + 21 indexer layers × 128 × 2 B = 95,232 B (kv_lora_rank, qk_rope_head_dim, index_head_dim, indexer_types). Fixed part: no fixed part. Deployed (FP8 layout; FP8 cache in vLLM's GLM-5.3 recipe): 78 × 656 B (512 FP8 + 4 FP32 scales + 64 BF16) + 21 × 132 B (FP8 key + FP32 scale) = 53,940 B. At 128K: 131,072 × 95,232 B = 11.6 GiB in BF16.GLM-5.2, GLM-5.311.6 GiBHy4-preview: BF16 95,232 B per token — 78 layers × latent (512 + 64) × 2 B + 21 indexer layers × 128 × 2 B = 95,232 B (kv_lora_rank, qk_rope_head_dim, index_head_dim, indexer_types). Fixed part: no fixed part. Deployed format: not stated. At 128K: 131,072 × 95,232 B = 11.6 GiB in BF16.Hy4-preview11.6 GiBMistral Large 4: BF16 82,688 B per token — 54 layers × latent (512 + 64) × 2 B + 9 NoPE × 1024 + 44 indexer layers × 128 × 2 B = 82,688 B (9 of 72 layers reuse another's latent, kv_source_layers) (kv_lora_rank, qk_rope_head_dim, index_head_dim, indexer_types). Fixed part: no fixed part. Deployed (fp8_ds_mla, vLLM recipe): 54 × 656 B (512 FP8 + 4 FP32 scales + 64 BF16) + 9 × 528 + 44 × 132 B (FP8 key + FP32 scale) = 45,984 B. At 128K: 131,072 × 82,688 B = 10.1 GiB in BF16.Mistral Large 410.1 GiBpooled indexersGLM-5.3-Flash: BF16 11,968 B per token — 11 sparse layers × (latent 512 + 0 × 2 B + pooled indexer key 128 × 2 B ÷ 4) = 11,968 B (kv_lora_rank, qk_rope_head_dim, index_head_dim, index_kpool; the HF code caches raw keys: 16,918 B). Fixed part: 34 KDA layers × (state 64 × 128² × 4 B + conv 3 × 64 × 128 × 3 × 2 B) = 140.8 MiB. Deployed format: not stated. At 128K: 131,072 × 11,968 B + fixed part = 1.60 GiB in BF16.GLM-5.3-Flash1.60 GiBQwen3.8-Flash: BF16 25,344 B per token — 12 QSA layers × (K, V 2 × 2 × 256 × 2 B + pooled indexer key 128 × 2 B ÷ 4) = 25,344 B (num_key_value_heads, head_dim, indexer_head_dim, indexer_compress_ratio; HF caches raw keys: 27,648 B). Fixed part: 36 GDN layers × (state 48 × 128 × 128 × 4 B + conv (2·16·128 + 48·128) × 3 × 2 B) = 110.1 MiB. Deployed format: not stated. At 128K: 131,072 × 25,344 B + fixed part = 3.20 GiB in BF16.Qwen3.8-Flash3.20 GiBselection per KV groupStep-5-preview: BF16 72,128 B per token — 23 full layers × (K, V 2 × 4 × 192 × 2 B + region summary 256 × 2 B ÷ 8) = 72,128 B (num_attention_groups, head_dim, sparse_config.proxy_dim, region_block_size). Fixed part: 69 SWA layers × window 512 × 3072 B = 103.5 MiB (sliding_window). Deployed (FP8 region keys, serving code): 23 × (3072 B BF16 K, V + 256 B FP8 summary ÷ 8) = 71,392 B. At 128K: 131,072 × 72,128 B + fixed part = 8.91 GiB in BF16.Step-5-preview8.91 GiBMiniMax-M3: BF16 137,472 B per token — 60 layers × K, V 2 × 4 × 128 × 2 B + 57 sparse layers × index key 128 × 2 B = 137,472 B (num_key_value_heads, head_dim, sparse_index_dim, sparse_attention_freq). Fixed part: no fixed part. Deployed format: not stated. At 128K: 131,072 × 137,472 B = 16.8 GiB in BF16.MiniMax-M316.8 GiBMiniMax-M3.1-Flash: BF16 138,240 B per token — 60 layers × K, V 2 × 4 × 128 × 2 B + 60 sparse layers × index key 128 × 2 B = 138,240 B (num_key_value_heads, head_dim, sparse_index_dim, sparse_attention_freq). Fixed part: no fixed part. Deployed format: not stated. At 128K: 131,072 × 138,240 B = 16.9 GiB in BF16.MiniMax-M3.1-Flash16.9 GiBcompressed firstDeepSeek-V4-Flash: BF16 6,880 B per token — 21 CSA × (1024 + 256) B ÷ 4 + 20 HCA × 1024 B ÷ 128 = 6,880 B (compress_ratios, head_dim, index_head_dim). Fixed part: 43 layers × window 128 × 1024 B = 5.4 MiB (sliding_window). Deployed (FP8/BF16 + MXFP4, V4 report): 21 × (576 B FP8/BF16 entry + 68 B MXFP4 key) ÷ 4 + 20 × 576 B ÷ 128 = 3,471 B. At 128K: 131,072 × 6,880 B + fixed part = 865 MiB in BF16.DeepSeek-V4-Flash865 MiBDeepSeek-V4-Pro: BF16 9,848 B per token — 30 CSA × (1024 + 256) B ÷ 4 + 31 HCA × 1024 B ÷ 128 = 9,848 B (compress_ratios, head_dim, index_head_dim). Fixed part: 61 layers × window 128 × 1024 B = 7.6 MiB (sliding_window). Deployed (FP8/BF16 + MXFP4, V4 report): 30 × (576 B FP8/BF16 entry + 68 B MXFP4 key) ÷ 4 + 31 × 576 B ÷ 128 = 4,969.5 B. At 128K: 131,072 × 9,848 B + fixed part = 1.21 GiB in BF16.DeepSeek-V4-Pro1.21 GiBDeepSeek-V4.1-Flash: BF16 3,200 B per token — source layers 2, 8, 14, 20 only: 1280/2 + 1280/2 + 1280/2 + 1280/1 = 3,200 B (kv_source_layer_ids, compress_ratios, head_dim 512, index_head_dim 128). Fixed part: 40 layers × window 128 × 1024 B = 5.0 MiB. Deployed (FP4 + MXFP4, V4.1 report): 356/2 + 356/2 + 356/2 + 356/1 (288 B FP4 entry + 68 B MXFP4 key) = 890 B. At 128K: 131,072 × 3,200 B + fixed part = 405 MiB in BF16.DeepSeek-V4.1-Flash405 MiBcache bytes per token, growing part (log scale; 1 kB = 1,000 B)

Selection keeps 11,968 B to 138,240 B per token in BF16; hover or focus a row for its formula and config fields.

Figure 7.Cache bytes per token for the sparse-attention models in the map, in BF16 (outlined) and in a documented deployed format (solid). Compressing first is what moves a model down the axis.

Easy to get wrong

  • 64 times fewer attention FLOPs is not 64 times faster decode. With an indexer in every layer, decode reads fall by at most about 5 times, and the indexer reads more than the attention from about 10,000 tokens, not 34,500.
  • The savings depend on batch size. At batch 1 the weights dominate the bytes read, and GLM-5.2 at 128K reads about 13% less per step.
  • Below κ tokens DSA is dense attention: every prompt shorter than 2,048 tokens selects everything.
  • GLM-5’s “−30.35 on RULER@128K” for sliding windows is a drop from 75.28 to 44.93, not a score. Its “lossless by construction” for DSA cannot hold exactly, since ε>0\varepsilon > 0 whenever κ<t\kappa < t; its own ablation shows a 0.35-point gap.
  • Not every learned selector selects per query: MSA and Step-5-preview choose per KV group.
  • MSA’s “on par with GQA” is a 109B test model evaluated up to 128K, and MiniMax-M3’s “memory footprint” claim goes beyond the paper: M3 stores a full cache.

Check yourself

  1. Hy4-preview has 64 attention heads, kv_lora_rank: 512, qk_rope_head_dim: 64, index_n_heads: 32, index_head_dim: 128 and index_topk: 2048. In DeepSeek’s FP8 layout, where does one indexer layer read as many bytes per decoded token as its sparse attention, and where would that move with keys pooled by four? Where is the FLOP crossover?
    AnswerBytes: 656⋅2048/132≈10,200656 \cdot 2048 / 132 \approx 10{,}200 tokens. Pooled by four, the indexer reads 33 B per past token, so about 40,700. FLOPs: 139,264⋅2048/8,256≈34,500139{,}264 \cdot 2048 / 8{,}256 \approx 34{,}500, or about 138,000 pooled. Both keep growing linearly beyond.
  2. Why can’t the language-modeling loss train a DSA indexer, what trains it instead, and which tokens does that loss never correct in the sparse stage?
    AnswerTop-k is piecewise constant in the scores, so no gradient reaches them. A KL divergence from the head-summed main attention to the softmax of the indexer scores trains it. In the sparse stage both are taken over the selected set, so unselected tokens get no correction; only the dense warm-up covers every position.

In the map

Methods

Taxonomy nodes

Further reading

↑