Open Models Almanac

Token mixer · T1

Full attention on a memory budget

MHA, MQA, GQA and MLA trade capacity or compute for a smaller KV cache

Illustration of a long library shelf. On the left, rows of identical thick ledgers; towards the right, the ledgers give way to a few slim open notebooks, each read by a cluster of small reading lamps, and finally one small folded card read by a ring of lamps.

Suppose you serve a model the size of DeepSeek-V3 (61 layers, 128 attention heads of 128 dimensions each) but with the attention of the original Transformer. A single conversation of 128K tokens then needs 488 GiB of GPU memory just to remember its own past. That is more than six 80 GB H100s for one user, before a single weight is loaded. No model in this map comes close. The ones with exact attention in every layer store 2 to 3% of that, and they still attend to every past token.

This article explains how. The four designs it covers (multi-head, multi-query, grouped-query and multi-head latent attention) keep exact softmax attention over the whole context and change only what is stored per token. Each one gives something up to shrink that store.

What the KV cache is, and why it sets decode speed

Language models are run in two phases. Prefill processes the prompt: all its tokens go through the network at once, in large matrix multiplications. Decode then generates the answer one token at a time. To produce token tt, every attention layer compares the new token’s query with the keys of all earlier tokens and takes a weighted sum of their values. Those keys and values never change once computed, so inference engines keep them in GPU memory. This store is the KV cache.

With gg key/value heads of dimension dhd_h, stored at bb bytes per number, each token adds

bytes per token per layer=2⋅g⋅dh⋅b\text{bytes per token per layer} = 2 \cdot g \cdot d_h \cdot b

to the cache (the 2 is for keys and values). Multiply by the number of layers LL and the context length TT and you have the cache for one sequence. With g=128g=128, dh=128d_h=128, b=2b=2 (BF16) and L=61L=61 this is 3.8 MiB per token, which is where the 488 GiB come from.

Decoding a token also reads the whole cache once, and that sets its speed. Count the work for one head and one past token. Reading its key and value means moving 2dh2 d_h numbers, 4dh4 d_h bytes in BF16. Using them costs one dot product with the query and one scaled addition of the value, 2dh2 d_h multiply-adds or 4dh4 d_h floating-point operations (counting a multiply-add as two and ignoring the softmax). That is one operation per byte. An H100 can do about 990 TFLOP/s of BF16 arithmetic but read only 3.35 TB/s from its memory, so it needs roughly 300 operations per byte to keep its arithmetic units busy. At one operation per byte, decode attention runs at well under 1% of the chip’s arithmetic throughput, and its speed is set almost entirely by how many bytes the cache has. Engineers call this regime memory-bound. The ratio of operations to bytes is the arithmetic intensity.

Batching many users together doesn’t help here, unlike for the weight matrices. A weight matrix read once from memory serves every sequence in the batch, so a bigger batch means more work per byte. Each sequence has its own KV cache, so a bigger batch adds cache bytes exactly as fast as it adds work.

Preset
Bytes per number b

weightskeysvalueshatched = cached state · axis 0 to 500 GiB, linear

per token per layer
2 × 128 × 128 × 2 B = 64 KiB
per token (× L)
3.8 MiB
KV cache (× T × batch)
488 GiB
total with weights
488 GiB = 6.5× one 80 GB GPU
Figure 1.One 128K-token sequence with full multi-head attention would fill more than six GPUs. Move the KV-heads slider to see what sharing buys.

Every technique below shrinks the 2⋅g⋅dh2 \cdot g \cdot d_h part of the formula. Other articles in the series shrink TT, by keeping only recent tokens (T2) or selecting a few (T3), and LL, by making only some layers global (T5).

Multi-head attention: the baseline

In multi-head attention (MHA) each of the hh heads has its own projections. For an input xt∈Rdx_t \in \mathbb{R}^d and head ii:

qt,i=WiQxt,kt,i=WiKxt,vt,i=WiVxt.q_{t,i} = W^Q_i x_t,\quad k_{t,i} = W^K_i x_t,\quad v_{t,i} = W^V_i x_t .

Every head has its own keys and values, so g=hg = h and the cache holds 2hdh2 h d_h numbers per token per layer. Many models choose hdh=dh d_h = d, which makes this two copies of the model width. DeepSeek-V3 doesn’t: hdh=16,384h d_h = 16{,}384 against d=7,168d = 7{,}168, which is why its hypothetical MHA cache is so large.

Separate heads let each one define its own notion of relevance (keys) and bring back its own information (values). Probing studies find heads specialized for, for example, the previous token or syntactic relations (Clark et al., 2019). That flexibility is what the next three designs ration.

Multi-query attention: one key/value head for all

Shazeer (2019) proposed sharing a single key head and a single value head across all query heads. In multi-query attention (MQA) the queries stay per head, but

kt=WKxt,vt=WVxt(shared by all h heads).k_t = W^K x_t,\quad v_t = W^V x_t \quad\text{(shared by all } h \text{ heads)}.

The cache shrinks by a factor of hh, from 3.8 MiB to 30.5 KiB per token in our example. Decoding speeds up for the same reason: all hh heads use the same bytes, so the GPU does hh times as much work per byte read (an intensity of 128 instead of 1). The FLOPs per token stay about the same; what falls is the memory traffic.

Each head can still ask its own question, but all heads now search the same keys and retrieve from the same values. Ainslie et al. (2023) report that MQA loses quality against MHA and can make training less stable.

There is also a systems cost. Large models are usually run with tensor parallelism: each layer is split across several GPUs, and for attention the natural split is by heads, so each GPU holds the keys and values of its own heads. With one KV head there is nothing to split, and every GPU must hold a full copy of the cache.

Grouped-query attention: a dial between the two

Grouped-query attention (GQA; Ainslie et al., 2023) splits the hh query heads into gg groups, and each group shares one key head and one value head. g=hg = h is MHA, g=1g = 1 is MQA, and values in between trade cache size against capacity.

g = 4 (GQA)

Hover a query head to see the pair it reads.Each GPU holds 1 of the 4 pairs; nothing is duplicated.

Real models have 32 to 128 query heads; 16 keep the lines readable.

Figure 2.GQA interpolates between multi-head attention, where every query head has its own keys and values, and multi-query attention, with one shared pair. Only the bottom row is cached. With fewer KV heads than GPUs, the cache has to be copied.

Two findings made GQA the default. With 8 groups, Ainslie et al. obtained quality close to MHA at a cache cost close to MQA. And an existing MHA checkpoint can be converted: average the key and value heads within each group, then continue training the converted model briefly (they used about 5% of the original pre-training compute).

Among the models in the map whose layout the code check settles cleanly, 16 of 22 use GQA. (The other six are MLA models; the six DeepSeek-V4 variants use a different layout, described below, and are counted separately.) The number of KV heads is small. Qwen3.5, Qwen3.6, Qwen3.8 and both Nemotron 3 models use 2, MiniMax-M3 uses 4, and gpt-oss, Laguna, MiMo, Hy3 and Inkling use 8 (num_key_value_heads in each config.json). With 2 KV heads and 32 query heads, each key and value is shared by 16 queries.

The tensor-parallel problem from MQA comes back here. Qwen3.5’s 2 KV heads can be split over at most 2 GPUs; on 8 GPUs each KV head is copied four times. In practice, how far you want to shard a model limits how low gg goes.

What sharing costs: a rank view

“Some capacity” can be made precise. Stack the key projections of all heads into one map xt↦[kt,1;… ;kt,h]x_t \mapsto [k_{t,1}; \dots ; k_{t,h}]. Whatever the heads do with the keys afterwards, everything they can match on is contained in the image of this map, whose dimension is at most

  • min⁡(d,hdh)\min(d, h d_h) for MHA (7,168 in the DeepSeek-V3 shape),
  • g⋅dhg \cdot d_h for GQA (1,024 with 8 groups), because heads in a group have identical keys,
  • dhd_h for MQA (128).

The same bound holds for values. The cache stores exactly the numbers that span this space, so the cache size and the capacity bound are the same quantity: GQA buys its cache saving by limiting the joint key and value space of all heads to gdhg d_h dimensions each.

That this costs little quality is an empirical finding. It fits older evidence that many attention heads are redundant: in trained translation and BERT models, a large fraction of heads can be pruned with little loss (Michel et al., 2019; Voita et al., 2019). GQA keeps all the query heads, and those queries can still weight the past differently.

Multi-head latent attention: cache a code, rebuild the heads

DeepSeek-V2 (2024) introduced multi-head latent attention (MLA). It caches one latent vector per token,

ct=WDKVxt∈Rdc,c_t = W^{DKV} x_t \in \mathbb{R}^{d_c},

with dc=512d_c = 512 in DeepSeek-V2, V3 and V3.2, and rebuilds every head’s key and value from it with per-head up-projections:

kt,iC=WiUKct,vt,i=WiUVct.k^C_{t,i} = W^{UK}_i c_t, \qquad v_{t,i} = W^{UV}_i c_t .

(The superscript CC marks the content part of the key; a positional part comes later. In the released models ctc_t also passes through an RMSNorm before the up-projections.)

In the rank view, MLA’s keys span at most dc=512d_c = 512 dimensions, and so do its values. The difference from GQA lies in the structure:

  • GQA spends its cache on two separate codes, one for keys and one for values. MLA uses one code for both, so at the same cache size each of them can use the full width.
  • In GQA, each head reads one fixed slice of the cache (its group’s head). In MLA every head reads the whole latent through its own learned up-projection, so heads can combine the shared information differently.
  • Per head nothing is lost: a 128-dimensional key computed from a 512-dimensional code can still be any linear function of that code.

DeepSeek-V2 reports that MLA’s cache is about the size of GQA with 2.25 groups, and that in its ablations MLA scored better than MHA. GQA is in fact a special case: a GQA layer can be rewritten as an MLA layer with the same cache size (Meng et al., 2025, TransMLA), but not the reverse.

DeepSeek’s MLA also compresses the queries through a low-rank bottleneck (1,536 dimensions in V3.2). This saves activation memory during training and has nothing to do with the cache.

xtWDKVWQWOotq1k1v1q̃1 · 512attentionWUK1WUK1TWUV1q2k2v2q̃2 · 512attentionWUK2WUK2TWUV2q3k3v3q̃3 · 512attentionWUK3WUK3TWUV3ct · 512512 numbers per token per layer (+64 for position, next figure)⋮⋮125 more heads, h = 128WOi WUVicached per token per layerprefillct: 512 numbersdecodect: 512 numbers

Prefill: every head builds its own key ki = WUKic and value vi = WUVic (128 numbers each, 128 heads) from the cached latent.

Figure 3.The same function computed two ways. Prefill builds keys and values for every head; decode folds the up-projections into the query and the output, so every head reads the shared latent.

Absorption: never building the keys

If MLA rebuilt every head’s keys from the latents at each decoding step, it would save memory but waste compute: 128 heads × every past token × an up-projection. What makes MLA fast at decode is that the up-projections can be moved off the cache. The content score of head ii against past token ss is

(qt,iC)⊤ks,iC=(qt,iC)⊤WiUKcs=q~t,i ⊤cs,q~t,i=(WiUK)⊤qt,iC∈Rdc.\left(q^C_{t,i}\right)^\top k^C_{s,i} = \left(q^C_{t,i}\right)^\top W^{UK}_i c_s = \tilde q_{t,i}^{\,\top} c_s, \qquad \tilde q_{t,i} = \left(W^{UK}_i\right)^\top q^C_{t,i} \in \mathbb{R}^{d_c}.

So each query is transformed once, into a 512-dimensional vector, and scored directly against the cached latents. The value side works the same way. With attention weights ats(i)a^{(i)}_{ts} and output projection WO=[W1O,…,WhO]W^O = [W^O_1, \dots, W^O_h], the layer output is

ot=∑i=1hWiO WiUV∑sats(i) cs,o_t = \sum_{i=1}^{h} W^O_i\, W^{UV}_i \sum_{s} a^{(i)}_{ts}\, c_s ,

so each head first averages latents, and WiOWiUVW^O_i W^{UV}_i is applied once per head and token, or precomputed as one matrix. After this absorption, decoding with MLA is multi-query attention with one shared “head” of width 512: all 128 query heads read the same 512 numbers per past token. The result is exact up to floating-point rounding; only the order of the matrix products changes.

For prefill the balance tips the other way. With thousands of new tokens at once, building per-head keys and values explicitly and running ordinary multi-head attention is cheaper. Engines therefore typically run MLA with two kernels, an explicit multi-head form for new tokens and the absorbed form for decode. The choice really depends on how many new tokens meet how many cached ones, so a long prompt continuing a cached conversation may use the absorbed form for its cached part.

RoPE gets in the way: the decoupled key

Absorption needs WiUKW^{UK}_i to sit between qq and cc with nothing in between. Rotary position embedding (RoPE, see T4) puts something in between: it rotates queries and keys by matrices RtR_t, RsR_s that depend on their positions, and the score becomes q⊤Rt⊤RsWiUKcsq^\top R_t^\top R_s W^{UK}_i c_s. A matrix that changes with every past token can’t be folded into one fixed query transform.

Figure 4.The decoupled RoPE key. Position travels in a small separate key that all heads share, so the content part of the key can still be absorbed.

MLA keeps position out of the latent. Each head’s query and key get a second, small part of dhR=64d_h^R = 64 dimensions that carries RoPE: a per-head positional query qt,iRq^R_{t,i}, and a single positional key ksR=RoPEs(WKRxs)k^R_s = \mathrm{RoPE}_s(W^{KR} x_s) computed from the input and shared by all heads. The score is the sum of the two parts,

scoret,s,i=q~t,i ⊤cs  +  (qt,iR)⊤ksR,\text{score}_{t,s,i} = \tilde q_{t,i}^{\,\top} c_s \;+\; \left(q^R_{t,i}\right)^\top k^R_s ,

scaled by 1/dh+dhR1/\sqrt{d_h + d_h^R}. The first term is absorbed as above. The second is a small MQA-style term with a shared, rotated key, which is cached next to the latent. Per token per layer the cache holds dc+dhR=576d_c + d_h^R = 576 numbers, 1,152 bytes in BF16. In the DeepSeek-V3 shape that is 68.6 KiB per token, 57 times less than MHA and 3.6 times less than GQA with 8 groups.

What MLA costs

Compute per byte goes up. That is intended. Count the work for one query token against one cached latent: 128 heads each take a 576-dimensional dot product and add a 512-dimensional weighted latent, about 278,000 operations for 1,152 bytes, or 242 operations per byte. That is close to the H100’s balance point of about 300, where decode attention stops being memory-bound (computed; see the roofline figure).

The other costs are engineering:

  • Tensor parallelism has the same problem as in MQA: one latent per token, needed by every head. Serving stacks for MLA models often run the attention part data-parallel (each GPU serves different requests with its own cache) and split only the rest of the layer.
  • Prefill and decode use different kernels for the same function, which means more code to optimize and keep numerically consistent.
  • A common stabilizer, normalizing queries and keys before the dot product (QK-Norm, see R1), needs keys as explicit vectors, and in absorbed MLA they never exist. Kimi K2 introduced QK-Clip, which rescales the query and key projection weights when attention logits grow too large, for this reason.
Cache stored as
1101001,0001101001,000compute roof 989 TFLOP/s (H100 SXM)memory roof 3.35 TB/s × intensityridge ≈ 295 FLOP/B← memory-boundcompute-boundMHA, BF16 cache: 128 heads × (2·128 + 2·128) = 65,536 FLOP per past token, over 2·128·128 = 32,768 numbers × 2 B = 65,536 B → 1.0 FLOP/B. Attainable: min(989, 3.35 × 1.0) = 3 TFLOP/s, memory-bound.MHA 1.0GQA, g = 8, BF16 cache: 128 heads × (2·128 + 2·128) = 65,536 FLOP per past token, over 2·8·128 = 2,048 numbers × 2 B = 4,096 B → 16.0 FLOP/B. Attainable: min(989, 3.35 × 16.0) = 54 TFLOP/s, memory-bound.GQA 16MQA, BF16 cache: 128 heads × (2·128 + 2·128) = 65,536 FLOP per past token, over 2·128 = 256 numbers × 2 B = 512 B → 128.0 FLOP/B. Attainable: min(989, 3.35 × 128.0) = 429 TFLOP/s, memory-bound.MQA 128MLA (absorbed), BF16 cache: 2·128·(576 + 512) = 278,528 FLOP per past token, over 512 + 64 = 576 numbers × 2 B = 1,152 B → 241.8 FLOP/B. Attainable: min(989, 3.35 × 241.8) = 810 TFLOP/s, memory-bound.MLA 242Intensity, FLOP per byte of KV cache read (log scale)Attainable TFLOP/s (log scale)head sharing (MHA, GQA, MQA)latent (MLA)

MLA (absorbed), BF16 cache: 2·128·(576 + 512) = 278,528 FLOP per past token, over 512 + 64 = 576 numbers × 2 B = 1,152 B → 241.8 FLOP/B. Attainable: min(989, 3.35 × 241.8) = 810 TFLOP/s, memory-bound.

Figure 5.Work per byte of cache read during decoding. Plain multi-head attention leaves the GPU waiting on memory; MLA's absorbed form nearly reaches the compute roof.

In the map, MLA is used by DeepSeek-V3.2, GLM-5.2, GLM-5.3, GLM-5.3-Flash, Kimi K3 and Hy4-preview. Kimi K3 and Hy4 add an output gate (“Gated MLA”, see T6). Kimi K3 also applies no RoPE in its MLA layers, because its other layers carry position (T4). Its modeling code nevertheless keeps the 64-dimensional shared key part, unrotated (kv_a_proj_with_mqa outputs kv_lora_rank + qk_rope_head_dim, and rotary_emb is None), so its cache is still 576 numbers per token per layer.

Tying keys to values

The designs above still store a key and a value, even when shared. Two models in the map tie them together.

DeepSeek-V4 uses a single key/value head of width 512 in which, in the report’s words, “each compressed KV entry serves as both attention key and value”. The queries come through a low-rank projection. Because 64 or 128 heads of width 512 produce a very wide output, the output projection is split into groups (“grouped output projection”). The cached entries are also compressed along the sequence and handle position differently from plain RoPE; T3 and T4 cover both.

Gemma 4 does something similar in its global layers only, “re-using keys as values in the global attention layers”. In the 31B model these layers have 4 KV heads of dimension 512 (num_global_key_value_heads, global_head_dim), so each caches 4 × 512 numbers per token (4 KiB in BF16) instead of twice that.

Tying halves the cache once more and gives up the last independence between the two roles: what a token is matched on and what it contributes are the same stored vector, transformed differently only on the query and output sides.

Comparison

MHAMQAGQA (gg groups)MLAK=V (gg heads)
Cached per token per layer2hdh2 h d_h2dh2 d_h2gdh2 g d_hdc+dhRd_c + d_h^Rgdhg d_h
DeepSeek-V3 shape, BF1664 KiB512 B4 KiB (gg = 8)1,152 B4 KiB (Gemma 4 global, gg = 4, dhd_h = 512)
Joint key space (rank bound)min⁡(d,hdh)\min(d, h d_h)dhd_hgdhg d_hdcd_c (+ shared dhRd_h^R)gdhg d_h, shared with values
Decode FLOP per cache byte≈1≈hh≈h/gh/g≈240≈2h/g2h/g
Main costmemoryquality, TPsome quality, TP at small ggcompute, two kernels, TP, no QK-NormK/V independence
Convert from MHAn/ayes, with continued trainingyes, mean-pool + ~5% trainingfrom GQA: TransMLAno
In the mapnonenone pure16 models6 modelsDeepSeek-V4 family, Gemma 4
642561K4K16K1285122K8K32Kd = 7,168 (model width)y = x/2: head sharingMHA: cache 2·h·d_h = 2·128·128 = 32,768 numbers; key space min(d, h·d_h) = min(7,168, 16,384) = 7,168. Capped by the model width d, so it falls below the line.MHAGQA, g = 32: cache 2·g·d_h = 2·32·128 = 8,192 numbers; key space g·d_h = 32·128 = 4,096, on y = x/2.GQA g=32GQA, g = 16: cache 2·g·d_h = 2·16·128 = 4,096 numbers; key space g·d_h = 16·128 = 2,048, on y = x/2.GQA g=16GQA, g = 8: cache 2·g·d_h = 2·8·128 = 2,048 numbers; key space g·d_h = 8·128 = 1,024, on y = x/2.GQA g=8GQA, g = 4: cache 2·g·d_h = 2·4·128 = 1,024 numbers; key space g·d_h = 4·128 = 512.0, on y = x/2.GQA g=4GQA, g = 2: cache 2·g·d_h = 2·2·128 = 512.0 numbers; key space g·d_h = 2·128 = 256.0, on y = x/2.GQA g=2MQA: cache 2·d_h = 2·128 = 256 numbers; key space d_h = 128, on y = x/2.MQAMLA: cache d_c + d_h^R = 512 + 64 = 576 numbers; key space d_c = 512 (plus the shared 64-dimensional rotated key). Above the line: one 512-dimensional code serves both keys and values.MLACached numbers per token per layer (log scale)Rank bound of key space, dims (log scale)head sharing (MHA, GQA, MQA)latent (MLA)

MLA: cache d_c + d_h^R = 512 + 64 = 576 numbers; key space d_c = 512 (plus the shared 64-dimensional rotated key). Above the line: one 512-dimensional code serves both keys and values.

DeepSeek-V3 shape: h = 128, d_h = 128, d = 7,168. A rank bound is an upper limit on what the keys can express, not a measure of quality.

Figure 6.For head sharing, a smaller cache means a smaller key space in proportion. MLA gets more key dimensions per cached number by sharing one code between keys and values.

Where the models in the map sit

The table counts only layers whose cache grows with context. Sliding-window and linear layers keep a state of fixed size (T2) and are left out. Numbers are computed from each model’s config.json at BF16, before any cache quantization (deployments often store FP8, halving them), and without the extra index caches of sparse-attention models (T3). They rank memory cost only; the models differ in size, quality and everything else.

ModelGlobal layersLayoutKV per token (KiB)at 1M tokens (GiB)
MHA reference, DeepSeek-V3 shape61 of 61128 × 1283,9043,904
MiniMax-M360 of 60GQA, 4 × 128120.0120.0
GLM-5.278 of 78MLA87.887.8
DeepSeek-V3.261 of 61MLA68.668.6
MiMo-V2.5-Pro10 of 70GQA, 8 × (192 K + 128 V)50.050.0
Laguna-S-2.112 of 48GQA, 8 × 12848.048.0
Gemma 4 31B10 of 60K=V, 4 × 51240.040.0
gpt-oss-120b18 of 36GQA, 8 × 6436.036.0
Qwen3.5-397B15 of 60GQA, 2 × 25630.030.0
Kimi K324 of 93MLA27.027.0
Kolibri-1 (from its report)10 of 50GQA, 4 × 12820.020.0

(KiB per token and GiB at 1M tokens are numerically equal: 2202^{20} tokens × 1 KiB = 1 GiB.)

  • GQA
  • MLA
  • K=V
  • MHA reference
  • from its report
101001,00010,000MHA reference, DeepSeek-V3 shape: every one of 61 layers; MHA 128 × 128 = 32,768 numbers × 2 B = 64 KiB per layer × 61 = 3,904 KiB per token.MHA reference3,904 KiB · MHAMiniMax-M3: 60 of 60 layers global; GQA 4 × 128 = 1,024 numbers × 2 B = 2 KiB per layer × 60 = 120.0 KiB per token (3.1% of the MHA reference).MiniMax-M3120.0 KiB · GQAGLM-5.2: 78 of 78 layers global; MLA 512 + 64 = 576 numbers × 2 B = 1.125 KiB per layer × 78 = 87.8 KiB per token (2.2% of the MHA reference).GLM-5.287.8 KiB · MLADeepSeek-V3.2: 61 of 61 layers global; MLA 512 + 64 = 576 numbers × 2 B = 1.125 KiB per layer × 61 = 68.6 KiB per token (1.8% of the MHA reference).DeepSeek-V3.268.6 KiB · MLAMiMo-V2.5-Pro: 10 of 70 layers global; GQA 8 × (192 K + 128 V) = 2,560 numbers × 2 B = 5 KiB per layer × 10 = 50.0 KiB per token (1.3% of the MHA reference).MiMo-V2.5-Pro50.0 KiB · GQALaguna-S-2.1: 12 of 48 layers global; GQA 8 × 128 = 2,048 numbers × 2 B = 4 KiB per layer × 12 = 48.0 KiB per token (1.2% of the MHA reference).Laguna-S-2.148.0 KiB · GQAGemma 4 31B: 10 of 60 layers global; K=V 4 × 512 = 2,048 numbers × 2 B = 4 KiB per layer × 10 = 40.0 KiB per token (1.0% of the MHA reference).Gemma 4 31B40.0 KiB · K=Vgpt-oss-120b: 18 of 36 layers global; GQA 8 × 64 = 1,024 numbers × 2 B = 2 KiB per layer × 18 = 36.0 KiB per token (0.9% of the MHA reference).gpt-oss-120b36.0 KiB · GQAQwen3.5-397B: 15 of 60 layers global; GQA 2 × 256 = 1,024 numbers × 2 B = 2 KiB per layer × 15 = 30.0 KiB per token (0.8% of the MHA reference).Qwen3.5-397B30.0 KiB · GQAKimi K3: 24 of 93 layers global; MLA 512 + 64 = 576 numbers × 2 B = 1.125 KiB per layer × 24 = 27.0 KiB per token (0.7% of the MHA reference).Kimi K327.0 KiB · MLAKolibri-1: 10 of 50 layers global; GQA 4 × 128 = 1,024 numbers × 2 B = 2 KiB per layer × 10 = 20.0 KiB per token (0.5% of the MHA reference). From its report, not a config.Kolibri-120.0 KiB · GQA, from its reportKV cache per token, KiB (log scale)

Counting global layers only, the mapped models store 0.5–3.1% of the MHA reference.

Hover or focus a bar for its computation from config.json.

Sliding-window and linear layers keep a fixed-size state and are left out; the gap between the two settings is the saving from hybrid stacking (T5). BF16, before cache quantization.

Figure 7.KV cache per token for models in the map, counting only layers whose cache grows with context, BF16.

Most of the saving in current models comes from combining a compact layout with few global layers; T5 covers that second half. Once a GQA model uses only two KV heads, MLA is no longer much smaller: Qwen3.5’s global layer stores 2 KiB per token, against 1.1 KiB for MLA.

Easy to get wrong

  • MLA and GQA both shrink the cache, but by different means. GQA makes heads share keys and values. MLA gives every head its own keys and values, computed from one shared stored code.
  • Absorption is not an approximation. It changes the order of matrix products, and the result is the same up to rounding.
  • MQA and GQA save memory traffic, not arithmetic. A decoded token costs about the same FLOPs; it reads far fewer bytes.
  • A small KV cache doesn’t mean low quality, and a cache table doesn’t rank models. The quality cost of sharing is measured per design, at given scales, in the cited papers.
  • Code-check counts for “GQA” and “MLA” need care with the DeepSeek-V4 family, whose single shared key/value head fits neither label.

Old ideas in new places

The ingredients are classical. MQA is weight tying across heads. MLA is a bottleneck (low-rank) factorization of the key and value projections, of the kind long used to compress networks. Tying keys to values has a precedent in Reformer, which shared queries and keys for its hashing scheme (Kitaev et al., 2020). What is new is the reason: the factorizations are chosen to minimize bytes read per decoded token, not parameters or FLOPs.

Check yourself

  1. A model has 48 layers, 32 query heads, and 8 KV heads of dimension 128, and stores its cache in FP8. How large is the cache for one 256K-token sequence?
    Answer2 × 8 × 128 × 1 byte = 2 KiB per token per layer; × 48 = 96 KiB per token; × 262,144 tokens = 24 GiB.
  2. In that model, what is the arithmetic intensity of decode attention (BF16), and how many GPUs can share the KV heads under tensor parallelism without duplicating any?
    Answerh/g = 32/8 = 4 operations per byte; up to 8 GPUs.
  3. Why can’t WiUKW^{UK}_i be absorbed into the query when RoPE is applied to the whole key?
    AnswerRoPE multiplies each key by a rotation that depends on that key’s position. A matrix that changes with every past token can’t be folded into one fixed query transform. Hence the separate, small rotated key.
  4. Why doesn’t a larger batch make decode attention compute-bound?
    AnswerEach sequence has its own cache. A bigger batch adds cache bytes as fast as it adds work, so operations per byte stay the same.

In the map

Methods

Taxonomy nodes

Further reading

  • Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019), arXiv:1911.02150. The original MQA argument, framed in memory bandwidth.
  • Ainslie et al., GQA (2023), arXiv:2305.13245. The conversion recipe and quality as a function of the number of groups.
  • DeepSeek-AI, DeepSeek-V2 (2024), arXiv:2405.04434, §2.1 and appendix. MLA, the decoupled RoPE key, and the comparison with MHA, MQA and GQA.
  • Meng et al., TransMLA: Multi-Head Latent Attention Is All You Need (2025), arXiv:2502.07864. Why GQA is a special case of MLA, and converting GQA models.
  • Kimi Team, Kimi K2 (2025), arXiv:2507.20534. QK-Clip.
  • DeepSeek-V4 report (2026), §2.3, https://arxiv.org/html/2606.19348. Shared key-value MQA and grouped output projection.
  • Gemma 4 technical report (2026), https://arxiv.org/html/2607.02770. Keys reused as values in global layers.
  • Pope et al., Efficiently Scaling Transformer Inference (2022), arXiv:2211.05102. The memory-bandwidth view of decoding used throughout.
  • Michel et al., Are Sixteen Heads Really Better than One? (2019), arXiv:1905.10650; Voita et al., Analyzing Multi-Head Self-Attention (2019), arXiv:1905.09418. Evidence that many heads are redundant.
  • Clark et al., What Does BERT Look At? (2019), arXiv:1906.04341. Head specialization.
  • Kitaev et al., Reformer (2020), arXiv:2001.04451. Shared query-key attention.
↑