Suppose you serve a model the size of DeepSeek-V3 (61 layers, 128 attention heads of 128 dimensions each) but with the attention of the original Transformer. A single conversation of 128K tokens then needs 488 GiB of GPU memory just to remember its own past. That is more than six 80 GB H100s for one user, before a single weight is loaded. No model in this map comes close. The ones with exact attention in every layer store 2 to 3% of that, and they still attend to every past token.
This article explains how. The four designs it covers (multi-head, multi-query, grouped-query and multi-head latent attention) keep exact softmax attention over the whole context and change only what is stored per token. Each one gives something up to shrink that store.
What the KV cache is, and why it sets decode speed
Language models are run in two phases. Prefill processes the prompt: all its tokens go through the network at once, in large matrix multiplications. Decode then generates the answer one token at a time. To produce token , every attention layer compares the new token’s query with the keys of all earlier tokens and takes a weighted sum of their values. Those keys and values never change once computed, so inference engines keep them in GPU memory. This store is the KV cache.
With key/value heads of dimension , stored at bytes per number, each token adds
to the cache (the 2 is for keys and values). Multiply by the number of layers and the context length and you have the cache for one sequence. With , , (BF16) and this is 3.8 MiB per token, which is where the 488 GiB come from.
Decoding a token also reads the whole cache once, and that sets its speed. Count the work for one head and one past token. Reading its key and value means moving numbers, bytes in BF16. Using them costs one dot product with the query and one scaled addition of the value, multiply-adds or floating-point operations (counting a multiply-add as two and ignoring the softmax). That is one operation per byte. An H100 can do about 990 TFLOP/s of BF16 arithmetic but read only 3.35 TB/s from its memory, so it needs roughly 300 operations per byte to keep its arithmetic units busy. At one operation per byte, decode attention runs at well under 1% of the chip’s arithmetic throughput, and its speed is set almost entirely by how many bytes the cache has. Engineers call this regime memory-bound. The ratio of operations to bytes is the arithmetic intensity.
Batching many users together doesn’t help here, unlike for the weight matrices. A weight matrix read once from memory serves every sequence in the batch, so a bigger batch means more work per byte. Each sequence has its own KV cache, so a bigger batch adds cache bytes exactly as fast as it adds work.
weightskeysvalueshatched = cached state · axis 0 to 500 GiB, linear
- per token per layer
- 2 × 128 × 128 × 2 B = 64 KiB
- per token (× L)
- 3.8 MiB
- KV cache (× T × batch)
- 488 GiB
- total with weights
- 488 GiB = 6.5× one 80 GB GPU
Every technique below shrinks the part of the formula. Other articles in the series shrink , by keeping only recent tokens (T2) or selecting a few (T3), and , by making only some layers global (T5).
Multi-head attention: the baseline
In multi-head attention (MHA) each of the heads has its own projections. For an input and head :
Every head has its own keys and values, so and the cache holds numbers per token per layer. Many models choose , which makes this two copies of the model width. DeepSeek-V3 doesn’t: against , which is why its hypothetical MHA cache is so large.
Separate heads let each one define its own notion of relevance (keys) and bring back its own information (values). Probing studies find heads specialized for, for example, the previous token or syntactic relations (Clark et al., 2019). That flexibility is what the next three designs ration.
Multi-query attention: one key/value head for all
Shazeer (2019) proposed sharing a single key head and a single value head across all query heads. In multi-query attention (MQA) the queries stay per head, but
The cache shrinks by a factor of , from 3.8 MiB to 30.5 KiB per token in our example. Decoding speeds up for the same reason: all heads use the same bytes, so the GPU does times as much work per byte read (an intensity of 128 instead of 1). The FLOPs per token stay about the same; what falls is the memory traffic.
Each head can still ask its own question, but all heads now search the same keys and retrieve from the same values. Ainslie et al. (2023) report that MQA loses quality against MHA and can make training less stable.
There is also a systems cost. Large models are usually run with tensor parallelism: each layer is split across several GPUs, and for attention the natural split is by heads, so each GPU holds the keys and values of its own heads. With one KV head there is nothing to split, and every GPU must hold a full copy of the cache.
Grouped-query attention: a dial between the two
Grouped-query attention (GQA; Ainslie et al., 2023) splits the query heads into groups, and each group shares one key head and one value head. is MHA, is MQA, and values in between trade cache size against capacity.
Hover a query head to see the pair it reads.Each GPU holds 1 of the 4 pairs; nothing is duplicated.
Real models have 32 to 128 query heads; 16 keep the lines readable.
Two findings made GQA the default. With 8 groups, Ainslie et al. obtained quality close to MHA at a cache cost close to MQA. And an existing MHA checkpoint can be converted: average the key and value heads within each group, then continue training the converted model briefly (they used about 5% of the original pre-training compute).
Among the models in the map whose layout the code check settles cleanly, 16 of 22 use GQA.
(The other six are MLA models; the six DeepSeek-V4 variants use a different layout,
described below, and are counted separately.) The number of KV heads is small. Qwen3.5,
Qwen3.6, Qwen3.8 and both Nemotron 3 models use 2, MiniMax-M3 uses 4, and gpt-oss, Laguna,
MiMo, Hy3 and Inkling use 8 (num_key_value_heads in each config.json). With 2 KV heads
and 32 query heads, each key and value is shared by 16 queries.
The tensor-parallel problem from MQA comes back here. Qwen3.5’s 2 KV heads can be split over at most 2 GPUs; on 8 GPUs each KV head is copied four times. In practice, how far you want to shard a model limits how low goes.
What sharing costs: a rank view
“Some capacity” can be made precise. Stack the key projections of all heads into one map . Whatever the heads do with the keys afterwards, everything they can match on is contained in the image of this map, whose dimension is at most
- for MHA (7,168 in the DeepSeek-V3 shape),
- for GQA (1,024 with 8 groups), because heads in a group have identical keys,
- for MQA (128).
The same bound holds for values. The cache stores exactly the numbers that span this space, so the cache size and the capacity bound are the same quantity: GQA buys its cache saving by limiting the joint key and value space of all heads to dimensions each.
That this costs little quality is an empirical finding. It fits older evidence that many attention heads are redundant: in trained translation and BERT models, a large fraction of heads can be pruned with little loss (Michel et al., 2019; Voita et al., 2019). GQA keeps all the query heads, and those queries can still weight the past differently.
Multi-head latent attention: cache a code, rebuild the heads
DeepSeek-V2 (2024) introduced multi-head latent attention (MLA). It caches one latent vector per token,
with in DeepSeek-V2, V3 and V3.2, and rebuilds every head’s key and value from it with per-head up-projections:
(The superscript marks the content part of the key; a positional part comes later. In the released models also passes through an RMSNorm before the up-projections.)
In the rank view, MLA’s keys span at most dimensions, and so do its values. The difference from GQA lies in the structure:
- GQA spends its cache on two separate codes, one for keys and one for values. MLA uses one code for both, so at the same cache size each of them can use the full width.
- In GQA, each head reads one fixed slice of the cache (its group’s head). In MLA every head reads the whole latent through its own learned up-projection, so heads can combine the shared information differently.
- Per head nothing is lost: a 128-dimensional key computed from a 512-dimensional code can still be any linear function of that code.
DeepSeek-V2 reports that MLA’s cache is about the size of GQA with 2.25 groups, and that in its ablations MLA scored better than MHA. GQA is in fact a special case: a GQA layer can be rewritten as an MLA layer with the same cache size (Meng et al., 2025, TransMLA), but not the reverse.
DeepSeek’s MLA also compresses the queries through a low-rank bottleneck (1,536 dimensions in V3.2). This saves activation memory during training and has nothing to do with the cache.
Prefill: every head builds its own key ki = WUKic and value vi = WUVic (128 numbers each, 128 heads) from the cached latent.Decode: WUKi moves into the query, q̃i = (WUKi)Tqi (512 numbers), and WUVi into WO; all 128 heads attend to c directly. Nothing new is cached.ct: 512 numbers per token per layer (+64 for position, next figure).
Absorption: never building the keys
If MLA rebuilt every head’s keys from the latents at each decoding step, it would save memory but waste compute: 128 heads × every past token × an up-projection. What makes MLA fast at decode is that the up-projections can be moved off the cache. The content score of head against past token is
So each query is transformed once, into a 512-dimensional vector, and scored directly against the cached latents. The value side works the same way. With attention weights and output projection , the layer output is
so each head first averages latents, and is applied once per head and token, or precomputed as one matrix. After this absorption, decoding with MLA is multi-query attention with one shared “head” of width 512: all 128 query heads read the same 512 numbers per past token. The result is exact up to floating-point rounding; only the order of the matrix products changes.
For prefill the balance tips the other way. With thousands of new tokens at once, building per-head keys and values explicitly and running ordinary multi-head attention is cheaper. Engines therefore typically run MLA with two kernels, an explicit multi-head form for new tokens and the absorbed form for decode. The choice really depends on how many new tokens meet how many cached ones, so a long prompt continuing a cached conversation may use the absorbed form for its cached part.
RoPE gets in the way: the decoupled key
Absorption needs to sit between and with nothing in between. Rotary position embedding (RoPE, see T4) puts something in between: it rotates queries and keys by matrices , that depend on their positions, and the score becomes . A matrix that changes with every past token can’t be folded into one fixed query transform.
MLA keeps position out of the latent. Each head’s query and key get a second, small part of dimensions that carries RoPE: a per-head positional query , and a single positional key computed from the input and shared by all heads. The score is the sum of the two parts,
scaled by . The first term is absorbed as above. The second is a small MQA-style term with a shared, rotated key, which is cached next to the latent. Per token per layer the cache holds numbers, 1,152 bytes in BF16. In the DeepSeek-V3 shape that is 68.6 KiB per token, 57 times less than MHA and 3.6 times less than GQA with 8 groups.
What MLA costs
Compute per byte goes up. That is intended. Count the work for one query token against one cached latent: 128 heads each take a 576-dimensional dot product and add a 512-dimensional weighted latent, about 278,000 operations for 1,152 bytes, or 242 operations per byte. That is close to the H100’s balance point of about 300, where decode attention stops being memory-bound (computed; see the roofline figure).
The other costs are engineering:
- Tensor parallelism has the same problem as in MQA: one latent per token, needed by every head. Serving stacks for MLA models often run the attention part data-parallel (each GPU serves different requests with its own cache) and split only the rest of the layer.
- Prefill and decode use different kernels for the same function, which means more code to optimize and keep numerically consistent.
- A common stabilizer, normalizing queries and keys before the dot product (QK-Norm, see R1), needs keys as explicit vectors, and in absorbed MLA they never exist. Kimi K2 introduced QK-Clip, which rescales the query and key projection weights when attention logits grow too large, for this reason.
MLA (absorbed), BF16 cache: 2·128·(576 + 512) = 278,528 FLOP per past token, over 512 + 64 = 576 numbers × 2 B = 1,152 B → 241.8 FLOP/B. Attainable: min(989, 3.35 × 241.8) = 810 TFLOP/s, memory-bound.
In the map, MLA is used by DeepSeek-V3.2, GLM-5.2, GLM-5.3, GLM-5.3-Flash, Kimi K3 and
Hy4-preview. Kimi K3 and Hy4 add an output gate (“Gated MLA”, see T6). Kimi K3 also applies no
RoPE in its MLA layers, because its other layers carry position (T4). Its modeling code
nevertheless keeps the 64-dimensional shared key part, unrotated
(kv_a_proj_with_mqa outputs kv_lora_rank + qk_rope_head_dim, and rotary_emb is None),
so its cache is still 576 numbers per token per layer.
Tying keys to values
The designs above still store a key and a value, even when shared. Two models in the map tie them together.
DeepSeek-V4 uses a single key/value head of width 512 in which, in the report’s words, “each compressed KV entry serves as both attention key and value”. The queries come through a low-rank projection. Because 64 or 128 heads of width 512 produce a very wide output, the output projection is split into groups (“grouped output projection”). The cached entries are also compressed along the sequence and handle position differently from plain RoPE; T3 and T4 cover both.
Gemma 4 does something similar in its global layers only, “re-using keys as values in the
global attention layers”. In the 31B model these layers have 4 KV heads of dimension 512
(num_global_key_value_heads, global_head_dim), so each caches 4 × 512 numbers per token
(4 KiB in BF16) instead of twice that.
Tying halves the cache once more and gives up the last independence between the two roles: what a token is matched on and what it contributes are the same stored vector, transformed differently only on the query and output sides.
Comparison
| MHA | MQA | GQA ( groups) | MLA | K=V ( heads) | |
|---|---|---|---|---|---|
| Cached per token per layer | |||||
| DeepSeek-V3 shape, BF16 | 64 KiB | 512 B | 4 KiB ( = 8) | 1,152 B | 4 KiB (Gemma 4 global, = 4, = 512) |
| Joint key space (rank bound) | (+ shared ) | , shared with values | |||
| Decode FLOP per cache byte | ≈1 | ≈ | ≈ | ≈240 | ≈ |
| Main cost | memory | quality, TP | some quality, TP at small | compute, two kernels, TP, no QK-Norm | K/V independence |
| Convert from MHA | n/a | yes, with continued training | yes, mean-pool + ~5% training | from GQA: TransMLA | no |
| In the map | none | none pure | 16 models | 6 models | DeepSeek-V4 family, Gemma 4 |
MLA: cache d_c + d_h^R = 512 + 64 = 576 numbers; key space d_c = 512 (plus the shared 64-dimensional rotated key). Above the line: one 512-dimensional code serves both keys and values.
DeepSeek-V3 shape: h = 128, d_h = 128, d = 7,168. A rank bound is an upper limit on what the keys can express, not a measure of quality.
Where the models in the map sit
The table counts only layers whose cache grows with context. Sliding-window and linear
layers keep a state of fixed size (T2) and are left out. Numbers are computed from each
model’s config.json at BF16, before any cache quantization (deployments often store FP8,
halving them), and without the extra index caches of sparse-attention models (T3). They rank
memory cost only; the models differ in size, quality and everything else.
| Model | Global layers | Layout | KV per token (KiB) | at 1M tokens (GiB) |
|---|---|---|---|---|
| MHA reference, DeepSeek-V3 shape | 61 of 61 | 128 × 128 | 3,904 | 3,904 |
| MiniMax-M3 | 60 of 60 | GQA, 4 × 128 | 120.0 | 120.0 |
| GLM-5.2 | 78 of 78 | MLA | 87.8 | 87.8 |
| DeepSeek-V3.2 | 61 of 61 | MLA | 68.6 | 68.6 |
| MiMo-V2.5-Pro | 10 of 70 | GQA, 8 × (192 K + 128 V) | 50.0 | 50.0 |
| Laguna-S-2.1 | 12 of 48 | GQA, 8 × 128 | 48.0 | 48.0 |
| Gemma 4 31B | 10 of 60 | K=V, 4 × 512 | 40.0 | 40.0 |
| gpt-oss-120b | 18 of 36 | GQA, 8 × 64 | 36.0 | 36.0 |
| Qwen3.5-397B | 15 of 60 | GQA, 2 × 256 | 30.0 | 30.0 |
| Kimi K3 | 24 of 93 | MLA | 27.0 | 27.0 |
| Kolibri-1 (from its report) | 10 of 50 | GQA, 4 × 128 | 20.0 | 20.0 |
(KiB per token and GiB at 1M tokens are numerically equal: tokens × 1 KiB = 1 GiB.)
- GQA
- MLA
- K=V
- MHA reference
- from its report
Counting global layers only, the mapped models store 0.5–3.1% of the MHA reference.
Hover or focus a bar for its computation from config.json.
Sliding-window and linear layers keep a fixed-size state and are left out; the gap between the two settings is the saving from hybrid stacking (T5). BF16, before cache quantization.
Most of the saving in current models comes from combining a compact layout with few global layers; T5 covers that second half. Once a GQA model uses only two KV heads, MLA is no longer much smaller: Qwen3.5’s global layer stores 2 KiB per token, against 1.1 KiB for MLA.
Easy to get wrong
- MLA and GQA both shrink the cache, but by different means. GQA makes heads share keys and values. MLA gives every head its own keys and values, computed from one shared stored code.
- Absorption is not an approximation. It changes the order of matrix products, and the result is the same up to rounding.
- MQA and GQA save memory traffic, not arithmetic. A decoded token costs about the same FLOPs; it reads far fewer bytes.
- A small KV cache doesn’t mean low quality, and a cache table doesn’t rank models. The quality cost of sharing is measured per design, at given scales, in the cited papers.
- Code-check counts for “GQA” and “MLA” need care with the DeepSeek-V4 family, whose single shared key/value head fits neither label.
Old ideas in new places
The ingredients are classical. MQA is weight tying across heads. MLA is a bottleneck (low-rank) factorization of the key and value projections, of the kind long used to compress networks. Tying keys to values has a precedent in Reformer, which shared queries and keys for its hashing scheme (Kitaev et al., 2020). What is new is the reason: the factorizations are chosen to minimize bytes read per decoded token, not parameters or FLOPs.
Check yourself
- A model has 48 layers, 32 query heads, and 8 KV heads of dimension 128, and stores its cache
in FP8. How large is the cache for one 256K-token sequence?
Answer
2 × 8 × 128 × 1 byte = 2 KiB per token per layer; × 48 = 96 KiB per token; × 262,144 tokens = 24 GiB. - In that model, what is the arithmetic intensity of decode attention (BF16), and how many
GPUs can share the KV heads under tensor parallelism without duplicating any?
Answer
h/g = 32/8 = 4 operations per byte; up to 8 GPUs. - Why can’t be absorbed into the query when RoPE is applied to the whole key?
Answer
RoPE multiplies each key by a rotation that depends on that key’s position. A matrix that changes with every past token can’t be folded into one fixed query transform. Hence the separate, small rotated key. - Why doesn’t a larger batch make decode attention compute-bound?
Answer
Each sequence has its own cache. A bigger batch adds cache bytes as fast as it adds work, so operations per byte stay the same.
In the map
Methods
- Multi-query attention 2 sources
- Grouped-query attention 12 sources
- Multi-head Latent Attention 3 sources
- Gated Multi-head Latent Attention 3 sources
- Unified Keys and Values 2 sources
Taxonomy nodes
- softmax attention 18 methods
- global attention 5 methods
- grouped-query attention 5 methods
- multi-head latent attention 5 methods
Further reading
- Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019), arXiv:1911.02150. The original MQA argument, framed in memory bandwidth.
- Ainslie et al., GQA (2023), arXiv:2305.13245. The conversion recipe and quality as a function of the number of groups.
- DeepSeek-AI, DeepSeek-V2 (2024), arXiv:2405.04434, §2.1 and appendix. MLA, the decoupled RoPE key, and the comparison with MHA, MQA and GQA.
- Meng et al., TransMLA: Multi-Head Latent Attention Is All You Need (2025), arXiv:2502.07864. Why GQA is a special case of MLA, and converting GQA models.
- Kimi Team, Kimi K2 (2025), arXiv:2507.20534. QK-Clip.
- DeepSeek-V4 report (2026), §2.3, https://arxiv.org/html/2606.19348. Shared key-value MQA and grouped output projection.
- Gemma 4 technical report (2026), https://arxiv.org/html/2607.02770. Keys reused as values in global layers.
- Pope et al., Efficiently Scaling Transformer Inference (2022), arXiv:2211.05102. The memory-bandwidth view of decoding used throughout.
- Michel et al., Are Sixteen Heads Really Better than One? (2019), arXiv:1905.10650; Voita et al., Analyzing Multi-Head Self-Attention (2019), arXiv:1905.09418. Evidence that many heads are redundant.
- Clark et al., What Does BERT Look At? (2019), arXiv:1906.04341. Head specialization.
- Kitaev et al., Reformer (2020), arXiv:2001.04451. Shared query-key attention.
