Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentiongrouped-query attention

specific method · filed under model architecture

Grouped-query attention

Grouped-Query Attention uses query heads and reduces the number of key-value heads by tying adjacent query heads to a single shared key-value head.1 Each key-value head defines one GQA group, which helps address the computational cost of standard Softmax Attention that grows quadratically with the sequence length.2,3

Trade-off The KV cache does not shrink along with attention sparsification, though this trade-off is described as a sensible bargain.4

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. Grouped-Query Attention [Ainslie et al., 2023] uses query heads and reduces the number of key-value heads to , tying adjacent query heads to a single shared key-value head — MiniMax Sparse Attention
  2. Thus, each key-value head defines one GQA group — MiniMax Sparse Attention
  3. The cost of eq.˜1 is FLOPs, which grows quadratically with the sequence length — MiniMax Sparse Attention
  4. The cost is that the KV cache doesn't shrink along with attention sparsification — but trading token economy for quality is a sensible bargain — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram

Also called GQA, GQA, 8 KV heads, Grouped Query Attention (GQA), conventional GQA, Key/value sharing across query-head groups.

sources
12
models
7
labs adopt it
7
strongest
core

In practice

Checked as “Grouped-query attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codeadopted
num_attention_heads=16 · num_key_value_heads=4

16 query heads share 4 KV heads via repeat_kv in every attention layer (full and sliding).

num_attention_heads = 16num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_lfm3.py · L218
MiniMax-M3.1-Flashin its codenot stated
num_attention_heads=64 · num_key_value_heads=4

64 query heads share 4 KV heads in every attention layer via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_minimax_m3_vl.py · L406
self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_minimax_m3_vl.py · L410–412
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though the nemotron_h attention implementation file itself was not provided.

llm_config.num_attention_heads = 32llm_config.num_key_value_heads = 2
Step-5-Previewin its codenot stated
num_attention_heads=64 · num_attention_groups=4

64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads.

text_config.num_attention_heads = 64text_config.num_attention_groups = 4
self.num_heads = self.total_num_heads // tp_size
        self.total_num_kv_heads = num_kv_heads
model.py · L923–924
Kolibri-1in its codenot stated
num_attention_heads=48 · num_key_value_heads=4

48 query heads share 4 key/value heads via repeat_kv in every attention layer.

num_attention_heads = 48num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_kolibri1.py · L145
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_kolibri1.py · L121–122
MiMo-V2.6-Flashin its codenot stated
num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8

Full-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv, giving grouped-query attention throughout.

num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_mimo_v2.py · L82–83
MiMo-V2.6-Proin its codenot stated
num_attention_heads=128 · num_key_value_heads=8

128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via repeat_kv grouping.

num_attention_heads = 128num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
Qwen3.8-Flash-Nextin its codenot stated
num_attention_heads=24 · num_key_value_heads=2

Full-attention (indexed_attention) layers project 24 query heads but only 2 key/value heads, repeated via repeat_kv with num_key_value_groups.

text_config.num_attention_heads = 24text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen4_exp.py · L813
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_qwen4_exp.py · L820–822
DeepSeek-V4-Flash-Vision-Expin its codenot stated
num_attention_heads=64 · num_key_value_heads=1

num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads.

num_attention_heads = 64num_key_value_heads = 1
self.num_heads = config.num_attention_heads
        self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L789–790
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
modeling_deepseek_v4.py · L399
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention computation.

num_attention_heads = 32num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True
        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L877–882
Xiaomi-Robotics-1in its codenot stated
num_attention_heads=32 · num_key_value_heads=8

The Qwen3VL text attention uses 32 query heads with only 8 key/value heads, repeated via repeat_kv for grouped-query attention.

vlm_config.text_config.num_attention_heads = 32vlm_config.text_config.num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_mibot.py · L406
Laguna-S-2.1in its codeadopted
num_attention_heads=48 · num_key_value_heads=8

Each layer projects queries at num_attention_heads_per_layer (48 or 72) while K/V use num_key_value_heads=8, shared via repeat_kv groups.

num_key_value_heads = 8num_attention_heads = 48
self.num_key_value_groups = self.num_heads // config.num_key_value_heads
modeling_laguna.py · L334
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_laguna.py · L340–342
Inklingin its codeadopted
num_attention_heads=64 · num_key_value_heads=8

Both full and local attention layers project queries with more heads than K/V heads (64 query heads vs 8 KV heads full, 64 vs 16 local), implementing grouped-query attention.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 8
self.num_total_heads = num_heads
        self.num_total_kv_heads = num_kv_heads
        assert self.num_total_heads % tp_size == 0
        self.num_heads = self.num_total_heads // tp_size
attention.py · L95–98
Hy3in its codeadopted
num_attention_heads=64 · num_key_value_heads=8

64 query heads share 8 KV heads via repeat_kv in every attention layer.

num_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_hy_v3.py · L207
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot stated
num_attention_heads=64 · num_key_value_heads=2

Full-attention layers use 64 query heads but only 2 key/value heads, with repeat_kv expanding the KV heads to match query groups.

num_attention_heads = 64num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_nemotron_h.py · L877
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
modeling_nemotron_h.py · L839–840
MiniMax-M3in its codeadopted
num_attention_heads=64 · num_key_value_heads=4

The language model uses 64 query heads sharing 4 key/value heads via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_minimax_m3_vl.py · L406
self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_minimax_m3_vl.py · L410–412
Step-3.7-Flashin its codenot stated
num_attention_heads=64 · num_attention_groups=8

64 query heads share 8 key/value heads (96 query heads for sliding-attention layers also share 8 KV heads) via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_attention_groups = 8
self.num_key_value_groups = self.num_heads // config.num_key_value_heads
modeling_step3p7.py · L678
MolmoAct 2in its codenot stated
num_attention_heads=32 · num_key_value_heads=8

The text model's attention uses 32 query heads with 8 KV heads, repeated via repeat_kv for grouped-query attention.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 8
self.num_key_value_groups = (
            config.num_attention_heads // config.num_key_value_heads
        )
modeling_molmoact2.py · L2202–2204
key_states = repeat_kv(key_states, self.num_key_value_groups)
            value_states = repeat_kv(value_states, self.num_key_value_groups)
modeling_molmoact2.py · L2301–2302
MiMo-V2.5in its codeadopted
num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8

Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv.

num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
MiMo-V2.5-Proin its codenot stated
num_attention_heads=128 · num_key_value_heads=8

Full attention layers use 128 query heads sharing 8 key/value heads via repeat_kv grouping.

num_attention_heads = 128num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L254
Qwen3.6-35B-A3Bin its codenot stated
num_attention_heads=16 · num_key_value_heads=2

Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv (grouped-query attention).

text_config.num_attention_heads = 16text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L739
Gemma 4 31Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=16 · num_global_key_value_heads=4

Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 16text_config.num_global_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // layer_config.num_key_value_heads
modeling_gemma4.py · L1164
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gemma4.py · L826–827
Qwen3.5-397B-A17Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

Full-attention layers project 32 query heads down to 2 KV heads, which are repeated via repeat_kv to match the query heads.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L739
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_qwen3_5_moe.py · L746–748
gpt-oss-120bin its codeadopted
num_attention_heads=64 · num_key_value_heads=8

64 query heads share 8 key/value heads via repeat_kv in every attention layer

num_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_gpt_oss.py · L268
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gpt_oss.py · L237–238
Mistral-Large-4not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64 both) and attention uses MLA-style latent KV rather than GQA with fewer KV heads.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 64
DeepSeek-V4.1-Flash-1017not in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

This model uses DeepSeek-style multi-head latent attention (MLA) rather than GQA; keys/values are not projected with a separate smaller head count shared by query groups.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 1
DeepSeek-V4.1-Flashnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

This model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1 here reflects the MLA latent cache layout, not a GQA KV-head group.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 1
Hy4-previewnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=8

The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads.

num_attention_heads = 64num_key_value_heads = 8
# MLA expands the latent to one key/value per query head, so keys are never grouped.
        self.num_key_value_heads = self.num_attention_heads
configuration_hy_v4.py · L140–141
GLM-5.3-Flashnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 64
        if self.num_attention_heads != self.num_key_value_heads:
            raise ValueError(
                f"num_attention_heads ({self.num_attention_heads}) must be the same as "
                f"num_key_value_heads ({self.num_key_value_heads})."
            )
configuration_glm5_next.py · L208–212
GLM-5.3not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64/64) and the attention is MLA, so there is no query grouping over fewer KV heads.

num_attention_heads = 64num_key_value_heads = 64
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_glm_moe_dsa.py · L320
DeepSeek-V4-Pro-0813not in its codenot stated
num_key_value_heads=1

num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V4-Flash-0731not in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head.

num_key_value_heads = 1num_attention_heads = 64
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
Kimi K3not in its codenot stated
num_attention_heads=96 · num_key_value_heads=96

The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway.

text_config.num_attention_heads = 96text_config.num_key_value_heads = 96
self.num_key_value_heads = config.num_key_value_heads
        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
modeling_kimi_linear.py · L346–347
GLM-5.2not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64/64) and attention is MLA-based rather than GQA, so there is no head-sharing group.

num_attention_heads = 64num_key_value_heads = 64
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_glm_moe_dsa.py · L320
DeepSeek-V4-Flashnot in its codenot stated
num_key_value_heads=1

Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent.

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V4-Pronot in its codenot stated
num_attention_heads=128 · num_key_value_heads=1

num_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via broadcasting rather than grouped multi-head KV projection, and the model otherwise relies on MLA-style projections, so GQA as a head-grouping mechanism with multiple real KV heads is not present (it's MQA/MLA-like with one shared KV head).

num_key_value_heads = 1num_attention_heads = 128
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V3.2not in its codenot stated
num_attention_heads=128 · num_key_value_heads=128

num_attention_heads equals num_key_value_heads (128 each), so num_key_value_groups is 1 and there is no grouping; this model uses MLA rather than GQA.

num_attention_heads = 128num_key_value_heads = 128
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_deepseek_v32.py · L355
X-VLAnot in its codenot stated
verdict and lines

The Florence2 text encoder/decoder attention and the SoftPromptedTransformer attention both use standard multi-head attention with equal numbers of query, key and value heads, with no separate kv head count.

self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
modeling_florence2.py · L763–766
OpenVLA / OpenVLA-OFTcode unclearnot stated
model_type=llama

The language model is loaded via AutoModelForCausalLM.from_config using the Llama-2-7b text_config, but the actual Llama attention implementation (and any num_key_value_heads setting) is not present in the given files or config, so GQA cannot be confirmed from what is provided.

text_config.model_type = "llama"
self.language_model = AutoModelForCausalLM.from_config(
            config.text_config, attn_implementation=config._attn_implementation
        )
modeling_prismatic.py · L248–250

Documented in

Liquid AI
MiniMax
Poolside
Tencent (Hunyuan)
Thinking Machines Lab
Xiaomi

Evidence

15 quotes from 12 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 3used 5core 7

MiniMax-M3

MiniMaxadopted4 sources

Each attention module uses GQA with 32 query heads, 4 KV heads

usedmodel architecturein MiniMax Sparse AttentionMiniMax
5 more from this model

Compared with GQA, MSA dramatically reduces the attention compute and memory footprint while preserving model quality.

evaluatedmodel architecturein MiniMax-M3MiniMax

MSA matches GQA on downstream benchmarks while delivering prefill and decoding speedups at 1M context length.

evaluatedevaluation onlyMiniMax

Compared with GQA, MSA dramatically reduces the attention compute and memory footprint

evaluatedmodel architecturein MiniMax-M3MiniMax

MSA is built upon Grouped Query Attention (GQA).

coremodel architecturein MiniMax-M3MiniMax

GQA as the substrate, not MLA.

coremodel architecturein MiniMax-M3MiniMax

LFM3-1.7B

Liquid AIadopted1 source

interleaved with a small minority of GQA blocks

coremodel architecturein LFM2Liquid AI
1 more from this model

GQA reduces KV traffic by sharing keys/values across head groups while preserving multi-head queries

coremodel architecturein LFM2Liquid AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑