Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentiongrouped-query attention

methods · softmax attention

grouped-query attention

5 methods filed at this node or below it, from the sources of 10 models.

In practice

Checked as “Grouped-query attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codeadopted
num_attention_heads=16 · num_key_value_heads=4

16 query heads share 4 KV heads via repeat_kv in every attention layer (full and sliding).

num_attention_heads = 16num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_lfm3.py · L218
MiniMax-M3.1-Flashin its codenot stated
num_attention_heads=64 · num_key_value_heads=4

64 query heads share 4 KV heads in every attention layer via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_minimax_m3_vl.py · L406
self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_minimax_m3_vl.py · L410–412
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though the nemotron_h attention implementation file itself was not provided.

llm_config.num_attention_heads = 32llm_config.num_key_value_heads = 2
Step-5-Previewin its codenot stated
num_attention_heads=64 · num_attention_groups=4

64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads.

text_config.num_attention_heads = 64text_config.num_attention_groups = 4
self.num_heads = self.total_num_heads // tp_size
        self.total_num_kv_heads = num_kv_heads
model.py · L923–924
Kolibri-1in its codenot stated
num_attention_heads=48 · num_key_value_heads=4

48 query heads share 4 key/value heads via repeat_kv in every attention layer.

num_attention_heads = 48num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_kolibri1.py · L145
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_kolibri1.py · L121–122
MiMo-V2.6-Flashin its codenot stated
num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8

Full-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv, giving grouped-query attention throughout.

num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_mimo_v2.py · L82–83
MiMo-V2.6-Proin its codenot stated
num_attention_heads=128 · num_key_value_heads=8

128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via repeat_kv grouping.

num_attention_heads = 128num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
Qwen3.8-Flash-Nextin its codenot stated
num_attention_heads=24 · num_key_value_heads=2

Full-attention (indexed_attention) layers project 24 query heads but only 2 key/value heads, repeated via repeat_kv with num_key_value_groups.

text_config.num_attention_heads = 24text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen4_exp.py · L813
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_qwen4_exp.py · L820–822
DeepSeek-V4-Flash-Vision-Expin its codenot stated
num_attention_heads=64 · num_key_value_heads=1

num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads.

num_attention_heads = 64num_key_value_heads = 1
self.num_heads = config.num_attention_heads
        self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L789–790
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
modeling_deepseek_v4.py · L399
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention computation.

num_attention_heads = 32num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True
        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L877–882
Xiaomi-Robotics-1in its codenot stated
num_attention_heads=32 · num_key_value_heads=8

The Qwen3VL text attention uses 32 query heads with only 8 key/value heads, repeated via repeat_kv for grouped-query attention.

vlm_config.text_config.num_attention_heads = 32vlm_config.text_config.num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_mibot.py · L406
Laguna-S-2.1in its codeadopted
num_attention_heads=48 · num_key_value_heads=8

Each layer projects queries at num_attention_heads_per_layer (48 or 72) while K/V use num_key_value_heads=8, shared via repeat_kv groups.

num_key_value_heads = 8num_attention_heads = 48
self.num_key_value_groups = self.num_heads // config.num_key_value_heads
modeling_laguna.py · L334
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_laguna.py · L340–342
Inklingin its codeadopted
num_attention_heads=64 · num_key_value_heads=8

Both full and local attention layers project queries with more heads than K/V heads (64 query heads vs 8 KV heads full, 64 vs 16 local), implementing grouped-query attention.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 8
self.num_total_heads = num_heads
        self.num_total_kv_heads = num_kv_heads
        assert self.num_total_heads % tp_size == 0
        self.num_heads = self.num_total_heads // tp_size
attention.py · L95–98
Hy3in its codeadopted
num_attention_heads=64 · num_key_value_heads=8

64 query heads share 8 KV heads via repeat_kv in every attention layer.

num_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_hy_v3.py · L207
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot stated
num_attention_heads=64 · num_key_value_heads=2

Full-attention layers use 64 query heads but only 2 key/value heads, with repeat_kv expanding the KV heads to match query groups.

num_attention_heads = 64num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_nemotron_h.py · L877
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
modeling_nemotron_h.py · L839–840
MiniMax-M3in its codeadopted
num_attention_heads=64 · num_key_value_heads=4

The language model uses 64 query heads sharing 4 key/value heads via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_minimax_m3_vl.py · L406
self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_minimax_m3_vl.py · L410–412
Step-3.7-Flashin its codenot stated
num_attention_heads=64 · num_attention_groups=8

64 query heads share 8 key/value heads (96 query heads for sliding-attention layers also share 8 KV heads) via repeat_kv grouping.

text_config.num_attention_heads = 64text_config.num_attention_groups = 8
self.num_key_value_groups = self.num_heads // config.num_key_value_heads
modeling_step3p7.py · L678
MolmoAct 2in its codenot stated
num_attention_heads=32 · num_key_value_heads=8

The text model's attention uses 32 query heads with 8 KV heads, repeated via repeat_kv for grouped-query attention.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 8
self.num_key_value_groups = (
            config.num_attention_heads // config.num_key_value_heads
        )
modeling_molmoact2.py · L2202–2204
key_states = repeat_kv(key_states, self.num_key_value_groups)
            value_states = repeat_kv(value_states, self.num_key_value_groups)
modeling_molmoact2.py · L2301–2302
MiMo-V2.5in its codeadopted
num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8

Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv.

num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L259
MiMo-V2.5-Proin its codenot stated
num_attention_heads=128 · num_key_value_heads=8

Full attention layers use 128 query heads sharing 8 key/value heads via repeat_kv grouping.

num_attention_heads = 128num_key_value_heads = 8
self.num_key_value_groups = self.num_attention_heads // self.num_key_value_heads
modeling_mimo_v2.py · L254
Qwen3.6-35B-A3Bin its codenot stated
num_attention_heads=16 · num_key_value_heads=2

Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv (grouped-query attention).

text_config.num_attention_heads = 16text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L739
Gemma 4 31Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=16 · num_global_key_value_heads=4

Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 16text_config.num_global_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // layer_config.num_key_value_heads
modeling_gemma4.py · L1164
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gemma4.py · L826–827
Qwen3.5-397B-A17Bin its codenot stated
num_attention_heads=32 · num_key_value_heads=2

Full-attention layers project 32 query heads down to 2 KV heads, which are repeated via repeat_kv to match the query heads.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L739
self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_qwen3_5_moe.py · L746–748
gpt-oss-120bin its codeadopted
num_attention_heads=64 · num_key_value_heads=8

64 query heads share 8 key/value heads via repeat_kv in every attention layer

num_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_gpt_oss.py · L268
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gpt_oss.py · L237–238
Mistral-Large-4not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64 both) and attention uses MLA-style latent KV rather than GQA with fewer KV heads.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 64
DeepSeek-V4.1-Flash-1017not in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

This model uses DeepSeek-style multi-head latent attention (MLA) rather than GQA; keys/values are not projected with a separate smaller head count shared by query groups.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 1
DeepSeek-V4.1-Flashnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

This model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1 here reflects the MLA latent cache layout, not a GQA KV-head group.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 1
Hy4-previewnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=8

The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads.

num_attention_heads = 64num_key_value_heads = 8
# MLA expands the latent to one key/value per query head, so keys are never grouped.
        self.num_key_value_heads = self.num_attention_heads
configuration_hy_v4.py · L140–141
GLM-5.3-Flashnot in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 64
        if self.num_attention_heads != self.num_key_value_heads:
            raise ValueError(
                f"num_attention_heads ({self.num_attention_heads}) must be the same as "
                f"num_key_value_heads ({self.num_key_value_heads})."
            )
configuration_glm5_next.py · L208–212
GLM-5.3not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64/64) and the attention is MLA, so there is no query grouping over fewer KV heads.

num_attention_heads = 64num_key_value_heads = 64
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_glm_moe_dsa.py · L320
DeepSeek-V4-Pro-0813not in its codenot stated
num_key_value_heads=1

num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V4-Flash-0731not in its codenot stated
num_attention_heads=64 · num_key_value_heads=1

num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head.

num_key_value_heads = 1num_attention_heads = 64
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
Kimi K3not in its codenot stated
num_attention_heads=96 · num_key_value_heads=96

The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway.

text_config.num_attention_heads = 96text_config.num_key_value_heads = 96
self.num_key_value_heads = config.num_key_value_heads
        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
modeling_kimi_linear.py · L346–347
GLM-5.2not in its codenot stated
num_attention_heads=64 · num_key_value_heads=64

num_attention_heads equals num_key_value_heads (64/64) and attention is MLA-based rather than GQA, so there is no head-sharing group.

num_attention_heads = 64num_key_value_heads = 64
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_glm_moe_dsa.py · L320
DeepSeek-V4-Flashnot in its codenot stated
num_key_value_heads=1

Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent.

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V4-Pronot in its codenot stated
num_attention_heads=128 · num_key_value_heads=1

num_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via broadcasting rather than grouped multi-head KV projection, and the model otherwise relies on MLA-style projections, so GQA as a head-grouping mechanism with multiple real KV heads is not present (it's MQA/MLA-like with one shared KV head).

num_key_value_heads = 1num_attention_heads = 128
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
DeepSeek-V3.2not in its codenot stated
num_attention_heads=128 · num_key_value_heads=128

num_attention_heads equals num_key_value_heads (128 each), so num_key_value_groups is 1 and there is no grouping; this model uses MLA rather than GQA.

num_attention_heads = 128num_key_value_heads = 128
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_deepseek_v32.py · L355
X-VLAnot in its codenot stated
verdict and lines

The Florence2 text encoder/decoder attention and the SoftPromptedTransformer attention both use standard multi-head attention with equal numbers of query, key and value heads, with no separate kv head count.

self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
modeling_florence2.py · L763–766
OpenVLA / OpenVLA-OFTcode unclearnot stated
model_type=llama

The language model is loaded via AutoModelForCausalLM.from_config using the Llama-2-7b text_config, but the actual Llama attention implementation (and any num_key_value_heads setting) is not present in the given files or config, so GQA cannot be confirmed from what is provided.

text_config.model_type = "llama"
self.language_model = AutoModelForCausalLM.from_config(
            config.text_config, attn_implementation=config._attn_implementation
        )
modeling_prismatic.py · L248–250

In this branch 5

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 5

Grouped-query attention core · 12 sources · 15 quotes
Multi-query attention used · 2 sources · 2 quotes
3 single mentions
Block-sparse grouped-query attention core · 1 source · 1 quote
Softplus-based per-head attention gating used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑