methods · softmax attention
grouped-query attention
5 methods filed at this node or below it, from the sources of 10 models.
In practice
Checked as “Grouped-query attention” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| LFM3-1.7B | in its code | adopted | num_attention_heads=16 · num_key_value_heads=416 query heads share 4 KV heads via repeat_kv in every attention layer (full and sliding). num_attention_heads = 16num_key_value_heads = 4 | code → |
| MiniMax-M3.1-Flash | in its code | not stated | num_attention_heads=64 · num_key_value_heads=464 query heads share 4 KV heads in every attention layer via repeat_kv grouping. text_config.num_attention_heads = 64text_config.num_key_value_heads = 4 | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | not stated | num_attention_heads=32 · num_key_value_heads=2The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though the nemotron_h attention implementation file itself was not provided. llm_config.num_attention_heads = 32llm_config.num_key_value_heads = 2 | code → |
| Step-5-Preview | in its code | not stated | num_attention_heads=64 · num_attention_groups=464 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads. text_config.num_attention_heads = 64text_config.num_attention_groups = 4 | code → |
| Kolibri-1 | in its code | not stated | num_attention_heads=48 · num_key_value_heads=448 query heads share 4 key/value heads via repeat_kv in every attention layer. num_attention_heads = 48num_key_value_heads = 4 | code → |
| MiMo-V2.6-Flash | in its code | not stated | num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8Full-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv, giving grouped-query attention throughout. num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8 | code → |
| MiMo-V2.6-Pro | in its code | not stated | num_attention_heads=128 · num_key_value_heads=8128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via repeat_kv grouping. num_attention_heads = 128num_key_value_heads = 8 | code → |
| Qwen3.8-Flash-Next | in its code | not stated | num_attention_heads=24 · num_key_value_heads=2Full-attention (indexed_attention) layers project 24 query heads but only 2 key/value heads, repeated via repeat_kv with num_key_value_groups. text_config.num_attention_heads = 24text_config.num_key_value_heads = 2 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | num_attention_heads=64 · num_key_value_heads=1num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads. num_attention_heads = 64num_key_value_heads = 1 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | num_attention_heads=32 · num_key_value_heads=2Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention computation. num_attention_heads = 32num_key_value_heads = 2 | code → |
| Xiaomi-Robotics-1 | in its code | not stated | num_attention_heads=32 · num_key_value_heads=8The Qwen3VL text attention uses 32 query heads with only 8 key/value heads, repeated via repeat_kv for grouped-query attention. vlm_config.text_config.num_attention_heads = 32vlm_config.text_config.num_key_value_heads = 8 | code → |
| Laguna-S-2.1 | in its code | adopted | num_attention_heads=48 · num_key_value_heads=8Each layer projects queries at num_attention_heads_per_layer (48 or 72) while K/V use num_key_value_heads=8, shared via repeat_kv groups. num_key_value_heads = 8num_attention_heads = 48 | code → |
| Inkling | in its code | adopted | num_attention_heads=64 · num_key_value_heads=8Both full and local attention layers project queries with more heads than K/V heads (64 query heads vs 8 KV heads full, 64 vs 16 local), implementing grouped-query attention. text_config.num_attention_heads = 64text_config.num_key_value_heads = 8 | code → |
| Hy3 | in its code | adopted | num_attention_heads=64 · num_key_value_heads=864 query heads share 8 KV heads via repeat_kv in every attention layer. num_attention_heads = 64num_key_value_heads = 8 | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not stated | num_attention_heads=64 · num_key_value_heads=2Full-attention layers use 64 query heads but only 2 key/value heads, with repeat_kv expanding the KV heads to match query groups. num_attention_heads = 64num_key_value_heads = 2 | code → |
| MiniMax-M3 | in its code | adopted | num_attention_heads=64 · num_key_value_heads=4The language model uses 64 query heads sharing 4 key/value heads via repeat_kv grouping. text_config.num_attention_heads = 64text_config.num_key_value_heads = 4 | code → |
| Step-3.7-Flash | in its code | not stated | num_attention_heads=64 · num_attention_groups=864 query heads share 8 key/value heads (96 query heads for sliding-attention layers also share 8 KV heads) via repeat_kv grouping. text_config.num_attention_heads = 64text_config.num_attention_groups = 8 | code → |
| MolmoAct 2 | in its code | not stated | num_attention_heads=32 · num_key_value_heads=8The text model's attention uses 32 query heads with 8 KV heads, repeated via repeat_kv for grouped-query attention. text_config.num_attention_heads = 32text_config.num_key_value_heads = 8 | code → |
| MiMo-V2.5 | in its code | adopted | num_attention_heads=64 · num_key_value_heads=4 · swa_num_key_value_heads=8Full-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv. num_attention_heads = 64num_key_value_heads = 4swa_num_key_value_heads = 8 | code → |
| MiMo-V2.5-Pro | in its code | not stated | num_attention_heads=128 · num_key_value_heads=8Full attention layers use 128 query heads sharing 8 key/value heads via repeat_kv grouping. num_attention_heads = 128num_key_value_heads = 8 | code → |
| Qwen3.6-35B-A3B | in its code | not stated | num_attention_heads=16 · num_key_value_heads=2Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv (grouped-query attention). text_config.num_attention_heads = 16text_config.num_key_value_heads = 2 | code → |
| Gemma 4 31B | in its code | not stated | num_attention_heads=32 · num_key_value_heads=16 · num_global_key_value_heads=4Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA. text_config.num_attention_heads = 32text_config.num_key_value_heads = 16text_config.num_global_key_value_heads = 4 | code → |
| Qwen3.5-397B-A17B | in its code | not stated | num_attention_heads=32 · num_key_value_heads=2Full-attention layers project 32 query heads down to 2 KV heads, which are repeated via repeat_kv to match the query heads. text_config.num_attention_heads = 32text_config.num_key_value_heads = 2 | code → |
| gpt-oss-120b | in its code | adopted | num_attention_heads=64 · num_key_value_heads=864 query heads share 8 key/value heads via repeat_kv in every attention layer num_attention_heads = 64num_key_value_heads = 8 | code → |
| Mistral-Large-4 | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=64num_attention_heads equals num_key_value_heads (64 both) and attention uses MLA-style latent KV rather than GQA with fewer KV heads. text_config.num_attention_heads = 64text_config.num_key_value_heads = 64 | code → |
| DeepSeek-V4.1-Flash-1017 | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=1This model uses DeepSeek-style multi-head latent attention (MLA) rather than GQA; keys/values are not projected with a separate smaller head count shared by query groups. text_config.num_attention_heads = 64text_config.num_key_value_heads = 1 | code → |
| DeepSeek-V4.1-Flash | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=1This model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1 here reflects the MLA latent cache layout, not a GQA KV-head group. text_config.num_attention_heads = 64text_config.num_key_value_heads = 1 | code → |
| Hy4-preview | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=8The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads. num_attention_heads = 64num_key_value_heads = 8 | code → |
| GLM-5.3-Flash | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=64num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA. text_config.num_attention_heads = 64text_config.num_key_value_heads = 64 | code → |
| GLM-5.3 | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=64num_attention_heads equals num_key_value_heads (64/64) and the attention is MLA, so there is no query grouping over fewer KV heads. num_attention_heads = 64num_key_value_heads = 64 | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | num_key_value_heads=1num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention num_key_value_heads = 1 | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=1num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head. num_key_value_heads = 1num_attention_heads = 64 | code → |
| Kimi K3 | not in its code | not stated | num_attention_heads=96 · num_key_value_heads=96The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway. text_config.num_attention_heads = 96text_config.num_key_value_heads = 96 | code → |
| GLM-5.2 | not in its code | not stated | num_attention_heads=64 · num_key_value_heads=64num_attention_heads equals num_key_value_heads (64/64) and attention is MLA-based rather than GQA, so there is no head-sharing group. num_attention_heads = 64num_key_value_heads = 64 | code → |
| DeepSeek-V4-Flash | not in its code | not stated | num_key_value_heads=1Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent. num_key_value_heads = 1 | code → |
| DeepSeek-V4-Pro | not in its code | not stated | num_attention_heads=128 · num_key_value_heads=1num_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via broadcasting rather than grouped multi-head KV projection, and the model otherwise relies on MLA-style projections, so GQA as a head-grouping mechanism with multiple real KV heads is not present (it's MQA/MLA-like with one shared KV head). num_key_value_heads = 1num_attention_heads = 128 | code → |
| DeepSeek-V3.2 | not in its code | not stated | num_attention_heads=128 · num_key_value_heads=128num_attention_heads equals num_key_value_heads (128 each), so num_key_value_groups is 1 and there is no grouping; this model uses MLA rather than GQA. num_attention_heads = 128num_key_value_heads = 128 | code → |
| X-VLA | not in its code | not stated | verdict and linesThe Florence2 text encoder/decoder attention and the SoftPromptedTransformer attention both use standard multi-head attention with equal numbers of query, key and value heads, with no separate kv head count. | code → |
| OpenVLA / OpenVLA-OFT | code unclear | not stated | model_type=llamaThe language model is loaded via AutoModelForCausalLM.from_config using the Llama-2-7b text_config, but the actual Llama attention implementation (and any num_key_value_heads setting) is not present in the given files or config, so GQA cannot be confirmed from what is provided. text_config.model_type = "llama" | code → |
In this branch 5
Everything filed at this node or below it, with one collapsible heading per child node.
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| Hy3 core | Grouped-query attention core— |
| DeepSeek-V4-Flash used | Multi-query attention used— |
| DeepSeek-V4-Pro used | Multi-query attention used— |
| GLM-5.2 mentioned | Multi-query attention mentioned— |
| Inkling used | Grouped-query attention used— |
| LFM3-1.7B core | Grouped-query attention coreSparse use of grouped-query attention blocks for global context core— |
| Laguna-S-2.1 core | Grouped-query attention coreSoftplus-based per-head attention gating used— |
| MiMo-V2.5 used | Grouped-query attention used— |
| MiniMax-M3 core | Block-sparse grouped-query attention coreGrouped-query attention core— |
| gpt-oss-120b used | Grouped-query attention used— |