Model techniques map
Techniquesmodel architecturetoken mixersoftmax attentiongrouped-query attention

specific method · filed under model architecture

Grouped-query attention

An attention method that groups query heads to share key-value heads; the cited eight-key-value-head configuration is not a separate method.

Also called GQA, GQA, 8 KV heads, Grouped Query Attention (GQA), conventional GQA.

sources
11
models
6
labs adopt it
6
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

evaluated 3used 5core 5

Documented in

MiMo-V2.5Hy3MiniMax-M3InklingLaguna-S-2.1gpt-oss-120b

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

In the code

Checked as “Grouped-query attention” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.

ModelCodeIts documentsWhat the code does
GLM-5.3-Flash not in its codenot statednum_attention_heads equals num_key_value_heads (64 == 64) and the config even enforces this in validate_architecture, so no query-head grouping occurs; the attention is MLA-style, not GQA. configuration_glm5_next.py#L209
DeepSeek-V4.1-Flash not in its codenot statedThis model uses multi-head latent attention (MLA) with a single-head latent KV cache rather than GQA; num_key_value_heads=1 reflects the MLA latent, not shared KV head groups. attention.py#L282
Hy4-preview not in its codenot statedThe config sets num_key_value_heads equal to num_attention_heads (MLA overrides it in __post_init__), so there is no head grouping; attention uses MLA instead of GQA. configuration_hy_v4.py#L138
DeepSeek-V4-Flash-0731 in its codenot statednum_key_value_heads=1 gives a single shared KV head (multi-query attention) broadcast to all 64 query heads via repeat_kv/num_key_value_groups. modeling_deepseek_v4.py#L782
NVIDIA-Nemotron-3-Ultra-550B-A55B in its codenot statedFull-attention layers use 64 query heads with only 2 key/value heads, repeated via repeat_kv (grouped-query attention). modeling_nemotron_h.py#L847
MiMo-V2.6-Flash in its codenot statedFull-attention layers use 64 query heads with 4 KV heads (and SWA layers use 64 query heads with 8 KV heads), repeated via repeat_kv, giving grouped-query attention throughout. modeling_mimo_v2.py#L259
DeepSeek-V4-Flash not in its codenot statednum_key_value_heads is 1 (a single shared KV head used as MQA within the core attention), but this is implemented as part of MLA-style latent projection with the K=V trick, not classic GQA with an explicit KV projection sized by num_key_value_heads used as a grouping mechanism; regardless, num_key_value_heads=1 means it's MQA-style sharing which is covered by MLA instead modeling_deepseek_v4.py#L782
MiMo-V2.5 in its code usedFull-attention layers use 64 query heads with 4 KV heads and SWA layers use 64 query heads with 8 KV heads, both grouped via repeat_kv. modeling_mimo_v2.py#L259
GLM-5.3 not in its codenot statednum_attention_heads equals num_key_value_heads (64/64), so there is no query-head grouping over fewer KV heads; this model uses MLA instead of GQA. modeling_glm_moe_dsa.py#L332
Hy3 in its code core64 query heads share 8 key/value heads via repeat_kv in every attention layer modeling_hy_v3.py#L218
GLM-5.2 not in its codenot statednum_attention_heads equals num_key_value_heads (64/64) and the attention is MLA, so there is no key/value head sharing group beyond the single latent projection. modeling_glm_moe_dsa.py#L332
MiniMax-M3 in its code coreThe text attention uses 64 query heads with only 4 key/value heads, repeated via repeat_kv/num_key_value_groups. modeling_minimax_m3_vl.py#L423
DeepSeek-V3.2 not in its codenot statednum_attention_heads equals num_key_value_heads (128/128) and attention uses MLA rather than shared KV head groups, so there is no GQA in the softmax attention layers. modeling_deepseek_v32.py#L369
DeepSeek-V4-Flash-Vision-Exp not in its codenot statednum_key_value_heads=1 but the KV path is a single shared MQA head broadcast via repeat_kv, not a grouped low-rank KV projection paired with per-head KV heads distinct from MLA — but this is MQA which counts as GQA per the definition; however the code implements a single-vector kv_proj (no low-rank latent split for per-head KV), so it is MQA (a degenerate GQA) — present. modeling_deepseek_v4.py#L782
DeepSeek-V4-Pro in its codenot statednum_key_value_heads is 1 (shared-KV multi-query attention) with 128 query heads all broadcasting from a single KV head via repeat_kv modeling_deepseek_v4.py#L781
DeepSeek-V4-Pro-0813 not in its codenot statedThe attention uses a single shared KV head broadcast to all query heads via repeat_kv (multi-query attention with num_key_value_heads=1, not grouped KV heads with independent projection), but the model is MLA-style (kv_proj to a shared low-rank head_dim); num_key_value_groups is set to num_attention_heads (MQA), and true GQA (multiple, but fewer, distinct KV heads) is not implemented. modeling_deepseek_v4.py#L782
Gemma 4 31B in its codenot stated32 query heads share 16 (or 4 for global layers) key/value heads via repeat_kv grouping in Gemma4TextAttention. modeling_gemma4.py#L1176
Inkling in its code usedBoth the global-attention layers (64 query heads / 8 KV heads) and the sliding-window layers (64 query heads / 16 KV heads) use grouped-query attention via the qkvr projection sized by num_kv_heads. attention.py#L103
Kimi K3 not in its codenot statedThe MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway. modeling_kimi_linear.py#L346
Laguna-S-2.1 in its code coreEach layer's queries (48 or 72 heads) share 8 key/value heads via repeat_kv-based grouped-query attention. modeling_laguna.py#L345
MiMo-V2.5-Pro in its codenot statedFull attention layers use 128 query heads sharing 8 key/value heads via repeat_kv grouping. modeling_mimo_v2.py#L254
MiMo-V2.6-Pro in its codenot stated128 query heads share 8 key/value heads in full-attention layers (and 8 KV heads for the 128 SWA query heads too), implemented via repeat_kv grouping. modeling_mimo_v2.py#L259
NVIDIA-Nemotron-3.5-Lightning-30B-A3B in its codenot statedFull-attention layers project 32 query heads down to 2 KV heads which are repeated via repeat_kv before the dot product. modeling_nemotron_h.py#L847
Qwen3.5-397B-A17B in its codenot statedFull-attention layers use 32 query heads with only 2 key/value heads, repeated via repeat_kv, giving grouped-query attention. modeling_qwen3_5_moe.py#L759
Qwen3.6-35B-A3B in its codenot statedFull-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv for grouped-query attention. modeling_qwen3_5_moe.py#L759
Qwen3.8-Flash-Next in its codenot statedFull/indexed attention layers project 24 query heads down to 2 KV heads and repeat_kv expands them for the dot product. modeling_qwen4_exp.py#L827
Step-3.7-Flash in its codenot statedText attention uses 64 query heads with 8 shared KV heads (num_key_value_groups = num_heads // num_key_value_heads) and repeat_kv expands the KV heads per group. modeling_step3p7.py#L688
gpt-oss-120b in its code used64 query heads share 8 key/value heads via repeat_kv grouping in every attention layer. modeling_gpt_oss.py#L276

Evidence

13 spans quoted from the sources, strongest treatment first.

Attention Heads | 64 (GQA, 8 KV heads, head dim 128)

coremodel architecturein Hy3Tencent Hunyuan
reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)·Tencent Hunyuan

MSA is built upon Grouped Query Attention (GQA).

coremodel architecturein MiniMax-M3MiniMax
MiniMax Sparse Attention·MiniMax·2026-06-11

GQA as the substrate, not MLA.

coremodel architecturein MiniMax-M3MiniMax
MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram·Hugging Face community blog·2026-05-29

grouped-query attention

coremodel architecturein Laguna S 2.1Poolside
poolside/Laguna-S-2.1 · Hugging Face·Poolside·2026-08-03

GQA with 64 heads over 8 KV heads

coremodel architecturein Hy3Tencent
tencent/Hy3 — 295B / 21B active · MOE · 256K ctx·vLLM·2026-07-06

uses Grouped Query Attention (GQA [12][13]) with 8 key-value heads

usedmodel architecturein gpt-oss-120b and gpt-oss-20bOpenAI
gpt-oss-120b & gpt-oss-20b Model Card·OpenAI·2025-08-08

Each attention module uses GQA with 32 query heads, 4 KV heads

usedmodel architecturein MiniMax Sparse AttentionMiniMax
MiniMax Sparse Attention·MiniMax·2026-06-11

the use of conventional GQA rather than MLA or a recurrent hybrid stack suggest that raw decoding speed may not be Inkling’s main advantage

usedmodel architecturein InklingThinking Machines Lab
Inkling: A New Open-Weight 975B MoE with a Few Surprises·Sebastian Raschka·2026-07-16

In both types of layers, we use Grouped Query Attention (GQA) with 8 KV heads

usedmodel architecturein Laguna XS.2Poolside
laguna m1 xs2 technical report·Poolside

Both SWA and GA utilize Grouped-Query Attention (GQA)

usedmodel architecturein MiMo-V2-FlashXiaomi
MiMo-V2-Flash Technical Report·Xiaomi LLM-Core·2026-01-06

MSA matches GQA on downstream benchmarks while delivering prefill and decoding speedups at 1M context length.

evaluatedevaluation onlyMiniMax
MiniMax Sparse Attention·MiniMax·2026-06-11

Compared with GQA, MSA dramatically reduces the attention compute and memory footprint

evaluatedmodel architecturein MiniMax-M3MiniMax
README·MiniMax

Compared with GQA, MSA dramatically reduces the attention compute and memory footprint while preserving model quality.

evaluatedmodel architecturein MiniMax-M3MiniMax
MiniMaxAI/MiniMax-M3 · Hugging Face·MiniMax·2026-06-11

Filed alongside

Other methods under model architecture :: token mixer :: softmax attention :: grouped-query attention.

Multi-query attentionBlock-sparse grouped-query attentionSoftplus-based per-head gating