Grouped-Query Attention uses query heads and reduces the number of key-value heads by tying adjacent query heads to a single shared key-value head.1 Each key-value head defines one GQA group, which helps address the computational cost of standard Softmax Attention that grows quadratically with the sequence length.2,3
Trade-off The KV cache does not shrink along with attention sparsification, though this trade-off is described as a sensible bargain.4
How this definition was written
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
Grouped-Query Attention [Ainslie et al., 2023] uses query heads and reduces the number of key-value heads to , tying adjacent query heads to a single shared key-value head — MiniMax Sparse Attention
The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though the nemotron_h attention implementation file itself was not provided.
64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads.
num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads.
Both full and local attention layers project queries with more heads than K/V heads (64 query heads vs 8 KV heads full, 64 vs 16 local), implementing grouped-query attention.
Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA.
This model uses DeepSeek-style multi-head latent attention (MLA) rather than GQA; keys/values are not projected with a separate smaller head count shared by query groups.
This model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1 here reflects the MLA latent cache layout, not a GQA KV-head group.
The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads.
num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA.
if self.num_attention_heads != self.num_key_value_heads:
raise ValueError(
f"num_attention_heads ({self.num_attention_heads}) must be the same as "
f"num_key_value_heads ({self.num_key_value_heads})."
)
num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention
num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head.
The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway.
Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent.
num_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via broadcasting rather than grouped multi-head KV projection, and the model otherwise relies on MLA-style projections, so GQA as a head-grouping mechanism with multiple real KV heads is not present (it's MQA/MLA-like with one shared KV head).
The Florence2 text encoder/decoder attention and the SoftPromptedTransformer attention both use standard multi-head attention with equal numbers of query, key and value heads, with no separate kv head count.
The language model is loaded via AutoModelForCausalLM.from_config using the Llama-2-7b text_config, but the actual Llama attention implementation (and any num_key_value_heads setting) is not present in the given files or config, so GQA cannot be confirmed from what is provided.