Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentiongrouped-query attention

specific method · filed under model architecture

Multi-query attention

Multi-query attention (MQA) is a core attention mechanism performed after selecting sparse KV entries.1 In this manner, each compressed KV entry serves as both the attention key and the attention value.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. After selecting the sparse KV entries, CSA then performs core attention in a Multi-Query Attention (MQA) [mqa] manner — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. where each compressed KV entry in serves as both attention key and value — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called Multi-Query Attention (MQA), Multi-Query Attention (MQA) mode of MLA.

sources
2
models
3
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
mentioned 1used 1
↑