specific method · filed under model architecture
Multi-query attention
An attention method described here as performing core attention in an MQA manner, including an MQA mode of MLA.
Also called Multi-Query Attention (MQA), Multi-Query Attention (MQA) mode of MLA.
- sources
- 2
- models
- 3
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
mentioned 1used 1
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
CSA then performs core attention in a Multi-Query Attention (MQA) manner
usedmodel architecturein DeepSeek-V4DeepSeek
In the future, we plan to migrate the inference backend to the training engine and uniformly adopt the Multi-Query Attention (MQA) mode of MLA for inference
mentionedmodel architecturein GLM-5Z.ai
Filed alongside
Other methods under model architecture :: token mixer :: softmax attention :: grouped-query attention.