DeepSeek-V4-Flash · DeepSeek-V4-Pro
DeepSeekCSA then performs core attention in a Multi-Query Attention (MQA) manner
specific method · filed under model architecture
Multi-query attention (MQA) is a core attention mechanism performed after selecting sparse KV entries.1 In this manner, each compressed KV entry serves as both the attention key and the attention value.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
After selecting the sparse KV entries, CSA then performs core attention in a Multi-Query Attention (MQA) [mqa] manner— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
where each compressed KV entry in serves as both attention key and value— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Also called Multi-Query Attention (MQA), Multi-Query Attention (MQA) mode of MLA.
2 quotes from 2 sources, one card per model, each opening with its strongest.
CSA then performs core attention in a Multi-Query Attention (MQA) manner
In the future, we plan to migrate the inference backend to the training engine and uniformly adopt the Multi-Query Attention (MQA) mode of MLA for inference