MQA Mode of Multi-Head Latent Attention

The MLA mode in which each latent key-value vector is shared across all query heads.

Also called MQA mode of MLA.

sources: 2
model: 1
lab adopt it: 1
strongest: core

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1core 1

Documented in

DeepSeek-V3.2

Evidence

2 spans quoted from the sources, strongest treatment first.

Therefore, we implement DSA based on the MQA (Shazeer, 2019) mode of MLA, where each latent vector (the key-value entry of MLA) will be shared across all query heads of the query token.

coremodel architecturein DeepSeek-V3.2DeepSeek

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models·DeepSeek·2025-12-02

we implement DSA based on the MQA mode of MLA, where each latent vector (the key-value entry of MLA) will be shared across all query heads of the query token.

usedmodel architecturein DeepSeek-V3.2DeepSeek

paper·DeepSeek

Filed alongside

Other methods under model architecture :: token mixer :: softmax attention :: multi-head latent attention.

Gated Multi-Head Latent Attention Multi-Head Latent Attention Multi-Head Latent Attention MHA/MQA Modes