Model techniques map
Techniquesmodel architecturetoken mixersoftmax attentionmulti-head latent attention

implementation detail · filed under model architecture

Multi-Head Latent Attention MHA/MQA Modes

An MLA configuration that uses MHA mode for training and prefilling and MQA mode for decoding.

Also called MHA and MQA modes of MLA.

source
1
model
1
lab adopt it
1
strongest
used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1

Documented in

DeepSeek-V3.2

Evidence

1 span quoted from the sources, strongest treatment first.

For DeepSeek-V3.1-Terminus, the MHA mode is used for training and prefilling, while the MQA mode is used for decoding.

usedmodel architecturein DeepSeek-V3.1-TerminusDeepSeek
paper·DeepSeek

Filed alongside

Other methods under model architecture :: token mixer :: softmax attention :: multi-head latent attention.

Gated Multi-Head Latent AttentionMulti-Head Latent AttentionMQA Mode of Multi-Head Latent Attention