specific method · filed under model architecture
Multi-Head Latent Attention
An attention method that uses reduced latent key-value vectors, which can be up-projected to reconstruct content keys and values during computation.
Also called MLA, Multi-latent Attention, Multi-latent attention (MLA).
- sources
- 2
- models
- 2
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1core 1not used 1
Documented in
In the code
Checked as “Multi-head latent attention” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | Softmax attention layers use DeepSeek-V2-style latent KV compression (kv_lora_rank=512, q_lora_rank=1536) with kv_b_proj up-projection per head, as implemented in Glm5NextTextAttention. modeling_glm5_next.py#L1132 |
| DeepSeek-V4.1-Flash | in its code | not stated | Keys/values are compressed into a shared low-rank latent (kv_norm/compressor) and queries/keys use a decoupled RoPE dimension, DeepSeek-V2/V3 style MLA. attention.py#L285 |
| Hy4-preview | in its code | not stated | Attention down-projects hidden states into a shared kv_lora_rank latent (kv_a_proj_with_mqa/kv_a_layernorm) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA with a decoupled RoPE key. modeling_hy_v4.py#L361 |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | Attention uses a shared single KV head projected directly to head_dim (MQA) with q/o low-rank projections, not a DeepSeek-V2-style down/up-projected latent KV cache; kv_lora_rank is absent from config. modeling_deepseek_v4.py#L793 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | The attention module is a standard multi-head/GQA attention with separate q/k/v projections; there is no low-rank KV latent projection or decoupled RoPE key mechanism. modeling_nemotron_h.py#L851 |
| MiMo-V2.6-Flash | not in its code | not stated | There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections with GQA instead. |
| DeepSeek-V4-Flash | in its code | not stated | Queries and KV are both down-projected through low-rank LoRA projections (q_lora_rank, and a shared kv_proj to head_dim) then up-projected, with a decoupled/partial RoPE slice, matching DeepSeek-style MLA. modeling_deepseek_v4.py#L789 |
| MiMo-V2.5 | not in its code | not stated | There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections. |
| GLM-5.3 | in its code | not stated | Keys/values are compressed into a shared low-rank kv_lora_rank latent (with a decoupled RoPE key) and up-projected per head via kv_b_proj, DeepSeek-V2/V3 style MLA. modeling_glm_moe_dsa.py#L339 |
| Hy3 | not in its code | not stated | Attention uses standard separate q/k/v projections sized by head counts, with no low-rank latent kv projection anywhere in the code |
| GLM-5.2 | in its code | core | Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3-style MLA. modeling_glm_moe_dsa.py#L339 |
| MiniMax-M3 | not in its code | not used | There is no latent KV down-projection/up-projection (no kv_lora_rank/q_lora_rank in config); attention uses standard per-head K/V projections instead. |
| DeepSeek-V3.2 | in its code | not stated | Keys/values are compressed into a shared kv_lora_rank latent and up-projected per head via kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA. modeling_deepseek_v32.py#L376 |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | There is no shared low-rank KV latent that is up-projected per head with a decoupled RoPE key (DeepSeek-V2 style); instead keys/values collapse to a single shared vector of size head_dim (MQA), with q using a low-rank latent only for queries, not keys/values. modeling_deepseek_v4.py#L793 |
| DeepSeek-V4-Pro | not in its code | not stated | Attention uses a single shared KV head with q_lora_rank down-projection for queries but no kv_lora_rank latent down/up-projection for keys/values as in DeepSeek-V2 MLA; kv_proj projects directly to one head_dim-sized KV, not a shared low-rank latent modeling_deepseek_v4.py#L793 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | Queries and KV are down-projected via low-rank q_lora_rank/kv_proj (to a single head_dim latent) and up-projected per head, with a decoupled rope key applied to the trailing rope slice, DeepSeek-V2/V3-MLA style. modeling_deepseek_v4.py#L789 |
| Gemma 4 31B | not in its code | not stated | No low-rank KV latent projection (kv_lora_rank/q_lora_rank) exists in the text attention implementation; attention uses standard per-head k/v projections. |
| Inkling | not in its code | not stated | There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config, and attention.py projects K/V directly with per-head width, not a shared latent). |
| Kimi K3 | in its code | not stated | KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style. modeling_kimi_linear.py#L378 |
| Laguna-S-2.1 | not in its code | not stated | Attention uses separate q/k/v projections sized by head_dim and num_key_value_heads with no low-rank latent cache, so MLA is not implemented. modeling_laguna.py#L350 |
| MiMo-V2.5-Pro | not in its code | not stated | No latent KV down/up projection mechanism (kv_lora_rank/q_lora_rank) exists in the code or config; attention uses standard per-head q/k/v projections. |
| MiMo-V2.6-Pro | not in its code | not stated | The attention module projects q/k/v directly per head with no low-rank latent down/up projection (no kv_lora_rank/q_lora_rank in config or code). |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | The attention module uses standard per-head q/k/v projections with no low-rank KV latent or decoupled RoPE key mechanism. modeling_nemotron_h.py#L851 |
| Qwen3.5-397B-A17B | not in its code | not stated | The attention module projects K/V directly with num_key_value_heads and there is no low-rank latent kv projection or kv_lora_rank in the config or code. |
| Qwen3.6-35B-A3B | not in its code | not stated | No kv_lora_rank/q_lora_rank low-rank KV latent projection exists; attention uses standard separate k_proj/v_proj heads. modeling_qwen3_5_moe.py#L766 |
| Qwen3.8-Flash-Next | not in its code | not stated | No kv_lora_rank/q_lora_rank down-projection into a shared latent exists in the code or config; attention uses standard separate k_proj/v_proj. |
| Step-3.7-Flash | not in its code | not stated | The attention module uses standard separate q/k/v projections sized by head_dim with no low-rank latent KV projection or lora rank config keys. |
| gpt-oss-120b | not in its code | not stated | Attention uses standard separate q/k/v projections with no low-rank latent KV cache mechanism. |
Evidence
3 spans quoted from the sources, strongest treatment first.
MLAPO (Multi-head Latent Attention Pre-processing Optimization)
coremodel architecturein GLM-5Z.ai
By employing reduced key-value vectors, Multi-latent attention (MLA) matches the effectiveness of Grouped-Query Attention (GQA) but offers superior GPU memory savings and faster processing for long-context sequences.
usedmodel architecturein GLM-5Z.ai
GQA as the substrate, not MLA.
not usedmodel architecturein MiniMax-M3MiniMax
Filed alongside
Other methods under model architecture :: token mixer :: softmax attention :: multi-head latent attention.