Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionmulti-head latent attention

specific method · filed under model architecture

Multi-head Latent Attention

Multi-head Latent Attention is an attention method that uses reduced key-value vectors instead of full key-value representations.1 It matches the effectiveness of Grouped-Query Attention while saving more GPU memory and processing long-context sequences faster.2

Trade-off It saves GPU memory and speeds up long-context processing, but has high computational cost during decoding.3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. By employing reduced key-value vectors, Multi-latent attention (MLA — GLM-5: from Vibe Coding to Agentic Engineering
  2. matches the effectiveness of Grouped-Query Attention (GQA) but offers superior GPU memory savings and faster processing for long-context sequences — GLM-5: from Vibe Coding to Agentic Engineering
  3. offers superior GPU memory savings and faster processing for long-context sequences — GLM-5: from Vibe Coding to Agentic Engineering
  4. Another disadvantage of MLA is its high computational cost during decoding — GLM-5: from Vibe Coding to Agentic Engineering

Also called MLA, Multi-head Latent Attention (MLA), Multi-latent Attention, Multi-latent attention (MLA).

sources
3
models
3
labs adopt it
2
strongest
core

In practice

Checked as “Multi-head latent attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codeadopted
kv_lora_rank=512 · qk_rope_head_dim=64

Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2 style.

text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64
self.kv_a_proj_with_mqa = (
            nn.Linear(self.hidden_size, config.kv_lora_rank + self.qk_rope_head_dim, bias=config.attention_bias)
            if not self.is_kv_shared_layer
            else None
        )
modeling_mistral_large4.py · L350–354
self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_mistral_large4.py · L358–362
DeepSeek-V4.1-Flash-1017in its codenot stated
q_lora_rank=1280 · qk_rope_head_dim=64 · compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…

The text config defines kv_lora_rank-style low-rank KV compression (kv_source_layer_ids/compress_ratios), q_lora_rank and a decoupled RoPE head dim, and the code builds fused wq_a/wkv projections and a compressed KV cache consistent with DeepSeek MLA.

text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…
("attn.fused_wqa_wkv", "attn.wq_a", 0),
            ("attn.fused_wqa_wkv", "attn.wkv", 1),
model.py · L1371–1372
DeepSeek-V4.1-Flashin its codenot stated
q_lora_rank=1280 · qk_rope_head_dim=64 · head_dim=512

The attention mechanism is DeepSeek-style MLA with q_lora_rank/kv-related ranks and a decoupled RoPE key dimension configured in text_config.

text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64text_config.head_dim = 512
    def _select_dsv4_attn_cls(vllm_config: VllmConfig) -> type[DeepseekV4Attention]:
model.py · L203
Hy4-previewin its codenot stated
kv_lora_rank=512 · q_lora_rank=2048 · use_mla=true

Keys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA, and use_mla is true.

kv_lora_rank = 512q_lora_rank = 2048use_mla = true
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = HYV4RMSNorm(config.kv_lora_rank)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_hy_v4.py · L349–359
GLM-5.3-Flashin its codenot stated
kv_lora_rank=512 · q_lora_rank=1536

Glm5NextTextAttention down-projects KV into a kv_lora_rank latent cached and expands via kv_b_proj per head, DeepSeek-V2/V3 style MLA.

text_config.kv_lora_rank = 512text_config.q_lora_rank = 1536
        self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = Glm5NextTextRMSNorm(self.kv_lora_rank, eps=config.rms_norm_eps)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_glm5_next.py · L1131–1141
GLM-5.3in its codenot stated
kv_lora_rank=512 · q_lora_rank=2048 · qk_rope_head_dim=64

Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA.

kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = GlmMoeDsaRMSNorm(config.kv_lora_rank)
modeling_glm_moe_dsa.py · L327–332
kv_nope = self.kv_b_proj(kv_nope).view(key_shape).transpose(1, 2)
        k_nope, value_states = torch.split(kv_nope, [self.qk_nope_head_dim, self.v_head_dim], dim=-1)
modeling_glm_moe_dsa.py · L359–360
DeepSeek-V4-Pro-0813in its codenot stated
q_lora_rank=1536 · qk_rope_head_dim=64

Queries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head, DeepSeek-style, with a decoupled rope key applied via partial rotary

q_lora_rank = 1536qk_rope_head_dim = 64
self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
        self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L797–802
Kimi K3in its codenot stated
kv_lora_rank=512 · q_lora_rank=1536 · qk_rope_head_dim=64

KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style.

text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64text_config.q_lora_rank = 1536
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            self.kv_lora_rank + self.qk_rope_head_dim,
            bias=False,
        )
        self.kv_a_layernorm = KimiRMSNorm(self.kv_lora_rank)
modeling_kimi_linear.py · L378–383
compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
        k_pass, k_rot = torch.split(
            compressed_kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L426–428
GLM-5.2in its codeadopted
kv_lora_rank=512 · q_lora_rank=2048 · qk_rope_head_dim=64

Attention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA.

kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
modeling_glm_moe_dsa.py · L327–331
kv_nope = self.kv_b_proj(kv_nope).view(key_shape).transpose(1, 2)
        k_nope, value_states = torch.split(kv_nope, [self.qk_nope_head_dim, self.v_head_dim], dim=-1)
modeling_glm_moe_dsa.py · L359–360
DeepSeek-V3.2in its codenot stated
kv_lora_rank=512 · q_lora_rank=1536 · qk_rope_head_dim=64

Attention compresses KV into a shared kv_lora_rank latent with a decoupled RoPE key, then up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA.

kv_lora_rank = 512q_lora_rank = 1536qk_rope_head_dim = 64
compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
        kv_pass, k_rot = torch.split(compressed_kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
        # Both latents are viewed as single-head, 4D tensors, as expected by `expand_kv`
        k_pass = self.kv_a_layernorm(kv_pass).view(batch_size, 1, seq_length, self.kv_lora_rank)
modeling_deepseek_v32.py · L418–421
LFM3-1.7Bnot in its codenot stated
verdict and lines

No low-rank KV latent projection exists; attention uses standard per-head q/k/v projections sized by num_key_value_heads.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

Attention uses standard per-head q/k/v projections with GQA, with no latent kv_lora_rank/q_lora_rank down-projection mechanism in the code or config.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No kv_lora_rank/q_lora_rank keys appear in the text_config and the nemotron_h attention code is not included, so MLA cannot be confirmed or ruled out.

Step-5-Previewnot in its codenot stated
verdict and lines

No kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not a latent low-rank KV cache.

Kolibri-1not in its codenot stated
verdict and lines

Attention uses standard separate q/k/v projections with no low-rank latent cache or kv_lora_rank, so MLA is not implemented.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections with GQA instead.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

The attention module projects q/k/v directly per head with no low-rank latent down/up projection (no kv_lora_rank/q_lora_rank in config or code).

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

No kv_lora_rank/q_lora_rank or latent KV down/up-projection is present; attention uses standard per-head k_proj/v_proj instead.

DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
q_lora_rank=1024

There is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with q_lora_rank only compressing queries is used, so this is not DeepSeek-V2-style MLA.

q_lora_rank = 1024
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L801–802
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

There is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses standard separate q/k/v projections.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L881–883
DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

The attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a shared low-rank latent that is up-projected per head DeepSeek-V2 style; there is no kv_lora_rank, only a direct kv_proj to head_dim.

self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L801–802
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No low-rank KV latent projection (kv_lora_rank/q_lora_rank) exists anywhere in the attention code or config; attention uses standard per-head q/k/v projections.

Laguna-S-2.1not in its codenot stated
verdict and lines

There is no kv_lora_rank/q_lora_rank low-rank latent projection; attention uses standard per-head K/V projections sized by num_key_value_heads.

Inklingnot in its codenot stated
verdict and lines

No kv_lora_rank/q_lora_rank-style latent down/up projection exists; attention here is a GQA relative-position design with separate per-head K/V projections, not MLA.

Hy3not in its codenot stated
verdict and lines

Attention uses standard separate q/k/v projections sized by head_dim with no low-rank latent kv projection anywhere in the code.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

The attention module is a standard multi-head/GQA attention with separate q/k/v projections, no latent KV down-projection or decoupled RoPE key as in DeepSeek-V2 MLA.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L881–883
MiniMax-M3not in its codenot used
verdict and lines

Attention uses standard per-head q/k/v projections with GQA, with no latent low-rank KV projection or lora rank config present.

Step-3.7-Flashnot in its codenot stated
verdict and lines

Attention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the code or config.

MolmoAct 2not in its codenot stated
verdict and lines

There is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV projection instead.

DeepSeek-V4-Flashcode unclearnot stated
q_lora_rank=1024 · num_key_value_heads=1

Queries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a decoupled RoPE key in the DeepSeek-V2 sense), so this resembles MLA's query side but the KV side is single-head MQA rather than a low-rank latent up-projected per head.

q_lora_rank = 1024num_key_value_heads = 1
self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
        self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
modeling_deepseek_v4.py · L797–801
DeepSeek-V4-Pronot in its codenot stated
q_lora_rank=1536

There is no kv_lora_rank; keys/values are not down-projected into a shared low-rank latent and up-projected per head, instead a single shared KV head (head_dim=512) is produced directly by kv_proj, so this is not DeepSeek-V2-style MLA.

q_lora_rank = 1536
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L801–802
MiMo-V2.5not in its codenot stated
verdict and lines

There is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No latent KV down/up projection mechanism (kv_lora_rank/q_lora_rank) exists in the code or config; attention uses standard per-head q/k/v projections.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

No kv_lora_rank/q_lora_rank low-rank KV latent mechanism exists; attention uses standard separate k_proj/v_proj with GQA instead.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No kv_lora_rank/q_lora_rank/latent KV cache mechanism exists; attention uses standard per-head K/V projections sized by num_key_value_heads.

Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

No kv_lora_rank/q_lora_rank-based latent projection exists; attention uses standard per-head q/k/v projections with GQA instead.

X-VLAnot in its codenot stated
verdict and lines

No low-rank KV latent projection exists anywhere in the attention implementations; keys/values are projected directly to full embed_dim as in standard attention.

gpt-oss-120bnot in its codenot stated
verdict and lines

No low-rank KV latent projection exists; attention uses standard separate Q/K/V projections sized by head counts

OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

No kv_lora_rank, q_lora_rank, or latent attention mechanism appears anywhere in the config or code; the language model is a stock Llama-2 backbone.

text_config.model_type = "llama"

Documented in

Mistral AI
Z.ai (Zhipu AI)
MiniMax

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 2not used 1

GLM-5.2

Z.ai (Zhipu AI)adopted1 source

By employing reduced key-value vectors, Multi-latent attention (MLA) matches the effectiveness of Grouped-Query Attention (GQA) but offers superior GPU memory savings and faster processing for long-context sequences.

usedmodel architecturein GLM-5Z.ai
1 more from this model

MLAPO (Multi-head Latent Attention Pre-processing Optimization)

coremodel architecturein GLM-5Z.ai
↑