methods · softmax attention
multi-head latent attention
5 methods filed at this node or below it, from the sources of 5 models.
In practice
Checked as “Multi-head latent attention” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | adopted | kv_lora_rank=512 · qk_rope_head_dim=64Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2 style. text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64 | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | q_lora_rank=1280 · qk_rope_head_dim=64 · compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…The text config defines kv_lora_rank-style low-rank KV compression (kv_source_layer_ids/compress_ratios), q_lora_rank and a decoupled RoPE head dim, and the code builds fused wq_a/wkv projections and a compressed KV cache consistent with DeepSeek MLA. text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1… | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | q_lora_rank=1280 · qk_rope_head_dim=64 · head_dim=512The attention mechanism is DeepSeek-style MLA with q_lora_rank/kv-related ranks and a decoupled RoPE key dimension configured in text_config. text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64text_config.head_dim = 512 | code → |
| Hy4-preview | in its code | not stated | kv_lora_rank=512 · q_lora_rank=2048 · use_mla=trueKeys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA, and use_mla is true. kv_lora_rank = 512q_lora_rank = 2048use_mla = true | code → |
| GLM-5.3-Flash | in its code | not stated | kv_lora_rank=512 · q_lora_rank=1536Glm5NextTextAttention down-projects KV into a kv_lora_rank latent cached and expands via kv_b_proj per head, DeepSeek-V2/V3 style MLA. text_config.kv_lora_rank = 512text_config.q_lora_rank = 1536 | code → |
| GLM-5.3 | in its code | not stated | kv_lora_rank=512 · q_lora_rank=2048 · qk_rope_head_dim=64Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA. kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | q_lora_rank=1536 · qk_rope_head_dim=64Queries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head, DeepSeek-style, with a decoupled rope key applied via partial rotary q_lora_rank = 1536qk_rope_head_dim = 64 | code → |
| Kimi K3 | in its code | not stated | kv_lora_rank=512 · q_lora_rank=1536 · qk_rope_head_dim=64KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style. text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64text_config.q_lora_rank = 1536 | code → |
| GLM-5.2 | in its code | adopted | kv_lora_rank=512 · q_lora_rank=2048 · qk_rope_head_dim=64Attention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA. kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64 | code → |
| DeepSeek-V3.2 | in its code | not stated | kv_lora_rank=512 · q_lora_rank=1536 · qk_rope_head_dim=64Attention compresses KV into a shared kv_lora_rank latent with a decoupled RoPE key, then up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA. kv_lora_rank = 512q_lora_rank = 1536qk_rope_head_dim = 64 | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo low-rank KV latent projection exists; attention uses standard per-head q/k/v projections sized by num_key_value_heads. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesAttention uses standard per-head q/k/v projections with GQA, with no latent kv_lora_rank/q_lora_rank down-projection mechanism in the code or config. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo kv_lora_rank/q_lora_rank keys appear in the text_config and the nemotron_h attention code is not included, so MLA cannot be confirmed or ruled out. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not a latent low-rank KV cache. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesAttention uses standard separate q/k/v projections with no low-rank latent cache or kv_lora_rank, so MLA is not implemented. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesThere is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections with GQA instead. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesThe attention module projects q/k/v directly per head with no low-rank latent down/up projection (no kv_lora_rank/q_lora_rank in config or code). | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesNo kv_lora_rank/q_lora_rank or latent KV down/up-projection is present; attention uses standard per-head k_proj/v_proj instead. | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | q_lora_rank=1024There is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with q_lora_rank only compressing queries is used, so this is not DeepSeek-V2-style MLA. q_lora_rank = 1024 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesThere is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses standard separate q/k/v projections. | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesThe attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a shared low-rank latent that is up-projected per head DeepSeek-V2 style; there is no kv_lora_rank, only a direct kv_proj to head_dim. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo low-rank KV latent projection (kv_lora_rank/q_lora_rank) exists anywhere in the attention code or config; attention uses standard per-head q/k/v projections. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesThere is no kv_lora_rank/q_lora_rank low-rank latent projection; attention uses standard per-head K/V projections sized by num_key_value_heads. | code → |
| Inkling | not in its code | not stated | verdict and linesNo kv_lora_rank/q_lora_rank-style latent down/up projection exists; attention here is a GQA relative-position design with separate per-head K/V projections, not MLA. | code → |
| Hy3 | not in its code | not stated | verdict and linesAttention uses standard separate q/k/v projections sized by head_dim with no low-rank latent kv projection anywhere in the code. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesThe attention module is a standard multi-head/GQA attention with separate q/k/v projections, no latent KV down-projection or decoupled RoPE key as in DeepSeek-V2 MLA. | code → |
| MiniMax-M3 | not in its code | not used | verdict and linesAttention uses standard per-head q/k/v projections with GQA, with no latent low-rank KV projection or lora rank config present. | code → |
| Step-3.7-Flash | not in its code | not stated | verdict and linesAttention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the code or config. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesThere is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV projection instead. | code → |
| DeepSeek-V4-Flash | code unclear | not stated | q_lora_rank=1024 · num_key_value_heads=1Queries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a decoupled RoPE key in the DeepSeek-V2 sense), so this resembles MLA's query side but the KV side is single-head MQA rather than a low-rank latent up-projected per head. q_lora_rank = 1024num_key_value_heads = 1 | code → |
| DeepSeek-V4-Pro | not in its code | not stated | q_lora_rank=1536There is no kv_lora_rank; keys/values are not down-projected into a shared low-rank latent and up-projected per head, instead a single shared KV head (head_dim=512) is produced directly by kv_proj, so this is not DeepSeek-V2-style MLA. q_lora_rank = 1536 | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesThere is no low-rank KV latent projection (no kv_lora_rank/q_lora_rank in config or code); attention uses standard per-head q/k/v projections. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo latent KV down/up projection mechanism (kv_lora_rank/q_lora_rank) exists in the code or config; attention uses standard per-head q/k/v projections. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesNo kv_lora_rank/q_lora_rank low-rank KV latent mechanism exists; attention uses standard separate k_proj/v_proj with GQA instead. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo kv_lora_rank/q_lora_rank/latent KV cache mechanism exists; attention uses standard per-head K/V projections sized by num_key_value_heads. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesNo kv_lora_rank/q_lora_rank-based latent projection exists; attention uses standard per-head q/k/v projections with GQA instead. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo low-rank KV latent projection exists anywhere in the attention implementations; keys/values are projected directly to full embed_dim as in standard attention. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo low-rank KV latent projection exists; attention uses standard separate Q/K/V projections sized by head counts | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | model_type=llamaNo kv_lora_rank, q_lora_rank, or latent attention mechanism appears anywhere in the config or code; the language model is a stock Llama-2 backbone. text_config.model_type = "llama" | code → |
In this branch 5
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 5
2 from one source, or only mentioned as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| DeepSeek-V3.2 core | MQA mode of MLA coreMHA and MQA modes of MLA used— |
| GLM-5.2 core | Multi-head Latent Attention core— |
| Kimi K3 core | Gated Multi-head Latent Attention core— |
| MiniMax-M3 not used | Multi-head Latent Attention not used— |
| Mistral-Large-4 core | Multi-head Latent Attention coreRagged Multi-head Latent Attention not used— |