methods · model architecture
normalization & residual
36 methods filed at this node or below it, from the sources of 16 models.
In practice
Checked as “QK normalization” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| LFM3-1.7B | in its code | not stated | verdict and linesQueries and keys are each passed through an RMSNorm over the head dimension before RoPE and the attention dot product. | code → |
| MiniMax-M3.1-Flash | in its code | not stated | use_qk_norm=true · qk_norm_type=per_headQueries and keys are each normalized per head with RMSNorm before RoPE and the dot product, and use_qk_norm is true in this config. text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head" | code → |
| Step-5-Preview | in its code | not stated | verdict and linesQueries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product. | code → |
| Kolibri-1 | in its code | adopted | verdict and linesQueries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention. | code → |
| Qwen3.8-Flash-Next | in its code | not stated | verdict and linesQueries and keys are each passed through an RMSNorm (q_norm/k_norm) per head before RoPE and the attention dot product. | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | verdict and linesQueries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention computation. | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | verdict and linesQueries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | verdict and linesQueries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product. | code → |
| Xiaomi-Robotics-1 | in its code | not stated | verdict and linesQwen3VLTextAttention applies RMSNorm to queries and keys per head before rotary embedding and the attention dot product. | code → |
| Laguna-S-2.1 | in its code | not stated | verdict and linesQueries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and the attention dot product. | code → |
| Inkling | in its code | not stated | verdict and linesQueries and keys are each passed through per-head RMSNorm (q_norm/k_norm) before the attention dot product. | code → |
| Hy3 | in its code | not stated | qk_norm=trueQueries and keys are each passed through a per-head RMSNorm before RoPE and the attention dot product, and qk_norm is enabled in config. qk_norm = true | code → |
| MiniMax-M3 | in its code | not stated | use_qk_norm=true · qk_norm_type=per_headuse_qk_norm is true and the attention module applies per-head RMSNorm to queries and keys before RoPE/dot-product. text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head" | code → |
| Step-3.7-Flash | in its code | not stated | verdict and linesQueries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot product. | code → |
| MolmoAct 2 | in its code | not stated | use_qk_norm=true · qk_norm_type=qwen3use_qk_norm is true and qk_norm_type is 'qwen3', causing MolmoAct2RMSNorm to be applied per-head to queries and keys before the dot product. text_config.use_qk_norm = truetext_config.qk_norm_type = "qwen3" | code → |
| DeepSeek-V4-Flash | in its code | not stated | verdict and linesQueries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product. | code → |
| DeepSeek-V4-Pro | in its code | not stated | verdict and linesQueries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are normalized with kv_norm as well. | code → |
| Qwen3.6-35B-A3B | in its code | not stated | verdict and linesQueries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention. | code → |
| Gemma 4 31B | in its code | not stated | verdict and linesText attention normalizes queries and keys per head with Gemma4RMSNorm (q_norm/k_norm) before applying RoPE and computing the dot product. | code → |
| Qwen3.5-397B-A17B | in its code | not stated | verdict and linesQueries and keys in full_attention layers are normalized per head with RMSNorm (q_norm/k_norm) before RoPE and the dot product. | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesQueries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm), which is part of the separate sparse indexer, not the main QK path. | code → |
| DeepSeek-V4.1-Flash-1017 | code unclear | not stated | verdict and linesEngram's own query/key projections use an RMS-normalized dot product internally, but the files given do not show the main attention module's query/key projections (DeepseekV4Attention) to confirm or deny QK normalization in the softmax attention path. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo use_qk_norm/qk_layernorm keys appear in the config and the attention implementation file is not provided, so QK normalization cannot be confirmed. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesQuery and key tensors are used directly for RoPE and attention without any RMSNorm/LayerNorm normalization step in MiMoV2Attention. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesQuery and key states are used directly after projection and RoPE with no RMSNorm/LayerNorm applied to them before the dot product. | code → |
| DeepSeek-V4.1-Flash | code unclear | not stated | verdict and linesThe DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not included in the provided sources, so this cannot be confirmed. | code → |
| Hy4-preview | not in its code | not stated | verdict and linesMain attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch. | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesThere is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesThere is no RMSNorm/LayerNorm applied to the main attention q_states/key_states before the dot product (only q_a_layernorm/kv_a_layernorm normalize the low-rank latents, and the indexer k_norm normalizes the indexer's key, not the main attention heads). | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesNemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesKimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention). | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesNo RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's separate k has a LayerNorm, which is not the main QK dot product). | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesNemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesNo RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from projection/RoPE into matmul. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesOnly the indexer's key gets a LayerNorm (k_norm); the main MLA query and key states (q_states, key_states) are not normalized before the dot product. | code → |
| X-VLA | not in its code | not stated | verdict and linesThe language-model attention paths (Florence2Attention family) have no q/k normalization; the only q_norm/k_norm code exists in transformer.py's Attention class guarded by qk_norm flag, which is never set true or exposed via XVLAConfig/SoftPromptedTransformer, so it defaults off for this model. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesQuery and key states are projected and rotary-embedded but never normalized before the attention dot product | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo use_qk_norm, qk_layernorm config key or per-head query/key normalization code is present in the given files. | code → |
In this branch 36
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 36
23 from one source, or only mentioned as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.