Open Models Almanac
Methodsmodel architecturenormalization & residual

methods · model architecture

normalization & residual

36 methods filed at this node or below it, from the sources of 16 models.

In practice

Checked as “QK normalization” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm over the head dimension before RoPE and the attention dot product.

query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_lfm3.py · L255–256
MiniMax-M3.1-Flashin its codenot stated
use_qk_norm=true · qk_norm_type=per_head

Queries and keys are each normalized per head with RMSNorm before RoPE and the dot product, and use_qk_norm is true in this config.

text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head"
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_minimax_m3_vl.py · L431–432
Step-5-Previewin its codenot stated
verdict and lines

Queries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product.

self.q_norm = make_step5_rms_norm(
            self.head_dim,
            eps=rms_norm_eps,
            zero_centered=self.zero_centered,
            dtype=norm_dtype,
        )
model.py · L1127–1132
Kolibri-1in its codeadopted
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

        self.q_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_kolibri1.py · L158–159
        query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_kolibri1.py · L177–178
Qwen3.8-Flash-Nextin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) per head before RoPE and the attention dot product.

query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen4_exp.py · L859–860
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention computation.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
DeepSeek-V4-Pro-0813in its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)

        kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L828–832
DeepSeek-V4-Flash-0731in its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Xiaomi-Robotics-1in its codenot stated
verdict and lines

Qwen3VLTextAttention applies RMSNorm to queries and keys per head before rotary embedding and the attention dot product.

self.q_norm = Qwen3VLTextRMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
        self.k_norm = Qwen3VLTextRMSNorm(
            self.head_dim, eps=config.rms_norm_eps
        )  # thus post q_norm does not need reshape
modeling_mibot.py · L423–426
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_mibot.py · L440–441
Laguna-S-2.1in its codenot stated
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and the attention dot product.

self.q_norm = LagunaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = LagunaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_laguna.py · L352–353
query_states = self.q_norm(query_states).transpose(1, 2)
        key_states = self.k_norm(key_states).transpose(1, 2)
modeling_laguna.py · L373–374
Inklingin its codenot stated
verdict and lines

Queries and keys are each passed through per-head RMSNorm (q_norm/k_norm) before the attention dot product.

self.q_norm = InklingRMSNorm(head_dim, eps=config.rms_norm_eps)
        self.k_norm = InklingRMSNorm(head_dim, eps=config.rms_norm_eps)
attention.py · L134–135
Hy3in its codenot stated
qk_norm=true

Queries and keys are each passed through a per-head RMSNorm before RoPE and the attention dot product, and qk_norm is enabled in config.

qk_norm = true
query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_hy_v3.py · L241–242
MiniMax-M3in its codenot stated
use_qk_norm=true · qk_norm_type=per_head

use_qk_norm is true and the attention module applies per-head RMSNorm to queries and keys before RoPE/dot-product.

text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head"
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_minimax_m3_vl.py · L431–432
Step-3.7-Flashin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot product.

query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_step3p7.py · L713–714
MolmoAct 2in its codenot stated
use_qk_norm=true · qk_norm_type=qwen3

use_qk_norm is true and qk_norm_type is 'qwen3', causing MolmoAct2RMSNorm to be applied per-head to queries and keys before the dot product.

text_config.use_qk_norm = truetext_config.qk_norm_type = "qwen3"
if config.use_qk_norm:
            k_norm_size = (
                config.head_dim
                if config.qk_norm_type == "qwen3"
                else config.num_key_value_heads * config.head_dim
            )
            self.k_norm = MolmoAct2RMSNorm(k_norm_size, eps=config.layer_norm_eps)
modeling_molmoact2.py · L2224–2230
if (
            self.q_norm is not None
            and self.k_norm is not None
            and self.qk_norm_type == "qwen3"
        ):
            query_states = self.q_norm(query_states)
            key_states = self.k_norm(key_states)
modeling_molmoact2.py · L2274–2280
DeepSeek-V4-Flashin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
DeepSeek-V4-Proin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are normalized with kv_norm as well.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Qwen3.6-35B-A3Bin its codenot stated
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen3_5_moe.py · L776–777
Gemma 4 31Bin its codenot stated
verdict and lines

Text attention normalizes queries and keys per head with Gemma4RMSNorm (q_norm/k_norm) before applying RoPE and computing the dot product.

self.q_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
modeling_gemma4.py · L1177
self.k_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
            self.v_norm = Gemma4RMSNorm(self.head_dim, eps=config.rms_norm_eps, with_scale=False)
modeling_gemma4.py · L1181–1182
Qwen3.5-397B-A17Bin its codenot stated
verdict and lines

Queries and keys in full_attention layers are normalized per head with RMSNorm (q_norm/k_norm) before RoPE and the dot product.

self.q_norm = Qwen3_5MoeRMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
        self.k_norm = Qwen3_5MoeRMSNorm(
            self.head_dim, eps=config.rms_norm_eps
        )
modeling_qwen3_5_moe.py · L755–758
query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen3_5_moe.py · L776–777
Mistral-Large-4not in its codenot stated
verdict and lines

Queries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm), which is part of the separate sparse indexer, not the main QK path.

DeepSeek-V4.1-Flash-1017code unclearnot stated
verdict and lines

Engram's own query/key projections use an RMS-normalized dot product internally, but the files given do not show the main attention module's query/key projections (DeepseekV4Attention) to confirm or deny QK normalization in the softmax attention path.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No use_qk_norm/qk_layernorm keys appear in the config and the attention implementation file is not provided, so QK normalization cannot be confirmed.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

Query and key tensors are used directly for RoPE and attention without any RMSNorm/LayerNorm normalization step in MiMoV2Attention.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

Query and key states are used directly after projection and RoPE with no RMSNorm/LayerNorm applied to them before the dot product.

DeepSeek-V4.1-Flashcode unclearnot stated
verdict and lines

The DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not included in the provided sources, so this cannot be confirmed.

Hy4-previewnot in its codenot stated
verdict and lines

Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch.

GLM-5.3-Flashnot in its codenot stated
verdict and lines

There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path.

GLM-5.3not in its codenot stated
verdict and lines

There is no RMSNorm/LayerNorm applied to the main attention q_states/key_states before the dot product (only q_a_layernorm/kv_a_layernorm normalize the low-rank latents, and the indexer k_norm normalizes the indexer's key, not the main attention heads).

q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
modeling_glm_moe_dsa.py · L382–383
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
Kimi K3not in its codenot stated
verdict and lines

KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention).

GLM-5.2not in its codenot stated
verdict and lines

No RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's separate k has a LayerNorm, which is not the main QK dot product).

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
MiMo-V2.5not in its codenot stated
verdict and lines

No RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from projection/RoPE into matmul.

DeepSeek-V3.2not in its codenot stated
verdict and lines

Only the indexer's key gets a LayerNorm (k_norm); the main MLA query and key states (q_states, key_states) are not normalized before the dot product.

query_states = torch.cat((q_pass, q_rot), dim=-1)

        key_states, value_states = self.expand_kv(k_pass, k_rot)
modeling_deepseek_v32.py · L430–432
X-VLAnot in its codenot stated
verdict and lines

The language-model attention paths (Florence2Attention family) have no q/k normalization; the only q_norm/k_norm code exists in transformer.py's Attention class guarded by qk_norm flag, which is never set true or exposed via XVLAConfig/SoftPromptedTransformer, so it defaults off for this model.

self.q_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
        self.k_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
transformer.py · L117–118
gpt-oss-120bnot in its codenot stated
verdict and lines

Query and key states are projected and rotary-embedded but never normalized before the attention dot product

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_gpt_oss.py · L298–300
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No use_qk_norm, qk_layernorm config key or per-head query/key normalization code is present in the given files.

In this branch 36

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 36

Manifold-Constrained Hyper-Connections core · 9 sources · 9 quotes
Attention Residuals core · 7 sources · 7 quotes
Gated Residual core · 6 sources · 8 quotes
Adaptive Layer Normalization (AdaLN) core · 3 sources · 3 quotes
Identity Hyper-Connections core · 3 sources · 3 quotes
Head-wise gated attention core · 2 sources · 2 quotes
Hyper-Connections core · 2 sources · 2 quotes
Per-branch scalar residual write gate core · 2 sources · 2 quotes
QK Normalization core · 2 sources · 3 quotes
Sandwich normalization core · 2 sources · 2 quotes
Single-Pass mHC core · 2 sources · 2 quotes
RMSNorm used · 2 sources · 2 quotes
QK-Clip evaluated · 3 sources · 3 quotes
23 from one source, or only mentioned as not used
Block Attention Residuals core · 1 source · 2 quotes
Four-branch residual stream core · 1 source · 1 quote
GatedNorm core · 1 source · 3 quotes
Learned per-layer depth gate core · 1 source · 2 quotes
Two-phase Block AttnRes schedule core · 1 source · 1 quote
Widened residual stream core · 1 source · 1 quote
Adaptive RMSNorm used · 1 source · 1 quote
Bounded positive gates used · 1 source · 1 quote
Depth-conditioning gate initialization used · 1 source · 1 quote
Learned depth gate on expert conditioning used · 1 source · 1 quote
Post-embedding RMSNorm used · 1 source · 1 quote
Pre-LN used · 1 source · 1 quote
Zero-centered RMSNorm used · 1 source · 1 quote
Simplified AltUp variant evaluated · 1 source · 1 quote
Full Attention Residuals mentioned · 1 source · 1 quote
Sparse gated residual writes not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash coreManifold-Constrained Hyper-Connections core—
DeepSeek-V4.1-Flash coreSingle-Pass mHC coreManifold-Constrained Hyper-Connections used—
Hy4-preview coreIdentity Hyper-Connections core—
DeepSeek-V4-Flash coreManifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—
DeepSeek-V4-Flash-Vision-Exp coreHyper-Connections core—
DeepSeek-V4-Pro coreManifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—
GR00T N1.7 coreAdaptive Layer Normalization (AdaLN) core—
Inkling usedPost-embedding RMSNorm used—
Kimi K3 coreAttention Residuals coreBlock Attention Residuals coreTwo-phase Block AttnRes schedule coreFull Attention Residuals mentioned—
Kolibri-1 coreQK Normalization coreQK-Clip evaluated—
LFM3-1.7B coreHead-wise gated attention coreSandwich normalization core—
MolmoAct 2 coreLearned per-layer depth gate coreDepth-conditioning gate initialization usedLearned depth gate on expert conditioning used—
Qwen3.8-Flash-Next coreAll-branch, elementwise read gating in Gated Residual coreDynamic gating of residual reads and writes coreElement-wise data-dependent residual read gate coreFour-branch residual stream coreFour-stream Hyper-Connection Combine update coreGated Residual coreGatedNorm coreIndependent per-branch normalization in Gated Residual coreLow-rank gated Mix for Hyper-Connection residual streams corePer-branch scalar residual write gate coreWidened residual stream coreBounded positive gates usedData-dependent residual read and write operators usedZero-centered RMSNorm usedHyper-Connections evaluatedManifold-Constrained Hyper-Connections evaluatedSimplified AltUp variant evaluatedQK-Clip not usedSparse gated residual writes not used—
Xiaomi-Robotics-1 usedAdaptive Layer Normalization (AdaLN) used—
gpt-oss-120b usedPre-LN usedRMSNorm used—
π0.5 usedAdaptive RMSNorm used—
↑