Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

QK Normalization

QK-Norm applies an RMSNorm to each attention head's query and key before their dot product. With the usual 1/sqrt(d_h) scale, it caps every attention logit at sqrt(d_h) times the largest query and key gains, so logits can no longer grow with the size of the query and key weights.

Written for the explainer Normalization and logit control.

Also called QK Normalisation, per-head RMS normalization.

sources
2
model
1
lab adopt it
1
strongest
core

In practice

Checked as “QK normalization” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm over the head dimension before RoPE and the attention dot product.

query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_lfm3.py · L255–256
MiniMax-M3.1-Flashin its codenot stated
use_qk_norm=true · qk_norm_type=per_head

Queries and keys are each normalized per head with RMSNorm before RoPE and the dot product, and use_qk_norm is true in this config.

text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head"
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_minimax_m3_vl.py · L431–432
Step-5-Previewin its codenot stated
verdict and lines

Queries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product.

self.q_norm = make_step5_rms_norm(
            self.head_dim,
            eps=rms_norm_eps,
            zero_centered=self.zero_centered,
            dtype=norm_dtype,
        )
model.py · L1127–1132
Kolibri-1in its codeadopted
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

        self.q_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_kolibri1.py · L158–159
        query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_kolibri1.py · L177–178
Qwen3.8-Flash-Nextin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) per head before RoPE and the attention dot product.

query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen4_exp.py · L859–860
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention computation.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
DeepSeek-V4-Pro-0813in its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)

        kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L828–832
DeepSeek-V4-Flash-0731in its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Xiaomi-Robotics-1in its codenot stated
verdict and lines

Qwen3VLTextAttention applies RMSNorm to queries and keys per head before rotary embedding and the attention dot product.

self.q_norm = Qwen3VLTextRMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
        self.k_norm = Qwen3VLTextRMSNorm(
            self.head_dim, eps=config.rms_norm_eps
        )  # thus post q_norm does not need reshape
modeling_mibot.py · L423–426
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_mibot.py · L440–441
Laguna-S-2.1in its codenot stated
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and the attention dot product.

self.q_norm = LagunaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = LagunaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_laguna.py · L352–353
query_states = self.q_norm(query_states).transpose(1, 2)
        key_states = self.k_norm(key_states).transpose(1, 2)
modeling_laguna.py · L373–374
Inklingin its codenot stated
verdict and lines

Queries and keys are each passed through per-head RMSNorm (q_norm/k_norm) before the attention dot product.

self.q_norm = InklingRMSNorm(head_dim, eps=config.rms_norm_eps)
        self.k_norm = InklingRMSNorm(head_dim, eps=config.rms_norm_eps)
attention.py · L134–135
Hy3in its codenot stated
qk_norm=true

Queries and keys are each passed through a per-head RMSNorm before RoPE and the attention dot product, and qk_norm is enabled in config.

qk_norm = true
query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_hy_v3.py · L241–242
MiniMax-M3in its codenot stated
use_qk_norm=true · qk_norm_type=per_head

use_qk_norm is true and the attention module applies per-head RMSNorm to queries and keys before RoPE/dot-product.

text_config.use_qk_norm = truetext_config.qk_norm_type = "per_head"
query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_minimax_m3_vl.py · L431–432
Step-3.7-Flashin its codenot stated
verdict and lines

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot product.

query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_step3p7.py · L713–714
MolmoAct 2in its codenot stated
use_qk_norm=true · qk_norm_type=qwen3

use_qk_norm is true and qk_norm_type is 'qwen3', causing MolmoAct2RMSNorm to be applied per-head to queries and keys before the dot product.

text_config.use_qk_norm = truetext_config.qk_norm_type = "qwen3"
if config.use_qk_norm:
            k_norm_size = (
                config.head_dim
                if config.qk_norm_type == "qwen3"
                else config.num_key_value_heads * config.head_dim
            )
            self.k_norm = MolmoAct2RMSNorm(k_norm_size, eps=config.layer_norm_eps)
modeling_molmoact2.py · L2224–2230
if (
            self.q_norm is not None
            and self.k_norm is not None
            and self.qk_norm_type == "qwen3"
        ):
            query_states = self.q_norm(query_states)
            key_states = self.k_norm(key_states)
modeling_molmoact2.py · L2274–2280
DeepSeek-V4-Flashin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
DeepSeek-V4-Proin its codenot stated
verdict and lines

Queries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are normalized with kv_norm as well.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Qwen3.6-35B-A3Bin its codenot stated
verdict and lines

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen3_5_moe.py · L776–777
Gemma 4 31Bin its codenot stated
verdict and lines

Text attention normalizes queries and keys per head with Gemma4RMSNorm (q_norm/k_norm) before applying RoPE and computing the dot product.

self.q_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
modeling_gemma4.py · L1177
self.k_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
            self.v_norm = Gemma4RMSNorm(self.head_dim, eps=config.rms_norm_eps, with_scale=False)
modeling_gemma4.py · L1181–1182
Qwen3.5-397B-A17Bin its codenot stated
verdict and lines

Queries and keys in full_attention layers are normalized per head with RMSNorm (q_norm/k_norm) before RoPE and the dot product.

self.q_norm = Qwen3_5MoeRMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
        self.k_norm = Qwen3_5MoeRMSNorm(
            self.head_dim, eps=config.rms_norm_eps
        )
modeling_qwen3_5_moe.py · L755–758
query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen3_5_moe.py · L776–777
Mistral-Large-4not in its codenot stated
verdict and lines

Queries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm), which is part of the separate sparse indexer, not the main QK path.

DeepSeek-V4.1-Flash-1017code unclearnot stated
verdict and lines

Engram's own query/key projections use an RMS-normalized dot product internally, but the files given do not show the main attention module's query/key projections (DeepseekV4Attention) to confirm or deny QK normalization in the softmax attention path.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No use_qk_norm/qk_layernorm keys appear in the config and the attention implementation file is not provided, so QK normalization cannot be confirmed.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

Query and key tensors are used directly for RoPE and attention without any RMSNorm/LayerNorm normalization step in MiMoV2Attention.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

Query and key states are used directly after projection and RoPE with no RMSNorm/LayerNorm applied to them before the dot product.

DeepSeek-V4.1-Flashcode unclearnot stated
verdict and lines

The DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not included in the provided sources, so this cannot be confirmed.

Hy4-previewnot in its codenot stated
verdict and lines

Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch.

GLM-5.3-Flashnot in its codenot stated
verdict and lines

There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path.

GLM-5.3not in its codenot stated
verdict and lines

There is no RMSNorm/LayerNorm applied to the main attention q_states/key_states before the dot product (only q_a_layernorm/kv_a_layernorm normalize the low-rank latents, and the indexer k_norm normalizes the indexer's key, not the main attention heads).

q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
modeling_glm_moe_dsa.py · L382–383
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
Kimi K3not in its codenot stated
verdict and lines

KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention).

GLM-5.2not in its codenot stated
verdict and lines

No RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's separate k has a LayerNorm, which is not the main QK dot product).

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
MiMo-V2.5not in its codenot stated
verdict and lines

No RMSNorm or LayerNorm is applied to query/key states before the dot product in MiMoV2Attention; only rotary embedding is applied.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from projection/RoPE into matmul.

DeepSeek-V3.2not in its codenot stated
verdict and lines

Only the indexer's key gets a LayerNorm (k_norm); the main MLA query and key states (q_states, key_states) are not normalized before the dot product.

query_states = torch.cat((q_pass, q_rot), dim=-1)

        key_states, value_states = self.expand_kv(k_pass, k_rot)
modeling_deepseek_v32.py · L430–432
X-VLAnot in its codenot stated
verdict and lines

The language-model attention paths (Florence2Attention family) have no q/k normalization; the only q_norm/k_norm code exists in transformer.py's Attention class guarded by qk_norm flag, which is never set true or exposed via XVLAConfig/SoftPromptedTransformer, so it defaults off for this model.

self.q_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
        self.k_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
transformer.py · L117–118
gpt-oss-120bnot in its codenot stated
verdict and lines

Query and key states are projected and rotary-embedded but never normalized before the attention dot product

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_gpt_oss.py · L298–300
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No use_qk_norm, qk_layernorm config key or per-head query/key normalization code is present in the given files.

Documented in

Aleph Alpha

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 2

Kolibri-1

Aleph Alphaadopted2 sources

Queries and keys are normalised per head with RMS normalisation (Henry et al. 2020; B. Zhang and Sennrich 2019) before RoPE

coremodel architecturein KolibriAleph Alpha
2 more from this model

Every block normalises queries and keys per head with RMS normalisation before RoPE

coremodel architecturein KolibriAleph Alpha

We used per-head RMS normalization of the query and key representations.

usedmodel architecturein KolibriAleph Alpha
↑