QK-Norm applies an RMSNorm to each attention head's query and key before their dot product. With the usual 1/sqrt(d_h) scale, it caps every attention logit at sqrt(d_h) times the largest query and key gains, so logits can no longer grow with the size of the query and key weights.
Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention
Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product.
Qwen3VLTextAttention applies RMSNorm to queries and keys per head before rotary embedding and the attention dot product.
self.q_norm = Qwen3VLTextRMSNorm(self.head_dim, eps=config.rms_norm_eps) # unlike olmo, only on the head dim!
self.k_norm = Qwen3VLTextRMSNorm(
self.head_dim, eps=config.rms_norm_eps
) # thus post q_norm does not need reshape
if (
self.q_norm is not None
and self.k_norm is not None
and self.qk_norm_type == "qwen3"
):
query_states = self.q_norm(query_states)
key_states = self.k_norm(key_states)
Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product.
Queries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are normalized with kv_norm as well.
Queries and keys in full_attention layers are normalized per head with RMSNorm (q_norm/k_norm) before RoPE and the dot product.
self.q_norm = Qwen3_5MoeRMSNorm(self.head_dim, eps=config.rms_norm_eps) # unlike olmo, only on the head dim!
self.k_norm = Qwen3_5MoeRMSNorm(
self.head_dim, eps=config.rms_norm_eps
)
Queries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm), which is part of the separate sparse indexer, not the main QK path.
Engram's own query/key projections use an RMS-normalized dot product internally, but the files given do not show the main attention module's query/key projections (DeepseekV4Attention) to confirm or deny QK normalization in the softmax attention path.
The DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not included in the provided sources, so this cannot be confirmed.
Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch.
There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path.
There is no RMSNorm/LayerNorm applied to the main attention q_states/key_states before the dot product (only q_a_layernorm/kv_a_layernorm normalize the low-rank latents, and the indexer k_norm normalizes the indexer's key, not the main attention heads).
KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention).
No RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's separate k has a LayerNorm, which is not the main QK dot product).
No RMSNorm/LayerNorm is applied to query or key states before the attention dot product in MiMoV2Attention; queries/keys go straight from projection/RoPE into matmul.
The language-model attention paths (Florence2Attention family) have no q/k normalization; the only q_norm/k_norm code exists in transformer.py's Attention class guarded by qk_norm flag, which is never set true or exposed via XVLAConfig/SoftPromptedTransformer, so it defaults off for this model.
self.q_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()
self.k_norm = norm_layer(self.head_dim) if qk_norm else nn.Identity()