Open Models Almanac

Residual & normalization · R1

Normalization and logit control

Most norms and clips in a modern decoder cost almost nothing to run; what each costs is a constraint on the architecture, and what each bounds is narrower than its name suggests

Illustration of a tall stack of shallow trays on a plain table, each held level by two small clips, with fine sand heaped evenly in every tray except one near the top, where a cone of sand rises above the rim.

In 2023 a Google team training an 8-billion-parameter vision transformer at Adam learning rate 10−310^{-3} watched the loss fall normally and then, within 2,000 steps, climb. Dehghani et al. traced it to the attention logits, the scores a head computes before its softmax. Without a fix they “quickly grow to over 50000 in magnitude”, the softmax turns one-hot, and the loss diverges. Two years later Moonshot trained a 53B-parameter mixture-of-experts model with Muon, an optimizer that replaces each weight update by the nearest matrix whose singular values are all equal, and saw the largest logits “quickly exceed a magnitude of 1000”.

The fixes are small. Normalizing each head’s queries and keys adds 512 parameters to a decoder layer of Qwen3.6-35B-A3B that holds 836 million, counting all its experts (computed below). Clipping the weights after an update adds none. Every language model in this site’s map normalizes its residual stream with RMSNorm, and in the three models we costed all the norms of a layer together hold under 0.01% of its parameters.

These bounds demand something of the architecture instead. Query-key normalization needs each head’s key as a vector, and multi-head latent attention, which caches one compressed vector per token, never forms that vector when it generates text. A norm also bounds only its own output; the learned per-channel scales it multiplies by, its gains, can still grow. This article takes the norms and clips in today’s open-weight models one at a time: what each one bounds, what it needs, and who uses it.

The block, and where the norms sit

We use the series notation: model width dd, hh query heads, gg key-value heads, head width dhd_h, LL layers. The worked example throughout is a full-attention layer of Qwen3.6-35B-A3B: d=2,048d = 2{,}048, 16 query heads and 2 key-value heads of width 256 (num_attention_heads: 16, num_key_value_heads: 2, head_dim: 256), so each key-value head serves 8 query heads (grouped-query attention, T1). Its feed-forward part is a mixture of experts: a token is routed to a few of 256 experts, so of the layer’s 836 million parameters it uses 56 million.

A decoder block reads a residual stream xlx_l (one vector of width dd per token), runs an attention sublayer and a feed-forward sublayer, and adds each sublayer’s output back to the stream. Where the norms go relative to those additions is the first design choice, and we come to it after the norm itself. The primer 00 has the block in full.

RMSNorm

A norm rescales each token’s vector by a number computed from that vector alone. LayerNorm (Ba et al., 2016) subtracts the mean of the dd entries, divides by their standard deviation, then multiplies by a learned gain γ∈Rd\gamma \in \mathbb{R}^d and adds a learned bias. Zhang and Sennrich (2019) dropped the mean and the bias:

RMSNorm(x)=γ⊙xRMS(x),RMS(x)=1d∑ixi2+ϵ.\text{RMSNorm}(x) = \gamma \odot \frac{x}{\text{RMS}(x)}, \qquad \text{RMS}(x) = \sqrt{\tfrac{1}{d}\textstyle\sum_i x_i^2 + \epsilon} .

Their hypothesis was that LayerNorm works because it is invariant to rescaling its input, and that its invariance to adding a constant to every entry, which RMSNorm gives up, is “dispensable”. On a 2019 Transformer-base translation model RMSNorm gave BLEU “comparable to LayerNorm” and cut step time by 7%, and OLMo 2 found “no difference” between RMSNorm and a LayerNorm without parameters.

Ignore ϵ\epsilon for a moment. The vector u=x/RMS(x)u = x/\text{RMS}(x) then has ∥u∥2=∑ixi2/(1d∑ixi2)=d\|u\|^2 = \sum_i x_i^2 / (\frac{1}{d}\sum_i x_i^2) = d, whatever xx was. RMSNorm puts every token on a sphere of radius d\sqrt d (45.3 for d=2,048d = 2{,}048), and the gain stretches each axis of that sphere. (ϵ\epsilon only matters for vectors so small that RMS(x)2\text{RMS}(x)^2 is near ϵ\epsilon.) This has two consequences:

  • The output of a norm is bounded by its gains: ∥RMSNorm(x)∥≤max⁡i∣γi∣d\|\text{RMSNorm}(x)\| \le \max_i |\gamma_i| \sqrt d, for any input. The next linear layer’s output is bounded only by that times the layer’s largest singular value, so it can still grow with that weight matrix.
  • The budget of dd is shared. If one channel is far larger than the rest, it takes nearly all of it. With d=2,048d = 2{,}048, one channel at 1,000 and the rest at 1, RMS(x)≈22.1\text{RMS}(x) \approx 22.1, so the small channels come out at 1/22.1=0.0451/22.1 = 0.045 (computed).
  • fifteen channels at 1
  • the outlier channel
  • outlier before its gain
Before the norm: x (linear axis, 0 to 100)0501001001After the norm: γ x / RMS(x) (linear axis, 0 to √16 = 4)012343.9970.040channel 1 … 16Before the norm: x (linear axis, 0 to 100)0501001001After the norm: γ x / RMS(x) (linear axis, 0 to √16 = 4)012343.9970.040channel 1 … 16

RMS(x) = 25.02. Length of x / RMS(x) = 4.00 = √16, at every outlier size. After the norm the fifteen small channels are 0.040 each, the outlier 3.997. At d = 2,048 with the same outlier, the small channels come out at 0.412.

Figure 1.RMSNorm divides by the root mean square, so the normalized vector always has length sqrt(d). Move the outlier up and the other channels shrink toward zero after the norm, even though their input did not change. With a small gain on the outlier channel, the next layer barely sees the outlier, and the other channels are still divided by the large RMS.

The site’s code checks, which test each model’s modeling code against a checklist, cover 39 models with usable code, four of them robot models. Every residual-stream norm in all of them is an RMSNorm without bias, except in X-VLA, a robot model whose language part is a BART-style encoder-decoder with LayerNorm after each addition. LayerNorm survives elsewhere only in side modules such as vision towers.

A norm is cheap in parameters and arithmetic. In one full-attention layer of Qwen3.6-35B-A3B the four norms (two on the residual stream, one on queries, one on keys) have 4,608 parameters, against 56 million that a token uses. They do about 43,500 operations per token against 112 million for the layer’s matrix products (2×2 \times 56.1M), a share of 3.9×10−43.9 \times 10^{-4} (computed). DeepSeek-V4-Flash and Gemma 4 31B land in the same range.

Their memory traffic matters more. When a model generates one token per step (decode), the layer’s weights are read once for the whole batch, while each token’s norms read and write its own activations, about 44 kB per token here (BF16). At batch 1 that is 3.9×10−43.9 \times 10^{-4} of the layer’s weight bytes; at batch 64 in the dense Gemma 4 31B it is about 1% (computed, cache reads left out). Serving engines therefore fuse norms into the residual addition or the next kernel, so that a norm costs no extra kernel launch or memory round trip.

Placement: before, after, or both

Post-norm and pre-norm

The original Transformer normalized after each addition, xl+1=Norm(xl+F(xl))x_{l+1} = \text{Norm}(x_l + F(x_l)), now called post-norm. It keeps the stream at a fixed scale, and it is hard to train deep. Xiong et al. (2020) showed that at initialization the gradient of the last layer’s weights in a post-norm stack doesn’t shrink with depth, while in pre-norm it scales as 1/L1/\sqrt{L}. Large gradients near the output make the first full-size updates too big for the top layers, so post-norm depends on a warm-up, a learning rate raised from near zero over the first few thousand steps. Without one, Xiong et al.’s post-norm translation model reached 8.45 BLEU against about 34 with it, and Nguyen and Salazar (2019) found post-norm “often fails without warmup even with 5 or 6 layers”.

Pre-norm moves the norm inside the branch:

xl+1=xl+F(Norm(xl)),x_{l+1} = x_l + F\big(\text{Norm}(x_l)\big),

with one more norm before the output head. Every sublayer now reads a vector of known scale, and the identity path from the bottom to the top is never rescaled. It is the default: 31 of the 35 language models in the map (the 39 minus the robot models) are pre-norm. In 9 of them, DeepSeek’s V4 and V4.1 models, GLM-5.3-Flash and Hy4-preview, the stream is widened into several parallel streams mixed by learned matrices (hyper-connections), and Kimi K3 reads earlier layers’ outputs through attention residuals; both are the subject of R2, and the depth analysis below is for the plain single stream.

Pre-norm’s stability has two costs. When post-norm does train, it can be better: Nguyen and Salazar measured 27.58 BLEU for post-norm and 26.83 for pre-norm on WMT’14 English-German. The second cost is the stream itself. Nothing normalizes the sum, so each sublayer’s output adds to it. Xiong et al. prove that at initialization the expected squared norm of layer ll‘s stream lies between (1+l/2)d(1 + l/2)d and (1+3l/2)d(1 + 3l/2)d, so its RMS grows like l\sqrt{l}. A sublayer output of fixed size then changes a deep layer’s normalized input less than a shallow layer’s.

In trained models the picture is less regular. Kim et al. (2025) found the hidden-state variance of pre-norm models of 400M to 3.2B parameters “escalates exponentially to extremely large values as optimization proceeds”, passing the FP16 maximum after 0.5B tokens at 3.2B, and the layers ending “markedly more redundant”. Sun et al. (2024) found something narrower in Llama-2: a handful of “massive activations”, above 15,000 in LLaMA2-70B, that appear abruptly in one early layer on the first token and a few delimiters, then stay constant. NVIDIA reports residual norms 3 orders of magnitude apart across depth in Nemotron 3 Super and 4 in Ultra. Step-5-Preview keeps its stream in FP32 ("fp32_residual_connection": true).

A norm on the way out as well

Gemma 2 normalized “the input and output of each transformer sub-layer”, and its reports call this “post-norm and pre-norm”. The second norm acts on the sublayer’s output before the addition, which is different from the original post-norm:

xl+1=xl+Normout(F(Normin(xl))).x_{l+1} = x_l + \text{Norm}_\text{out}\Big(F\big(\text{Norm}_\text{in}(x_l)\big)\Big).

Kim et al. call this peri-norm, and others call it sandwich norm. Each addition now has length at most d\sqrt d times the largest output gain, so the output gain sets how fast the stream grows. Kolibri-1 initializes its output-norm gains at 1/L1/\sqrt{L} (0.141 for 50 blocks) and its input-norm gains at 1. If its embeddings have squared norm dd and each of the 100 sublayer outputs has squared norm d/50d/50 and is uncorrelated with the stream, they add 2d2d, and the stream’s RMS grows by 3=1.73\sqrt 3 = 1.73 over the stack (computed, a toy). The report gives no reason; shrinking residual branches at initialization is a common choice in deep networks.

Three of the 35 language models do this, and the code confirms each: Gemma 4 31B (post_attention_layernorm and post_feedforward_layernorm, applied to the sublayer outputs), Kolibri-1 (post_self_attn_layernorm and post_mlp_layernorm; its report: the sublayers “read from the residual stream through an RMSNorm and write back through a second one”), and LFM3-1.7B ("use_sandwich_norm": true). LFM3’s predecessor LFM2 was pre-norm. None of the three reports ablates the placement. Kim et al.’s ablation at 400M to 3.2B found peri-norm better than pre-norm on every size they tested, with fewer loss surges, and found that output norms with gains frozen at 1 still beat pre-norm alone, with learnable gains slightly better.

Output gain s:
  • post-norm
  • pre-norm, proven range
  • norm on each sublayer's output (toy)
residual RMS ÷ embedding RMS (linear)024681001020304050layer l10.05 toy8.725.101.00pre-norm (proven range)All three at initialization; the dashed line is a toy.residual RMS ÷ embedding RMS (linear)024681001020304050layer lperi-norm toy 10.05pre-norm, up to 8.72pre-norm, at least 5.10post-norm 1.00pre-norm (proven range)All three at initialization; the dashed line is a toy.

At layer 50: post-norm 1.00, pre-norm between 5.10 and 8.72, peri-norm toy (s = 1) 10.05, in units of the embedding's RMS.

Figure 2.At initialization, a pre-norm stack's stream grows like the square root of depth (the band is Xiong et al.'s proven range for the expected squared norm); post-norm stays flat. A norm on each sublayer's output, as in Gemma 4, Kolibri-1 and LFM3, adds a fixed amount per sublayer set by its gain: at gain 1 the toy grows faster than Xiong's pre-norm (whose sublayers start with small outputs), at Kolibri's 1/sqrt(L) it ends only 1.7 times above where it began.

A fourth arrangement normalizes only the output: xl+1=xl+Norm(F(xl))x_{l+1} = x_l + \text{Norm}(F(x_l)), with no input norm. OLMo 2 and the 34B Chameleon model use it, both after Swin Transformer v2. OLMo 2 found it helped only together with QK-Norm: “In isolation, neither of these changes yield good results”. Together they cut a gradient spike score (the share of steps whose gradient norm is seven standard deviations off its rolling average) from 0.108 to 0.069. No model in the map ships it.

Zero-centred gains

If a norm moves growth into its gains, decaying the gains is one way to hold it back. Plain weight decay (θ←θ−ηλθ\theta \leftarrow \theta - \eta\lambda\theta) pulls a gain toward 0, though; since the next matrix can absorb any scale the gain loses, decay can move scale into that matrix instead. Many labs exempt norm weights from decay, Kolibri among them. DeepSeek-V4.1 states that its “normalization-layer weights are also subject to weight decay”.

The zero-centred form stores the gain as an offset from 1:

RMSNorm(x)=(1+w)⊙xRMS(x),w initialized to 0.\text{RMSNorm}(x) = (1 + w) \odot \frac{x}{\text{RMS}(x)}, \qquad w \text{ initialized to } 0 .

In exact arithmetic the function is the same; only the parameter changes. Decay now pulls the gain to 1: w←w−ηλww \leftarrow w - \eta\lambda w is the same step as γ←γ−ηλ(γ−1)\gamma \leftarrow \gamma - \eta\lambda(\gamma - 1). That reading is ours: Qwen’s model card for Qwen3-Next lists “zero-centered and weight-decayed layernorm” among its stability measures, and the Qwen3.8-Next report adopts zero-centred RMSNorm “to constrain the growth of RMSNorm weights”, with no formula or ablation. The form predates Qwen: the Transformers code for Google’s first Gemma already computes output * (1.0 + self.weight.float()).

In the map it is a lab choice, with 7 models from 3 labs: Qwen3.5, 3.6 and 3.8 (torch.zeros(dim), then 1.0 + self.weight), MiniMax-M3 and M3.1-Flash ("use_gemma_norm": true), and Step-3.7-Flash and Step-5-Preview. Gemma 4 itself doesn’t use it: Gemma4RMSNorm initializes its weight to ones and multiplies. The Qwen models apply it to every norm except the gated output norms inside their linear-attention layers.

Two differences survive outside exact arithmetic. A gain near 1 stored in BF16 has a resolution of 2−7≈0.0082^{-7} \approx 0.008, while an offset near 0 keeps far finer steps (computed from the format). And a zero-centred checkpoint loaded into a plain RMSNorm kernel runs with gains near 0 and gives garbage without an error, which Step-5-Preview’s serving code guards against ("Step5 requires zero-centered RMSNorm weights").

Attention logits that grow

Back to the failure from the opening. In one head, the logit for query ii and key jj is

zij=qi⊤kjdh,qi=Wq Norm(xi),kj=Wk Norm(xj).z_{ij} = \frac{q_i^\top k_j}{\sqrt{d_h}}, \qquad q_i = W_q\, \text{Norm}(x_i), \quad k_j = W_k\, \text{Norm}(x_j).

The pre-norm bounds Norm(x)\text{Norm}(x), so ∣zij∣|z_{ij}| is bounded by a constant times ∥Wq⊤Wk∥2≤∥Wq∥2∥Wk∥2\|W_q^{\top} W_k\|_2 \le \|W_q\|_2 \|W_k\|_2, the product of the two largest singular values of the head’s projections. Weights that grow make logits that grow quadratically, and softmax turns large logit gaps into one-hot weights; Zhai et al. called that attention entropy collapse. Wortsman et al. (2023) reproduced the failure in small models with AdamW: every run whose largest logit passed 10410^4 diverged, and the logit growth came from the norms of qq and kk, “not due to an increase in their alignment”. They hypothesize the cause is “the quadratic dependence of attention logits on parameter norms”, and that is why both fixes below act on norms.

Moonshot’s Kimi K2 report says the explosion “occurs more frequently with Muon but less with AdamW in our experiments”, and offers a hypothesis: Muon’s updates have all singular values equal, so they are more likely to push up the top singular values of WqW_q and WkW_k. The optimizer side is in O1. For the fixes below, the growth is a property of the two weight matrices, and Wortsman’s runs show it without Muon.

The first fix was to cap the logit itself. Gemma 2 applied ctanh⁡(z/c)c \tanh(z/c) to attention logits with c=50c = 50 and to the output logits with c=30c = 30. The K2 report rejects this for attention because “the dot products between queries and keys can still grow excessively before capping is applied”: the softmax sees capped logits, but the weights keep growing. Gemma 3 replaced it “with QK-norm”, and in the map no language model soft-caps attention logits. Gemma 4 and Mistral Large 4 soft-cap the output logits at 30 ("final_logit_softcapping": 30.0). The output softmax has its own fix, z-loss, a penalty on log⁡2\log^2 of the softmax’s normalizer; Wortsman et al. and Chameleon use it, and no report in the map does.

QK-Norm

QK-Norm applies an RMSNorm to each head’s query and key before the dot product:

q^=γq⊙qRMS(q),k^=γk⊙kRMS(k),z=q^⊤k^dh.\hat q = \gamma_q \odot \frac{q}{\text{RMS}(q)}, \qquad \hat k = \gamma_k \odot \frac{k}{\text{RMS}(k)}, \qquad z = \frac{\hat q^\top \hat k}{\sqrt{d_h}} .

The normalized axis is the head’s dhd_h channels, and one gain vector of width dhd_h is shared by all heads of a layer. In the worked example that is 256 numbers for queries and 256 for keys: 512 parameters in a layer of 836 million (computed). OLMo 2 normalizes the whole query projection across all heads at once instead; every model in the map normalizes each head, which also keeps the norm local when heads are split across GPUs (tensor parallelism). The norm runs before the rotary position embedding (RoPE), which rotates pairs of channels and keeps lengths, so the bound below survives it.

The bound follows from the sphere above. Each normalized vector has length at most max⁡i∣γi∣dh\max_i|\gamma_i|\sqrt{d_h}, so by Cauchy-Schwarz

∣z∣≤(max⁡i∣γq,i∣dh)(max⁡j∣γk,j∣dh)dh=dh max⁡i∣γq,i∣ max⁡j∣γk,j∣.|z| \le \frac{(\max_i |\gamma_{q,i}|\sqrt{d_h})(\max_j |\gamma_{k,j}|\sqrt{d_h})}{\sqrt{d_h}} = \sqrt{d_h}\, \max_i |\gamma_{q,i}| \, \max_j |\gamma_{k,j}| .

With unit gains and dh=256d_h = 256 that is 16, far below the thousands seen in failing runs. To allow a logit of 1,000 the gains of both queries and keys would have to reach about 7.9 (computed).

The precise reason this targets Wortsman’s diagnosis is an invariance. Scaling a head’s WqW_q by any a>0a > 0 or its WkW_k by any b>0b > 0 leaves q^\hat q and k^\hat k unchanged (with ϵ=0\epsilon = 0), so the logit depends only on the directions of the projections, not their size. Their growth no longer reaches the forward pass; it only shrinks each update relative to the weights, as with weight normalization. Logits can now grow only through 2dh2 d_h gain parameters per layer. With equal gains aa on queries and bb on keys, z=dh abcos⁡∠(q,k)z = \sqrt{d_h}\,ab\cos\angle(q, k): QK-Norm is cosine-similarity attention with a learned temperature, and per-channel gains generalize the temperature to one per channel.

The bound doesn’t make attention diffuse. Two logits of one query can differ by up to 32, a weight ratio of e32≈7.9×1013e^{32} \approx 7.9 \times 10^{13}, so one key among 262,144 can still take all but 3×10−93 \times 10^{-9} of the attention (computed). QK-Norm caps the scale of the logits at a level set by dhd_h and the gains; sharp attention stays possible. It also bounds nothing outside the dot product: not the values, not the head’s output, and not the output logits over the vocabulary, which Chameleon notes it “does not solve”.

  • QK-Norm cap
  • clip threshold; Gemma 4 tick
  • reported in runs that failed (different setups)
QK-Norm cap, dh = 128, unit gains11.3QK-Norm cap, dh = 256, unit gains16.0Gemma 4 (scale 1): 256QK-Clip threshold, Kimi K2100vanilla Muon, 53B MoE (Kimi K2 report)more than 1,000all runs above this diverged (Wortsmanet al.)10,0008B ViT without QK-Norm (Dehghani et al.)more than 50,000110102103104105largest attention logit (log scale)QK-Norm cap, dh = 128, unitgains11.3QK-Norm cap, dh = 256, unitgains16.0Gemma 4 (scale 1): 256QK-Clip threshold, Kimi K2100vanilla Muon, 53B MoE (Kimi K2report)more than 1,000all runs above this diverged(Wortsman et al.)10,0008B ViT without QK-Norm(Dehghani et al.)more than 50,000110102103104105largest attention logit (log scale)
Keys T:Head width:

At dh = 128 and gain product 1.00 the cap is 11.3, and one key can still take 0.99996 of the attention among T = 262,144 keys.

Figure 3.With unit gains, QK-Norm limits a logit to 11 or 16 for heads of width 128 or 256. Failing runs reached thousands. The cap still lets one key take almost all of a head's attention, so the norm limits scale, not sharpness. The QK-Norm bars assume the usual 1/sqrt(d_h) scale; Gemma 4, with scale 1, can reach 256. The bars for failures come from different models and setups and are not one experiment.

Henry et al. (2020) introduced the idea for translation in exactly this cosine form, against softmax saturation rather than divergence. Dehghani et al. used a LayerNorm version to stop the 8B ViT’s divergence, Wortsman et al. trained a 1.2B model with it at learning rate 0.3, and Chameleon’s 7B model “diverges after approximately 20% of a training epoch” without it.

The one ablation in the map’s documents is Kolibri-1’s, on a 0.6B-active, 7.8B-total proxy, single runs shown over its first 53B training tokens. Without QK-Norm the proxy “diverges early in training and does not recover”. Removing it only from the full-attention blocks, which have no position encoding, it trains but “its gradient norm spikes by several orders of magnitude in the second half of the run”. An adapted QK-Clip with threshold 100 in its place “also shows gradient-norm spikes”. At a tenth of the learning rate the run without QK-Norm trains stably and ends at a higher loss.

In the code checks, 20 of the 39 models have QK-Norm and 3 are unclear. Without the four robot models it is 18 of 35, from 10 labs: Aleph Alpha, DeepSeek (V4), Google, Liquid, MiniMax, Poolside, Qwen, StepFun, Tencent (Hy3) and Thinking Machines. Every one normalizes per head, and before RoPE where the layer has RoPE (Inkling has none, nor do Kolibri’s full-attention layers). The 14 models without it split cleanly: 7 use multi-head latent attention (DeepSeek-V3.2, GLM-5.2, GLM-5.3, GLM-5.3-Flash, Hy4-preview, Mistral Large 4, Kimi K3), and 7 have no norm on queries or keys at all (Nemotron 3 Ultra and 3.5 Lightning, gpt-oss-120b, four MiMo models). No model with latent attention has it. The next section is about why.

The fixed factor in front of the cosine varies. Gemma 4 drops the 1/dh1/\sqrt{d_h} (self.scaling = 1.0), so its query-key term runs up to dhd_h times the gain product, 256 or 512 at unit gains, and the gains learn the scale. Inkling divides by dhd_h instead (“q/k are per-head RMS-normed (unit norm), so Inkling scales by 1/head_dim”), which bounds that term by the gain product, and then adds a learned relative-position term and a length-dependent scale. DeepSeek-V4 normalizes its queries with no gain at all and puts the only gain on the key.

The bound also holds only for what sits between the norm and the dot product. Context-extension methods that rescale the rotated channels or the logits after the norm raise it: Laguna-S-2.1 multiplies RoPE’s cosines and sines by YaRN’s "attention_factor": 1.485 on its global layers, in both qq and kk, which raises the rotated part of the logit by up to 1.4852≈2.21.485^2 \approx 2.2; Inkling scales global-layer logits by 1+0.1log⁡(n/128,000)1 + 0.1\log(n/128{,}000) beyond 128K tokens.

QK-Norm requires each head’s qq and kk to exist as vectors before the dot product. In grouped-query attention they always do. It also adds a kernel step, which serving engines fuse with RoPE and the cache write (Step-5-Preview’s fused_qknorm_rope_cache). It is part of the forward pass, so a fine-tune keeps the norm whatever its optimizer; only the 2dh2 d_h gains can move the bound.

Latent attention and QK-Clip

Multi-head latent attention (MLA, T1) caches one latent vector cc of width dcd_c per token instead of per-head keys and values. A head’s key is WukhcW^h_{uk} c, an up-projection of the latent. Forming every head’s key for every cached token at every decode step is the cost MLA exists to avoid, so at decode time the up-projection is absorbed into the query, q⊤(Wukhc)=(Wukh⊤q)⊤cq^\top (W^h_{uk} c) = (W^{h\top}_{uk} q)^\top c, and the per-head key is never formed. The identity holds because WukhW^h_{uk} is linear. A key norm is not: RMS(Wukhc)\text{RMS}(W^h_{uk} c) differs per token and per head, so it can’t be folded into the query. In training and in prefill, which usually run MLA in the plain multi-head form (GLM-5’s report says so), the keys exist; the obstacle is decode. The K2 report keeps that scope: QK-Norm “is not applicable to multi-head latent attention (MLA), because its Key matrices are not fully materialized during inference.”

It is a cost, not an impossibility (our reading): the divisor is one number per token and head, which could be computed when the token enters the cache and stored beside the latent, 22% more cached numbers for DeepSeek-V3.2’s 128 heads beside its 576-wide latent (11% at 64 heads). No model in the map does this.

MLA models do normalize: they apply an RMSNorm to the query latent and to the KV latent (q_a_layernorm, kv_a_layernorm). That bounds the cached latent itself, which helps when it is stored at low precision, but not the logit: the norm acts before the up-projections WuqW_{uq} and WukW_{uk}, so qq and kk still scale with them. MLA also carries a separate rotary part, since a position-dependent rotation can’t be absorbed either: each head has its own rotary query, and all heads share one rotary key.

Moonshot’s answer, QK-Clip, bounds the logit through the weights. During the forward pass, record for each head hh the largest logit in the batch, Smax⁡hS^h_{\max}. After the optimizer step, if Smax⁡hS^h_{\max} exceeds a threshold τ\tau, set γh=τ/Smax⁡h\gamma_h = \tau / S^h_{\max} (a per-head factor, not a norm gain) and shrink that head’s weights. In plain multi-head attention, scaling WqhW^h_q and WkhW^h_k by γh\sqrt{\gamma_h} each scales every logit of that head by γh\gamma_h. For the weights that produced the batch this would bring the maximum to exactly τ\tau; the clip is applied to the already-updated weights, so the result is τ\tau times the ratio of the maximum after and before the update. In MLA the logit has two parts, z=qC⊤kC+qR⊤kRz = q^{C\top} k^C + q^{R\top} k^R (content and rotary, up to the constant scale). K2 scales the head’s content query and key up-projections by γh\sqrt{\gamma_h} and its rotary query by γh\gamma_h, and leaves the shared rotary key alone, since changing it would change every head. Both parts scale by γh\gamma_h, so the logit does.

Figure 4.For a head whose largest logit passed the threshold, QK-Clip shrinks the weights that belong to that head alone: the content query and key projections by the square root of gamma, the rotary query by gamma. Both parts of the logit then scale by gamma. The rotary key is shared by every head, so it is not touched.

A separate mid-scale model (53B total, 9B active) trained with plain Muon had logits above 1,000. K2 itself (1.04T total) trained with the clip at τ=100\tau = 100, and its largest logits decayed from the cap “to a stable range after approximately 30% of the training steps”: in the first 70,000 steps 12.7% of heads triggered the clip, and after that every head had fallen below the threshold. K2 trained on 15.5T tokens “without a single loss spike”. On a 0.5B-active test pair, clipping at τ=30\tau = 30 had “negligible effects on the loss curve”. Kimi K3, the Moonshot model in the map, trains with “the weight-clipping mechanism introduced in Kimi K2”, with no threshold given; its MLA layers have no rotary part.

In classical terms QK-Clip is a projection after an unconstrained update, like max-norm regularization. No gradient passes through it: the loss keeps pushing a head toward sharper attention and the clip pulls it back, so an active head sits near τ\tau. A Muon or AdamW update doesn’t shrink with the weights, so a clipped head’s next update is larger relative to them (our reading).

QK-Clip needs the per-head maximum logit, which a fused attention kernel computes inside its softmax but doesn’t keep: it must be written out and reduced across data-parallel GPUs. It is part of the training recipe, not the model, so a fine-tune of Kimi K3 has no bound unless the fine-tuner adds the clip. And the evidence is one lab’s: Kolibri’s adapted version for grouped-query attention still spiked where QK-Norm did not.

Two other ways out

QK-Clip isn’t the only answer for latent attention. Z.ai’s GLM-5 report (filed under GLM-5.2) found that with Muon, MLA at first fell short of grouped-query attention, and changed the optimizer. Plain Muon orthogonalizes the stacked up-projection of all heads at once; “Muon Split” orthogonalizes each head’s slice, which “enables projection weights for different attention heads to update at different scales”. With it, “the scale of attention logits of GLM-5 remains stable during pre-training without any clipping strategy”, an observation rather than an ablation. Kimi K3, DeepSeek-V4.1 and Qwen3.8 also split their optimizers per head. Like the clip, this is part of the recipe.

DeepSeek went the other way and changed the attention. DeepSeek-V4-Flash and V4-Pro use a shared key-value multi-query attention: one key-value head of width 512 per token, computed directly from the hidden state and read as both key and value (T3 covers the compressed entries pooled from it). Because the keys are formed explicitly, they can be normalized. The report: “The attention architecture of DeepSeek-V4 series allows us to directly apply RMSNorm on the attention queries and KV entries, which effectively prevents attention logits from exploding. Consequently, we do not employ the QK-Clip technique (Liu et al., 2025) in our Muon optimizer.” In the code each query head goes through an RMSNorm without a gain, and the key-value entry through one with a gain, both before RoPE.

The KV entry is also the value, so DeepSeek-V4 normalizes its values too, and because the entry carries RoPE, it un-rotates the attention output. Gemma 4’s global layers also reuse the key projection as the value ("attention_k_eq_v": true) but rotate only the key, and normalize values in every layer, without a gain. The DeepSeek report gives no ablation and says only that the norm “may improve training stability”.

A bound on the key has a second use. A low-precision number format stores a scale so that values fit its range, and a norm already bounds every element by its gain times dh\sqrt{d_h}. DeepSeek-V4.1-Flash stores its main KV cache in FP4 (its sliding-window cache stays FP8): its largest trained gain there is about 1, so each element of the 512-channel entry is at most 512≈22.6\sqrt{512} \approx 22.6 (the largest observed was “around 10”), and it drops NVFP4’s second-level scale because “omitting the global scale causes no measurable decrease in accuracy”. Kolibri argues the same for its FP8 KV cache during RL, “assuming sufficiently small gain parameters”. In grouped-query attention the values usually aren’t normalized, so only the keys get this bound; Gemma 4, which normalizes its values, is the exception. Number formats are in I1.

GatedNorm

The last method sits on the residual norm itself. Qiu et al. (2026), a Qwen paper, argue that the massive activations above have a job. With a pre-norm, a token whose stream holds one huge channel has every other channel shrunk by its RMSNorm (the figure at the top), and the model learns to use this as a way to scale a token down. The norm weight on such a channel ends up tiny, “0.006 vs. 1” in their example, so the next layer doesn’t see the outlier itself. Their fix is to supply the rescaling directly:

GatedNorm(u)=RMSNorm(u)⊙σ(W2 SiLU(W1 RMSNorm(u))),\text{GatedNorm}(u) = \text{RMSNorm}(u) \odot \sigma\Big(W_2\, \text{SiLU}\big(W_1\, \text{RMSNorm}(u)\big)\Big),

where W1W_1 and W2W_2 form a low-rank bottleneck (rank 16 in Qiu et al.). The gate reads the normalized vector and scales each channel by a number between 0 and 1, so GatedNorm keeps RMSNorm’s bound and adds a shrink that depends on the token; it is a squeeze-and-excitation gate applied per token.

The Qwen3.8-Next report tests it on a 28-layer MoE trained with AdamW at a constant learning rate three times the optimal one: the gate cut loss spikes from 32.0 to 3.2 per 10,000 steps and gradient-clipping events from 256 to 20. At a normal schedule the loss falls by only 0.002 on a plain pre-norm model and by 0.004 to 0.005 on models whose layers read earlier layers through attention (its Table 6, single runs). The report reads the stress test as suggesting “training at high learning rates requires a rescaling mechanism”.

In Qwen3.8-Flash-Next GatedNorm is merged into a widened residual stream, Gated Residual, which reads four parallel streams through one GatedNorm per sublayer and replaces the pre-norms and the final norm (R2 covers it). Its gate has rank 320 over a 10,240-wide input. It is the only norm here with a measurable cost: 6.55M gate parameters per instance, 97 instances (two per layer for 48 layers, plus one before the output head), about 641M parameters run for every token. Against the 6B active parameters the report gives, that is about 11% of the parameters and FLOPs per token (computed). Qwen credits the stack with training “without relying on explicit clipping methods such as qk-clip”; that claim rests on activation outliers and spikes, and Qwen3.8 also has QK-Norm, which bounds its attention logits (our reading). No other model in the map uses GatedNorm.

Side by side

MethodBoundsActs onNeedsExtra parameters (worked example or model)In the map
RMSNorm (pre-norm)each sublayer’s input, per tokenresidual stream, read sidenothingdd per norm; 4,096 per Qwen3.6 layer34 of 35 (Qwen3.8 uses GatedNorm)
Output norm (peri / sandwich)each sublayer’s writeresidual stream, write sidea choice of output gain2d2d per layer (Gemma 4: 645,120 in total)3 (Gemma 4, Kolibri-1, LFM3)
Zero-centred gainnothing new; makes decay pull gains to 1norm parametersnothingnone7 models, 3 labs
QK-Normattention logit, by dh\sqrt{d_h} × gainsper-head qq, kkper-head keys at decode2dh2d_h per layer (512)18 of 35, 10 labs
QK-Cliplogit, approximately, via weightsWqW_q, WkW_k after each stepper-head max logit written out by the kernel; the recipenoneKimi K3 (documents)
Per-head Muonnone directlyoptimizera split optimizer; the recipenoneGLM-5.2 (GLM-5 report), Kimi K3, DeepSeek-V4.1, Qwen3.8 (documents)
Output-logit soft-capoutput logits, at ±c\pm cfinal logitselementwise tanh⁡\tanhnoneGemma 4, Mistral Large 4 (c=30c = 30)
GatedNormchannel scale, by a gate in (0, 1)normalized streama low-rank gate6.55M per instance in Qwen3.8Qwen3.8-Flash-Next
Figure 5.The norms and clips in this article cost almost nothing per token, measured as extra parameters against the parameters a token uses: QK-Norm is about one hundred-thousandth of a layer. GatedNorm is the exception, because Qwen3.8's version carries a low-rank gate at every sublayer.

Where the models in the map sit

From the code of the 35 language models with a usable code check (robot models left out). “Pre” is a norm on the sublayer input only, “peri” adds one on the output, “GR” is Qwen’s Gated Residual, and “widened stream” means hyper-connections (R2). “Per-head” is QK-Norm as above; “latent” means MLA’s latent norms only. Multi-token-prediction modules carry norms of their own, and their code isn’t pinned for most models, so the table covers the main stack.

ModelsPlacementQueries and keysGain formOther logit controls
Gemma 4 31Bperiper-head, plus a value norm; K=V in global layers; scale 1.0w⋅xw \cdot xoutput soft-cap 30
Kolibri-1periper-head (also in position-free layers)w⋅xw \cdot xnone
LFM3-1.7Bperiper-headw⋅xw \cdot xnone
Qwen3.5-397B-A17B, Qwen3.6-35B-A3Bpreper-head1+w1 + wnone
Qwen3.8-Flash-NextGRper-head1+w1 + wGatedNorm
MiniMax-M3, M3.1-Flashpreper-head1+w1 + wnone
Step-3.7-Flash, Step-5-Previewpreper-head1+w1 + wnone
Laguna-S-2.1, Hy3preper-headw⋅xw \cdot xnone
Inklingpre, plus a norm after the embeddingper-head; scale 1/dh1/d_h; no RoPEw⋅xw \cdot xrelative-position logit term; output logits divided by 24
DeepSeek-V4 (5 models)pre (widened stream)queries without gain, K=V entry with gainw⋅xw \cdot xsinks
DeepSeek-V4.1-Flash (2)pre (widened stream)unclear: attention code not pinned; kv_norm visiblew⋅xw \cdot xsinks
DeepSeek-V3.2, GLM-5.2, GLM-5.3prelatentw⋅xw \cdot xnone
GLM-5.3-Flash, Hy4-previewpre (widened stream)latentw⋅xw \cdot xnone
Kimi K3pre, attention residualslatentw⋅xw \cdot xweight clip in training
Mistral Large 4prelatent (KV only); learned per-head log-position scale on queries, which sharpens with lengthw⋅xw \cdot xoutput soft-cap 30
Nemotron 3 Ultra, 3.5 Lightning, 3.5 Super-VLprenone (Super-VL unclear)w⋅xw \cdot xnone
gpt-oss-120b, MiMo-V2.5, V2.5-Pro, V2.6-Flash, V2.6-Proprenonew⋅xw \cdot xsinks; MiMo scales values by 0.707 or 0.612

The rows say what the code does, not what each lab found necessary: only Kolibri ablates QK-Norm, and nobody in the map ablates placement. Sinks are covered in T6, and bounded feed-forward activations (DeepSeek’s and MiMo’s SwiGLU clamps, Kimi K3’s capped gate) in C1.

Easy to get wrong

  • An MLA latent norm isn’t QK-Norm. q_a_layernorm and kv_a_layernorm normalize the latents before the up-projections, so the logit still scales with those weights. The code checks count them as absent, correctly.
  • “Post-norm” in Gemma’s reports means a norm on the sublayer output before the addition, not the original post-norm after it. OLMo 2’s “reordered norm” is a third thing: output only, no input norm.
  • QK-Norm bounds the scale of the logits, not the sharpness of attention, and it leaves the output logits alone.
  • In Llama-derived code, post_attention_layernorm is usually the feed-forward’s input norm. Read what the module wraps in forward, not its name; only Gemma 4’s is a norm on the attention output.
  • A zero-centred gain is the same function in exact arithmetic. It changes what weight decay and BF16 rounding do to the parameter; no report in the map measures its effect on stability.
  • A QK-Clip model doesn’t carry its bound. The clip is part of the training recipe, so a fine-tune with a high learning rate has none unless the fine-tuner adds it; QK-Norm stays in the forward pass.

Check yourself

  1. A head has dh=128d_h = 128 and QK-Norm with all gains at 1. What is the largest possible logit, and how large would both gain vectors’ largest entries have to be to allow a logit of 100?
    Answer∣z∣≤128≈11.3|z| \le \sqrt{128} \approx 11.3. To reach 100 you need 128 g2=100\sqrt{128}\, g^2 = 100, so g=100/11.31≈2.97g = \sqrt{100/11.31} \approx 2.97 on both queries and keys.
  2. In an MLA head, the largest logit in a batch is 250 and τ=100\tau = 100. By what factor does QK-Clip scale WqcW_{qc}, WkcW_{kc} and WqrW_{qr}, and what happens to the shared rotary key? Check that the logit scales by γ\gamma.
    Answerγ=100/250=0.4\gamma = 100/250 = 0.4. WqcW_{qc} and WkcW_{kc} by 0.4=0.632\sqrt{0.4} = 0.632, WqrW_{qr} by 0.4, kRk^R unchanged. Content part: 0.6322=0.40.632^2 = 0.4; rotary part: 0.4×1=0.40.4 \times 1 = 0.4. So z→0.4zz \to 0.4z: the maximum as measured before the update becomes 100.
  3. Why can’t DeepSeek-V3.2 use QK-Norm the way DeepSeek-V4 does?
    AnswerV3.2 uses MLA: at decode time a head’s key is never formed, because its up-projection is absorbed into the query, and a key norm divides by a different number for every token and head, which can’t be absorbed. Normalizing would mean forming the keys again or caching one extra number per token and head. V4 computes one key-value head directly from the hidden state, so the key exists and can be normalized; the same vector is the value, which gets normalized too.
  4. A peri-norm stack of 50 blocks (two sublayers each) has embeddings with squared norm dd and initializes every output-norm gain at 1/501/\sqrt{50}. Under the toy assumption that each write is uncorrelated with the stream, what is the stream’s squared norm after the last block?
    AnswerEach of the 100 sublayers adds s2d=d/50s^2 d = d/50, so d+100⋅d/50=3dd + 100 \cdot d/50 = 3d; the RMS has grown by 3≈1.73\sqrt 3 \approx 1.73.

In the map

Methods

Taxonomy nodes

Further reading

  • Zhang and Sennrich (2019), arXiv:1910.07467: RMSNorm, its invariance table and the speed measurements behind “comparable”.
  • Xiong et al. (2020), arXiv:2002.04745: the post-norm and pre-norm gradient analysis at initialization, and the warm-up experiments.
  • Kim et al. (2025), arXiv:2502.02732: Peri-LN, with variance and redundancy measurements in trained pre-norm models.
  • Wortsman et al. (2023), arXiv:2309.14322: attention-logit growth and output-logit divergence reproduced in small models, with qk-layernorm and z-loss.
  • Dehghani et al. (2023), arXiv:2302.05442: the 8B ViT divergence and the QK LayerNorm that fixed it.
  • Kimi Team (2025), arXiv:2507.20534: MuonClip, its Algorithm 1, the MLA split and the self-deactivation numbers.
  • DeepSeek-AI (2026), arXiv:2606.19348: shared key-value attention and the norm that replaced QK-Clip.
  • Qwen Team (2026), arXiv:2608.30320, and Qiu et al. (2026), arXiv:2601.22966: GatedNorm, the stress tests and the outlier-rescaling argument.
  • Team OLMo (2024), arXiv:2501.00656: the reordered-norm and QK-Norm ablation, with spike scores.
↑