Open Models Almanac
Methodsmodel architecturenormalization & residual

single source · model architecture

GatedNorm

GatedNorm multiplies an RMSNorm's output by a sigmoid gate computed from it through a low-rank bottleneck, so each channel can be scaled down per token. Qwen merged it into the Gated Residual of Qwen3.8.

Written for the explainer Normalization and logit control.

Also called GatedNorm elementwise self-gating, elementwise self-gate after RMSNorm, GatedNorm (multiplicative gated rescaling), multiplicative gate.

Filed under model architecture › normalization & residual