single source · model architecture
Independent per-branch normalization in Gated Residual
In Gated Residual, each branch of the widened residual stream is normalized separately before being combined.1 The normalization is a group RMSNorm applied over the widened stream, which gives a further gain over pooling the branches first.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
GR first normalizes each branch independently
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityNormalizing each branch separately, that is, a group RMSNorm over the widened stream, gives a further gain
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Also called group RMSNorm over the widened stream.
Filed under model architecture › normalization & residual