Open Models Almanac
Methodsmodel architecturenormalization & residual

single source · model architecture

Independent per-branch normalization in Gated Residual

In Gated Residual, each branch of the widened residual stream is normalized separately before being combined.1 The normalization is a group RMSNorm applied over the widened stream, which gives a further gain over pooling the branches first.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. GR first normalizes each branch independently — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. Normalizing each branch separately, that is, a group RMSNorm over the widened stream, gives a further gain — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Also called group RMSNorm over the widened stream.

Filed under model architecture › normalization & residual