single mention · model architecture
Zero-centered RMSNorm
A zero-centred RMSNorm stores its gain as an offset w from 1 and multiplies by (1 + w), with w starting at 0. The function is unchanged; weight decay then pulls the gain toward 1 instead of toward 0.
Written for the explainer Normalization and logit control.
Filed under model architecture › normalization & residual