Open Models Almanac
Methodsmodel architecturenormalization & residual

single mention · model architecture

Zero-centered RMSNorm

A zero-centred RMSNorm stores its gain as an offset w from 1 and multiplies by (1 + w), with w starting at 0. The function is unchanged; weight decay then pulls the gain toward 1 instead of toward 0.

Written for the explainer Normalization and logit control.

Filed under model architecture › normalization & residual