specific method · filed under model architecture
GatedNorm
An elementwise self-gate applied after RMSNorm to provide multiplicative rescaling.
Also called GatedNorm elementwise self-gating, elementwise self-gate after RMSNorm, GatedNorm (multiplicative gated rescaling), multiplicative gate.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
3 spans quoted from the sources, strongest treatment first.
the multiplicative gate supplies the necessary rescaling directly, keeping the training stable
adding a lightweight elementwise self-gate after RMSNorm, which we call GatedNorm, markedly improves training stability
we evaluate a single-variable pair on the 28-layer model at its optimal learning rate, keeping the AdmaW optimizer and structure fixed while toggling GatedNorm
Filed alongside
Other methods under model architecture :: normalization & residual.