specific method · filed under model architecture
RMSNorm
A normalization method based on the root mean square of activations; one cited use applies it to attention queries and compressed key-value entries.
Also called RMSNorm normalization, root mean square normalization (RMSNorm), root mean square normalization.
- sources
- 3
- models
- 4
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Root Mean Square Layer Normalization (Zhang & Sennrich, 2019) paper arxiv.orgintroduces RMSNorm
Evidence
3 spans quoted from the sources, strongest treatment first.
applying root mean square normalization [6] on the activations before each attention and MoE block
we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries
We also adopt RMSNorm (Zhang and Sennrich, 2019) for normalization
Filed alongside
Other methods under model architecture :: normalization & residual.