DeepSeek-V4-Flash · DeepSeek-V4-Pro
DeepSeekwe perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries
specific method · filed under model architecture
Normalizes activations by their root-mean-square scale without centering them.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See normalization & residual for the methods filed beside it.
Also called RMSNorm normalization, root mean square normalization (RMSNorm), root mean square normalization.
2 quotes from 2 sources, one card per model, each opening with its strongest.
we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries
applying root mean square normalization [6] on the activations before each attention and MoE block
Picked by hand, not extracted: where to read more, not evidence for anything on this page.