single mention · model architecture
Pre-LN
Pre-LN (pre-normalization) placement applies normalization to the activations before each block.1,2 Each attention and MoE block therefore receives already-normalized activations from the residual stream.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we use Pre-LN placement
— gpt-oss-120b & gpt-oss-20b Model Cardapplying root mean square normalization [6] on the activations before each attention and MoE block
— gpt-oss-120b & gpt-oss-20b Model Cardon the activations before each attention and MoE block
— gpt-oss-120b & gpt-oss-20b Model Card
Also called Pre-LN (pre-normalization) placement, Pre-LN placement.
Filed under model architecture › normalization & residual