Open Models Almanac
Methodsmodel architecturenormalization & residual

single mention · model architecture

Pre-LN

Pre-LN (pre-normalization) placement applies normalization to the activations before each block.1,2 Each attention and MoE block therefore receives already-normalized activations from the residual stream.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we use Pre-LN placement — gpt-oss-120b & gpt-oss-20b Model Card
  2. applying root mean square normalization [6] on the activations before each attention and MoE block — gpt-oss-120b & gpt-oss-20b Model Card
  3. on the activations before each attention and MoE block — gpt-oss-120b & gpt-oss-20b Model Card

Also called Pre-LN (pre-normalization) placement, Pre-LN placement.

Filed under model architecture › normalization & residual