Open Models Almanac
Methodsmodel architecturechannel mixerdense feed-forward network

implementation detail · filed under model architecture

SwiGLU clamping

SwiGLU clamping constrains the numerical ranges of the SwiGLU activation's components during training.1 It clamps the linear component of SwiGLU to a fixed range and caps the gate component's upper bound, which removes outliers and stabilizes training.2

Trade-off It eliminates outliers and stabilizes training without compromising model performance.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. clamping has been explicitly utilized to constrain numerical ranges — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. we clamped the linear component of SwiGLU to the range of , while capping the upper bound of the gate component at — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. effectively eliminates outliers and substantially aids in stabilizing the training process, without compromising performance — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called SwiGLU activation with clamping, SwiGLU activation function with clamping.

sources
2
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑