DeepSeek-V4-Pro
DeepSeekapplying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process
1 more from this model
SwiGLU Clamping
implementation detail · filed under model architecture
SwiGLU clamping constrains the numerical ranges of the SwiGLU activation's components during training.1 It clamps the linear component of SwiGLU to a fixed range and caps the gate component's upper bound, which removes outliers and stabilizes training.2
Trade-off It eliminates outliers and stabilizes training without compromising model performance.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
clamping has been explicitly utilized to constrain numerical ranges— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
we clamped the linear component of SwiGLU to the range of , while capping the upper bound of the gate component at— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
effectively eliminates outliers and substantially aids in stabilizing the training process, without compromising performance— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Also called SwiGLU activation with clamping, SwiGLU activation function with clamping.
2 quotes from 2 sources, one card per model, each opening with its strongest.
applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process
SwiGLU Clamping
applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process
Picked by hand, not extracted: where to read more, not evidence for anything on this page.