specific method · filed under model architecture
SwiGLU clamping
Clamping applied to SwiGLU activations to control outliers and improve training stability.
Also called SwiGLU activation with clamping, SwiGLU activation function with clamping.
- sources
- 3
- models
- 3
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- gpt-oss-120b & gpt-oss-20b Model Card paper arxiv.orgthe clamped SwiGLU DeepSeek cites as (OpenAI, 2025)
Evidence
3 spans quoted from the sources, strongest treatment first.
applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process
usedoptimizationin DeepSeek-V4DeepSeek
using SwiGLU activation function with clamping (OpenAI, 2025) at a threshold of 10.
usedunclearin DeepSeek-V4.1-FlashDeepSeek
SwiGLU Clamping
usedtraining objectivein DeepSeek-V4DeepSeek
Filed alongside
Other methods under model architecture :: channel mixer :: dense feed-forward network.