specific method · filed under model architecture
SwiGLU
A gated activation used in the MoE blocks and described as the activation function.
Also called gated SwiGLU activation, gated SwiGLU [9] activation function, SwiGLU activation, SwiGLU activation function.
- sources
- 3
- models
- 4
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- GLU Variants Improve Transformer (Shazeer, 2020) paper arxiv.orgintroduces SwiGLU
In the code
Checked as “SwiGLU feed-forward” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | Both the dense Glm5NextTextMLP and the MoE Glm5NextTextExperts compute silu(gate(x)) * up(x) (clamped SwiGLU) with hidden_act set to silu. modeling_glm5_next.py#L103 |
| DeepSeek-V4.1-Flash | in its code | used | hidden_act is 'silu' and MoE/shared expert MLPs use a gate_up_proj (SiLU-gated) * up projection pattern typical of SwiGLU feed-forward blocks. model.py#L926 |
| Hy4-preview | in its code | not stated | Both the dense HYV4MLP and the MoE HYV4Experts compute silu(gate(x)) * up(x), a SwiGLU feed-forward, with hidden_act set to silu. modeling_hy_v4.py#L506 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | Both the dense MLP and the MoE experts compute SiLU(gate(x)) * up(x) with hidden_act=silu. modeling_deepseek_v4.py#L1025 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | MLP and expert feed-forwards use a single up_proj/down_proj pair with a relu2 activation (act(up(x)) then down_proj), not a SiLU-gated gate/up product. modeling_nemotron_h.py#L613 |
| MiMo-V2.6-Flash | in its code | not stated | MiMoV2MLP (used both as dense MLP and MoE expert) computes down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act=silu, i.e. SwiGLU. modeling_mimo_v2.py#L132 |
| DeepSeek-V4-Flash | in its code | used | Both the dense MLP and the MoE experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU. modeling_deepseek_v4.py#L1025 |
| MiMo-V2.5 | in its code | not stated | MLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu. modeling_mimo_v2.py#L132 |
| GLM-5.3 | in its code | not stated | Both the dense MLP and the routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, matching hidden_act='silu'. modeling_glm_moe_dsa.py#L477 |
| Hy3 | in its code | not stated | Both the dense MLP and expert MLPs compute silu(gate(x)) * up(x), i.e. SwiGLU, with hidden_act set to silu modeling_hy_v3.py#L135 |
| GLM-5.2 | in its code | not stated | Both the dense MLP and routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, with hidden_act set to silu. modeling_glm_moe_dsa.py#L477 |
| MiniMax-M3 | in its code | not stated | Both the dense MLP and expert MLPs compute a SiLU-sigmoid-gated GLU (gate * sigmoid(gate*alpha)) multiplied by up, i.e. a SwiGLU variant (swigluoai), not GeGLU. modeling_minimax_m3_vl.py#L183 |
| DeepSeek-V3.2 | in its code | not stated | Both the dense MLP and MoE experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, matching hidden_act 'silu'. modeling_deepseek_v32.py#L506 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | Both the dense MLP and expert MLPs compute silu(gate(x)) * up(x) (with clamping), and hidden_act is silu in this config. modeling_deepseek_v4.py#L1025 |
| DeepSeek-V4-Pro | in its code | used | hidden_act is silu and both the dense MLP and the MoE experts compute act_fn(gate) * up with SiLU activation modeling_deepseek_v4.py#L1025 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | Both the dense MLP and expert MLPs compute SiLU(gate(x)) * up(x) with hidden_act=silu, clamped by swiglu_limit. modeling_deepseek_v4.py#L1025 |
| Gemma 4 31B | not in its code | not stated | The MLP computes act(gate(x)) * up(x), but hidden_activation is 'gelu_pytorch_tanh' (GeGLU), not SiLU-based SwiGLU. modeling_gemma4.py#L702 |
| Inkling | in its code | not stated | Both dense and shared-expert MLPs compute SiLU(gate(x)) * up(x) via silu_and_mul_triton / F.silu-based gating, consistent with hidden_act=silu. mlp.py#L57 |
| Kimi K3 | not in its code | not stated | hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU. modeling_kimi_linear.py#L79 |
| Laguna-S-2.1 | in its code | not stated | Both the dense LagunaMLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), and hidden_act is silu. modeling_laguna.py#L153 |
| MiMo-V2.5-Pro | in its code | not stated | MLP and expert feed-forwards compute silu(gate_proj(x)) * up_proj(x) then down_proj, with hidden_act set to silu. modeling_mimo_v2.py#L127 |
| MiMo-V2.6-Pro | in its code | not stated | Both dense and expert MLPs compute down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act set to silu, giving SwiGLU. modeling_mimo_v2.py#L132 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | Both the dense NemotronHMLP and the MoE experts use a single up_proj followed by a relu2 activation and down_proj with no gating multiplication, i.e. not SwiGLU. modeling_nemotron_h.py#L613 |
| Qwen3.5-397B-A17B | in its code | not stated | Both the dense MLP and the expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu. modeling_qwen3_5_moe.py#L839 |
| Qwen3.6-35B-A3B | in its code | not stated | Both the dense MLP and the expert MLPs compute silu(gate_proj(x)) * up_proj(x) before down_proj, i.e. SwiGLU with hidden_act=silu. modeling_qwen3_5_moe.py#L840 |
| Qwen3.8-Flash-Next | in its code | not stated | Both the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, matching hidden_act=silu. modeling_qwen4_exp.py#L915 |
| Step-3.7-Flash | in its code | not stated | Step3p7MLP and Step3p7Experts compute silu(gate_proj(x)) * up_proj(x) style gating, and hidden_act is normalized to 'silu' in the config post_init. modeling_step3p7.py#L547 |
| gpt-oss-120b | in its code | used | Expert MLPs compute a SiLU-based gated GLU (sigmoid-approximated SiLU gate times up-projection) as the feed-forward activation. modeling_gpt_oss.py#L86 |
Evidence
3 spans quoted from the sources, strongest treatment first.
The MoE blocks use the gated SwiGLU [9] activation function
usedmodel architecturein gpt-oss-120b and gpt-oss-20bOpenAI
each token-expert pair requires FLOPs (SwiGLU gate, up, and down projections)
usedmodel architecturein DeepSeek-V4DeepSeek
and SwiGLU (Shazeer, 2020) as the activation function.
usedmodel architecturein DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under model architecture :: channel mixer :: dense feed-forward network.