Open Models Almanac
Methodsmodel architecturechannel mixerdense feed-forward network

specific method · filed under model architecture

SwiGLU

A gated activation used in feed-forward blocks, including the cited MoE blocks.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See dense feed-forward network for the methods filed beside it.

Also called gated SwiGLU activation, gated SwiGLU [9] activation function, SwiGLU activation, SwiGLU activation function.

sources
2
models
3
labs adopt it
2
strongest
used

In practice

Checked as “SwiGLU feed-forward” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codenot stated
hidden_act=silu

Both the dense MLP and MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu.

text_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_mistral_large4.py · L508–509
LFM3-1.7Bin its codenot stated
verdict and lines

Lfm3MLP computes SiLU(w1(x)) * w3(x) then projects with w2, a SwiGLU feed-forward used by every layer.

def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))
modeling_lfm3.py · L134–135
MiniMax-M3.1-Flashin its codenot stated
hidden_act=swigluoai

Both the dense MLP and the MoE experts compute a SiLU-sigmoid-gated SwiGLU-style activation ((up+1)*gate*sigmoid(alpha*gate)) matching hidden_act swigluoai.

text_config.hidden_act = "swigluoai"
glu = gate * torch.sigmoid(gate * self.swiglu_alpha)
        return self.down_proj((up + 1.0) * glu)
modeling_minimax_m3_vl.py · L176–177
DeepSeek-V4.1-Flash-1017in its codenot stated
hidden_act=silu

hidden_act is silu and expert/FFN MLPs are built with gate_up_proj combining w1 (gate) and w3 (up) shards, matching a SiLU-gated SwiGLU MLP.

text_config.hidden_act = "silu"
("gate_up_proj", "w1", 0),
            ("gate_up_proj", "w3", 1),
model.py · L1369–1370
Step-5-Previewin its codenot stated
verdict and lines

MLP and expert feed-forwards use SiluAndMul (SiLU-gated) activation, explicitly rejecting any activation other than silu.

if hidden_act != "silu":
            raise ValueError(
                f"Unsupported activation: {hidden_act}. Only silu is supported for now."
            )
        self.act_fn = SiluAndMul()
model.py · L812–816
Kolibri-1in its codenot stated
hidden_act=silu

Both the shared-expert MLP and the routed experts compute silu(gate(x)) * up(x) as hidden_act is silu.

hidden_act = "silu"
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_kolibri1.py · L221
            gate, up = nn.functional.linear(current_state, self.gate_up_proj[expert_idx]).chunk(2, dim=-1)
            current_hidden_states = self.act_fn(gate) * up
modeling_kolibri1.py · L256–257
MiMo-V2.6-Flashin its codenot stated
hidden_act=silu

MiMoV2MLP (used both as dense MLP and MoE expert) computes down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act=silu, i.e. SwiGLU.

hidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L132
MiMo-V2.6-Proin its codenot stated
hidden_act=silu

Both dense and expert MLPs compute down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act set to silu, giving SwiGLU.

hidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L132
DeepSeek-V4.1-Flashin its codenot stated
hidden_act=silu

hidden_act is silu and the MoE/MLP implementation uses gate_up_proj with w1 (gate) and w3 (up) stacking typical of SiLU-gated SwiGLU MLPs.

text_config.hidden_act = "silu"
            ("gate_up_proj", "w1", 0),
            ("gate_up_proj", "w3", 1),
model.py · L1369–1370
Hy4-previewin its codenot stated
hidden_act=silu

Both the dense HYV4MLP and the routed HYV4Experts compute SiLU(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'.

hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_hy_v4.py · L495
return F.silu(gate) * up
modeling_hy_v4.py · L575
GLM-5.3-Flashin its codenot stated
hidden_act=silu

hidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both dense and expert FFNs.

text_config.hidden_act = "silu"
        gate = gate.clamp(min=None, max=self.swiglu_limit)
        up = up.clamp(min=-self.swiglu_limit, max=self.swiglu_limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_glm5_next.py · L101–103
Qwen3.8-Flash-Nextin its codenot stated
hidden_act=silu

Both the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, matching hidden_act=silu.

text_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_qwen4_exp.py · L903
current_hidden_states = self.act_fn(gate) * up
modeling_qwen4_exp.py · L939
DeepSeek-V4-Flash-Vision-Expin its codenot stated
hidden_act=silu

Both the dense MLP and the routed experts compute SiLU(gate(x)) * up(x) (clamped), consistent with hidden_act=silu.

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
gate, up = gate_up.chunk(2, dim=-1)
        gate = gate.clamp(max=self.limit)
        up = up.clamp(min=-self.limit, max=self.limit)
        return self.act_fn(gate) * up
modeling_deepseek_v4.py · L1073–1076
GLM-5.3in its codenot stated
hidden_act=silu

Both the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, with hidden_act="silu".

hidden_act = "silu"
def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_glm_moe_dsa.py · L464–466
DeepSeek-V4-Pro-0813in its codenot stated
hidden_act=silu

Both the shared MLP and routed experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
DeepSeek-V4-Flash-0731in its codenot stated
hidden_act=silu

Both the dense MLP and the expert MLPs compute SiLU(gate(x)) * up(x) with hidden_act set to silu.

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
Xiaomi-Robotics-1in its codenot stated
hidden_act=silu

Qwen3VLTextMLP computes silu(gate_proj(x)) * up_proj(x) then down_proj, a SwiGLU feed-forward, matching hidden_act 'silu' in the text config.

vlm_config.text_config.hidden_act = "silu"
def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_mibot.py · L483–485
Laguna-S-2.1in its codenot stated
verdict and lines

Both the dense LagunaMLP and the expert MLP compute act_fn(gate_proj(x)) * up_proj(x) with hidden_act='silu' (SiLU = SwiGLU).

down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_laguna.py · L142
Inklingin its codenot stated
verdict and lines

Dense and shared-expert MLPs compute SiLU(gate(x)) * up(x) (silu_and_mul / torch.nn.functional.silu gating), matching SwiGLU, consistent with hidden_act='silu' default (not overridden to gelu in this config).

gate_up, _ = self.gate_up_proj(x)
        x = silu_and_mul_triton(gate_up)
        x, _ = self.down_proj(x)
mlp.py · L57–59
hidden_states = torch.nn.functional.silu(gate) * up
moe.py · L403
Hy3in its codenot stated
hidden_act=silu

Both the dense MLP and the MoE experts compute silu(gate(x)) * up(x) before the down projection.

hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_hy_v3.py · L124
GLM-5.2in its codenot stated
hidden_act=silu

Both the dense MLP and the MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu.

hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_glm_moe_dsa.py · L465
MiniMax-M3in its codenot stated
hidden_act=swigluoai

Dense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU gating.

text_config.hidden_act = "swigluoai"
glu = gate * torch.sigmoid(gate * self.swiglu_alpha)
        return self.down_proj((up + 1.0) * glu)
modeling_minimax_m3_vl.py · L176–177
Step-3.7-Flashin its codenot stated
verdict and lines

Both the dense MLP and the MoE experts compute SiLU(gate_proj(x)) * up_proj(x) (hidden_act is normalized to 'silu'), matching SwiGLU.

gate = self.act_fn(self.gate_proj(x)).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(gate * up)
modeling_step3p7.py · L537–539
MolmoAct 2in its codenot stated
hidden_act=silu

LanguageModelMLP computes SiLU(gate) * x with hidden_act='silu', matching SwiGLU for every decoder layer's feed-forward.

text_config.hidden_act = "silu"
def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.ff_proj(x)
        x, gate = x.chunk(2, dim=-1)
        x = self.act(gate) * x
        x = self.ff_out(x)
        return x
modeling_molmoact2.py · L2351–2356
DeepSeek-V4-Flashin its codeadopted
hidden_act=silu

Both the dense MLP and the routed Experts module compute act_fn(gate) * up with hidden_act set to silu, i.e. SwiGLU.

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
DeepSeek-V4-Proin its codeadopted
hidden_act=silu

Both the dense DeepseekV4MLP and the MoE DeepseekV4Experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU.

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
MiMo-V2.5in its codenot stated
hidden_act=silu

MLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu.

hidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L132
MiMo-V2.5-Proin its codenot stated
hidden_act=silu

MLP and expert feed-forwards compute silu(gate_proj(x)) * up_proj(x) then down_proj, with hidden_act set to silu.

hidden_act = "silu"
return self.down_proj(self.act_fn(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
modeling_mimo_v2.py · L127
Qwen3.6-35B-A3Bin its codenot stated
hidden_act=silu

Both the shared/dense MLP and the routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, per hidden_act=silu.

text_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_qwen3_5_moe.py · L820
Qwen3.5-397B-A17Bin its codenot stated
hidden_act=silu

Both the dense MLP and expert MLPs compute act_fn(gate_proj(x)) * up_proj(x) with hidden_act=silu, i.e. SwiGLU.

text_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_qwen3_5_moe.py · L820
DeepSeek-V3.2in its codenot stated
hidden_act=silu

Both the dense DeepseekV32MLP and the MoE DeepseekV32Experts compute silu(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'.

hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_deepseek_v32.py · L492
gpt-oss-120bin its codeadopted
hidden_act=silu

Expert MLPs compute a sigmoid-gated (SiLU-style) gate times up-projection, with hidden_act silu, implementing a clamped SwiGLU

hidden_act = "silu"
glu = gate * torch.sigmoid(gate * self.alpha)
        gated_output = (up + 1) * glu
modeling_gpt_oss.py · L85–86
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
mlp_hidden_act=relu2

The text_config sets mlp_hidden_act to 'relu2' rather than a SiLU-gated activation, so the feed-forward/expert MLPs are not SwiGLU.

llm_config.mlp_hidden_act = "relu2"
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
mlp_hidden_act=relu2

Both the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared ReLU), not a SiLU-gated two-branch SwiGLU.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]

    def forward(self, x):
        return self.down_proj(self.act_fn(self.up_proj(x)))
modeling_nemotron_h.py · L643–648
Kimi K3not in its codenot stated
hidden_act=situ

hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU.

text_config.hidden_act = "situ"
situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate)
        if self.linear_beta is not None:
            up = self.linear_beta * torch.tanh(up / self.linear_beta)
        return (situ_a * up).to(x.dtype)
modeling_kimi_linear.py · L79–82
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
mlp_hidden_act=relu2

Both the dense MLP and the MoE experts use a single up_proj/down_proj pair with a relu2 activation (no gating projection, no SiLU), so this is not SwiGLU.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]
modeling_nemotron_h.py · L643–645
Gemma 4 31Bnot in its codenot stated
hidden_activation=gelu_pytorch_tanh

Text MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than SwiGLU.

text_config.hidden_activation = "gelu_pytorch_tanh"
def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_gemma4.py · L707–709
X-VLAnot in its codenot stated
activation_function=gelu

Feed-forward blocks use GELU activation (Florence2 text config activation_function is gelu, and transformer.py's Mlp uses nn.GELU), not a SiLU-gated SwiGLU structure.

florence_config.text_config.activation_function = "gelu"
self.activation_fn = ACT2FN[config.activation_function]
modeling_florence2.py · L1237
self.act = nn.GELU(approximate="tanh")
transformer.py · L70
OpenVLA / OpenVLA-OFTcode unclearnot stated
model_type=llama

The language model is a stock Llama-2 backbone loaded via AutoModelForCausalLM, whose MLP activation (hidden_act) is not set in the given text_config and the implementing file is not provided, so SwiGLU cannot be confirmed from this input.

text_config.model_type = "llama"

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑