methods · channel mixer
dense feed-forward network
7 methods filed at this node or below it, from the sources of 7 models.
In practice
Checked as “SwiGLU feed-forward” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | not stated | hidden_act=siluBoth the dense MLP and MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu. text_config.hidden_act = "silu" | code → |
| LFM3-1.7B | in its code | not stated | verdict and linesLfm3MLP computes SiLU(w1(x)) * w3(x) then projects with w2, a SwiGLU feed-forward used by every layer. | code → |
| MiniMax-M3.1-Flash | in its code | not stated | hidden_act=swigluoaiBoth the dense MLP and the MoE experts compute a SiLU-sigmoid-gated SwiGLU-style activation ((up+1)*gate*sigmoid(alpha*gate)) matching hidden_act swigluoai. text_config.hidden_act = "swigluoai" | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | hidden_act=siluhidden_act is silu and expert/FFN MLPs are built with gate_up_proj combining w1 (gate) and w3 (up) shards, matching a SiLU-gated SwiGLU MLP. text_config.hidden_act = "silu" | code → |
| Step-5-Preview | in its code | not stated | verdict and linesMLP and expert feed-forwards use SiluAndMul (SiLU-gated) activation, explicitly rejecting any activation other than silu. | code → |
| Kolibri-1 | in its code | not stated | hidden_act=siluBoth the shared-expert MLP and the routed experts compute silu(gate(x)) * up(x) as hidden_act is silu. hidden_act = "silu" | code → |
| MiMo-V2.6-Flash | in its code | not stated | hidden_act=siluMiMoV2MLP (used both as dense MLP and MoE expert) computes down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act=silu, i.e. SwiGLU. hidden_act = "silu" | code → |
| MiMo-V2.6-Pro | in its code | not stated | hidden_act=siluBoth dense and expert MLPs compute down_proj(act_fn(gate_proj(x)) * up_proj(x)) with hidden_act set to silu, giving SwiGLU. hidden_act = "silu" | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | hidden_act=siluhidden_act is silu and the MoE/MLP implementation uses gate_up_proj with w1 (gate) and w3 (up) stacking typical of SiLU-gated SwiGLU MLPs. text_config.hidden_act = "silu" | code → |
| Hy4-preview | in its code | not stated | hidden_act=siluBoth the dense HYV4MLP and the routed HYV4Experts compute SiLU(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'. hidden_act = "silu" | code → |
| GLM-5.3-Flash | in its code | not stated | hidden_act=siluhidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both dense and expert FFNs. text_config.hidden_act = "silu" | code → |
| Qwen3.8-Flash-Next | in its code | not stated | hidden_act=siluBoth the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, matching hidden_act=silu. text_config.hidden_act = "silu" | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | hidden_act=siluBoth the dense MLP and the routed experts compute SiLU(gate(x)) * up(x) (clamped), consistent with hidden_act=silu. hidden_act = "silu" | code → |
| GLM-5.3 | in its code | not stated | hidden_act=siluBoth the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, with hidden_act="silu". hidden_act = "silu" | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | hidden_act=siluBoth the shared MLP and routed experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU hidden_act = "silu" | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | hidden_act=siluBoth the dense MLP and the expert MLPs compute SiLU(gate(x)) * up(x) with hidden_act set to silu. hidden_act = "silu" | code → |
| Xiaomi-Robotics-1 | in its code | not stated | hidden_act=siluQwen3VLTextMLP computes silu(gate_proj(x)) * up_proj(x) then down_proj, a SwiGLU feed-forward, matching hidden_act 'silu' in the text config. vlm_config.text_config.hidden_act = "silu" | code → |
| Laguna-S-2.1 | in its code | not stated | verdict and linesBoth the dense LagunaMLP and the expert MLP compute act_fn(gate_proj(x)) * up_proj(x) with hidden_act='silu' (SiLU = SwiGLU). | code → |
| Inkling | in its code | not stated | verdict and linesDense and shared-expert MLPs compute SiLU(gate(x)) * up(x) (silu_and_mul / torch.nn.functional.silu gating), matching SwiGLU, consistent with hidden_act='silu' default (not overridden to gelu in this config). | code → |
| Hy3 | in its code | not stated | hidden_act=siluBoth the dense MLP and the MoE experts compute silu(gate(x)) * up(x) before the down projection. hidden_act = "silu" | code → |
| GLM-5.2 | in its code | not stated | hidden_act=siluBoth the dense MLP and the MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu. hidden_act = "silu" | code → |
| MiniMax-M3 | in its code | not stated | hidden_act=swigluoaiDense and expert MLPs compute a SiLU-sigmoid gated SwiGLU-OAI variant (gate * sigmoid(gate*alpha) multiplied by (up+1)) rather than GELU gating. text_config.hidden_act = "swigluoai" | code → |
| Step-3.7-Flash | in its code | not stated | verdict and linesBoth the dense MLP and the MoE experts compute SiLU(gate_proj(x)) * up_proj(x) (hidden_act is normalized to 'silu'), matching SwiGLU. | code → |
| MolmoAct 2 | in its code | not stated | hidden_act=siluLanguageModelMLP computes SiLU(gate) * x with hidden_act='silu', matching SwiGLU for every decoder layer's feed-forward. text_config.hidden_act = "silu" | code → |
| DeepSeek-V4-Flash | in its code | adopted | hidden_act=siluBoth the dense MLP and the routed Experts module compute act_fn(gate) * up with hidden_act set to silu, i.e. SwiGLU. hidden_act = "silu" | code → |
| DeepSeek-V4-Pro | in its code | adopted | hidden_act=siluBoth the dense DeepseekV4MLP and the MoE DeepseekV4Experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU. hidden_act = "silu" | code → |
| MiMo-V2.5 | in its code | not stated | hidden_act=siluMLP and expert FFNs compute down_proj(silu(gate_proj(x)) * up_proj(x)) with hidden_act set to silu. hidden_act = "silu" | code → |
| MiMo-V2.5-Pro | in its code | not stated | hidden_act=siluMLP and expert feed-forwards compute silu(gate_proj(x)) * up_proj(x) then down_proj, with hidden_act set to silu. hidden_act = "silu" | code → |
| Qwen3.6-35B-A3B | in its code | not stated | hidden_act=siluBoth the shared/dense MLP and the routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, per hidden_act=silu. text_config.hidden_act = "silu" | code → |
| Qwen3.5-397B-A17B | in its code | not stated | hidden_act=siluBoth the dense MLP and expert MLPs compute act_fn(gate_proj(x)) * up_proj(x) with hidden_act=silu, i.e. SwiGLU. text_config.hidden_act = "silu" | code → |
| DeepSeek-V3.2 | in its code | not stated | hidden_act=siluBoth the dense DeepseekV32MLP and the MoE DeepseekV32Experts compute silu(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'. hidden_act = "silu" | code → |
| gpt-oss-120b | in its code | adopted | hidden_act=siluExpert MLPs compute a sigmoid-gated (SiLU-style) gate times up-projection, with hidden_act silu, implementing a clamped SwiGLU hidden_act = "silu" | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | not in its code | not stated | mlp_hidden_act=relu2The text_config sets mlp_hidden_act to 'relu2' rather than a SiLU-gated activation, so the feed-forward/expert MLPs are not SwiGLU. llm_config.mlp_hidden_act = "relu2" | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | mlp_hidden_act=relu2Both the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared ReLU), not a SiLU-gated two-branch SwiGLU. mlp_hidden_act = "relu2" | code → |
| Kimi K3 | not in its code | not stated | hidden_act=situhidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU. text_config.hidden_act = "situ" | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | mlp_hidden_act=relu2Both the dense MLP and the MoE experts use a single up_proj/down_proj pair with a relu2 activation (no gating projection, no SiLU), so this is not SwiGLU. mlp_hidden_act = "relu2" | code → |
| Gemma 4 31B | not in its code | not stated | hidden_activation=gelu_pytorch_tanhText MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than SwiGLU. text_config.hidden_activation = "gelu_pytorch_tanh" | code → |
| X-VLA | not in its code | not stated | activation_function=geluFeed-forward blocks use GELU activation (Florence2 text config activation_function is gelu, and transformer.py's Mlp uses nn.GELU), not a SiLU-gated SwiGLU structure. florence_config.text_config.activation_function = "gelu" | code → |
| OpenVLA / OpenVLA-OFT | code unclear | not stated | model_type=llamaThe language model is a stock Llama-2 backbone loaded via AutoModelForCausalLM, whose MLP activation (hidden_act) is not set in the given text_config and the implementing file is not provided, so SwiGLU cannot be confirmed from this input. text_config.model_type = "llama" | code → |
In this branch 7
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 7
4 from one source, or only mentioned as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| DeepSeek-V4-Flash used | SwiGLU usedSwiGLU clamping used— |
| DeepSeek-V4-Pro used | SwiGLU usedSwiGLU clamping used— |
| Kimi K3 core | SiTU-GLU coreSigmoid Tanh Unit (SiTU) used— |
| Kolibri-1 not used | Leading dense feed-forward layers before MoE layers not used— |
| MiMo-V2.5 used | Dense feed-forward network used— |
| Qwen3.8-Flash-Next not used | SwiGLU-clip not used— |
| gpt-oss-120b used | SwiGLU used— |