taxonomy node · level 3
dense feed-forward network
6 methods filed at this node or below it, from the sources of 7 models.
model architecture :: channel mixer :: dense feed-forward network
Matching aids for the classifier: MLP block; SwiGLU; SiTU; activation function; SwiGLU clamping.
In this branch 6
Everything filed at this node or below it, with one collapsible heading per child node.
By model
Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | techniques |
|---|---|
| DeepSeek-V4.1-Flash used | SwiGLU usedSwiGLU clamping used— |
| DeepSeek-V4-Flash used | SwiGLU usedSwiGLU clamping used— |
| MiMo-V2.5 used | Dense feed-forward network used— |
| DeepSeek-V4-Pro used | SwiGLU usedSwiGLU clamping used— |
| Kimi K3 core | SiTU-GLU coreSiTU used— |
| Qwen3.8-Flash-Next not used | SwiGLU clipping not used— |
| gpt-oss-120b used | SwiGLU used— |