methods · model architecture
channel mixer
86 methods filed at this node or below it, from the sources of 37 models.
In practice
4 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.
adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
| Model | Mixture of experts | Shared expert | Auxiliary-loss-free load balancing (selection bias) | SwiGLU feed-forward |
|---|---|---|---|---|
| MiniMax-M3.1-Flash | in its code | in its code | in its code | in its code |
| DeepSeek-V4.1-Flash-1017 | in its code | in its code | in its code | in its code |
| Step-5-Preview | in its code | in its code | in its code | in its code |
| Kolibri-1 | in its code | in its code | in its code | in its code |
| DeepSeek-V4.1-Flash | in its code | in its code | in its code | in its code |
| Hy4-preview | in its code | in its code | in its code | in its code |
| GLM-5.3-Flash | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Flash-Vision-Exp | in its code | in its code | in its code | in its code |
| GLM-5.3 | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Pro-0813 | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Flash-0731 | in its code | in its code | in its code | in its code |
| Laguna-S-2.1 | in its code | in its code | in its code | in its code |
| Inkling | in its code | in its code | in its code | in its code |
| Hy3 | in its code | in its code | in its code | in its code |
| GLM-5.2 | in its code | in its code | in its code | in its code |
| MiniMax-M3 | in its code | in its code | in its code | in its code |
| Step-3.7-Flash | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Flash | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Pro | in its code | in its code | in its code | in its code |
| DeepSeek-V3.2 | in its code | in its code | in its code | in its code |
| Mistral-Large-4 | in its code | in its code | not in its code | in its code |
| MiMo-V2.6-Flash | in its code | not in its code | in its code | in its code |
| MiMo-V2.6-Pro | in its code | not in its code | in its code | in its code |
| Qwen3.8-Flash-Next | in its code | in its code | not in its code | in its code |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | in its code | in its code | not in its code |
| Kimi K3 | in its code | in its code | in its code | not in its code |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | in its code | in its code | not in its code |
| MiMo-V2.5 | in its code | not in its code | in its code | in its code |
| MiMo-V2.5-Pro | in its code | not in its code | in its code | in its code |
| Qwen3.6-35B-A3B | in its code | in its code | not in its code | in its code |
| Qwen3.5-397B-A17B | in its code | in its code | not in its code | in its code |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | in its code | code unclear | not in its code |
| gpt-oss-120b | in its code | not in its code | not in its code | in its code |
| LFM3-1.7B | not in its code | not in its code | not in its code | in its code |
| Xiaomi-Robotics-1 | not in its code | not in its code | not in its code | in its code |
| MolmoAct 2 | not in its code | not in its code | not in its code | in its code |
| Gemma 4 31B | not in its code | not in its code | not in its code | not in its code |
| X-VLA | not in its code | not in its code | not in its code | not in its code |
| OpenVLA / OpenVLA-OFT | not in its code | not in its code | not in its code | code unclear |
In this branch 86
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 3
dense feed-forward network 7
4 from one source, or only mentioned as not used
mixture of experts 76
12 from one source, or only mentioned as not used
expert routing 15
13 from one source, or only mentioned as not used
expert load balancing 25
20 from one source, or only mentioned as not used
fine-grained experts 8
latent mixture of experts 5
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.