Open Models Almanac
Methodsmodel architecturechannel mixer

methods · model architecture

channel mixer

86 methods filed at this node or below it, from the sources of 37 models.

In practice

4 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelMixture of expertsShared expertAuxiliary-loss-free load balancing (selection bias)SwiGLU feed-forward
MiniMax-M3.1-Flashin its codein its codein its codein its code
DeepSeek-V4.1-Flash-1017in its codein its codein its codein its code
Step-5-Previewin its codein its codein its codein its code
Kolibri-1in its codein its codein its codein its code
DeepSeek-V4.1-Flashin its codein its codein its codein its code
Hy4-previewin its codein its codein its codein its code
GLM-5.3-Flashin its codein its codein its codein its code
DeepSeek-V4-Flash-Vision-Expin its codein its codein its codein its code
GLM-5.3in its codein its codein its codein its code
DeepSeek-V4-Pro-0813in its codein its codein its codein its code
DeepSeek-V4-Flash-0731in its codein its codein its codein its code
Laguna-S-2.1in its codein its codein its codein its code
Inklingin its codein its codein its codein its code
Hy3in its codein its codein its codein its code
GLM-5.2in its codein its codein its codein its code
MiniMax-M3in its codein its codein its codein its code
Step-3.7-Flashin its codein its codein its codein its code
DeepSeek-V4-Flashin its codein its codein its codein its code
DeepSeek-V4-Proin its codein its codein its codein its code
DeepSeek-V3.2in its codein its codein its codein its code
Mistral-Large-4in its codein its codenot in its codein its code
MiMo-V2.6-Flashin its codenot in its codein its codein its code
MiMo-V2.6-Proin its codenot in its codein its codein its code
Qwen3.8-Flash-Nextin its codein its codenot in its codein its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codein its codein its codenot in its code
Kimi K3in its codein its codein its codenot in its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codein its codein its codenot in its code
MiMo-V2.5in its codenot in its codein its codein its code
MiMo-V2.5-Proin its codenot in its codein its codein its code
Qwen3.6-35B-A3Bin its codein its codenot in its codein its code
Qwen3.5-397B-A17Bin its codein its codenot in its codein its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codein its codecode unclearnot in its code
gpt-oss-120bin its codenot in its codenot in its codein its code
LFM3-1.7Bnot in its codenot in its codenot in its codein its code
Xiaomi-Robotics-1not in its codenot in its codenot in its codein its code
MolmoAct 2not in its codenot in its codenot in its codein its code
Gemma 4 31Bnot in its codenot in its codenot in its codenot in its code
X-VLAnot in its codenot in its codenot in its codenot in its code
OpenVLA / OpenVLA-OFTnot in its codenot in its codenot in its codecode unclear

In this branch 86

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 3

3 single mentions
Feature-wise Linear Modulation (FiLM) optional · 1 source · 2 quotes
Parallel MLP-Mixer-based action decoder evaluated · 1 source · 1 quote

dense feed-forward network 7

SiTU-GLU core · 2 sources · 3 quotes
SwiGLU used · 2 sources · 2 quotes
SwiGLU clamping used · 2 sources · 2 quotes
4 from one source, or only mentioned as not used
Dense feed-forward network used · 1 source · 1 quote
Sigmoid Tanh Unit (SiTU) used · 1 source · 1 quote
SwiGLU-clip not used · 1 source · 1 quote

mixture of experts 76

Mixture of experts core · 113 sources · 135 quotes
Sparse expert activation core · 9 sources · 9 quotes
Action expert core · 2 sources · 2 quotes
Asymmetric input/output parameter activation core · 2 sources · 2 quotes
DeepSeekMoE core · 2 sources · 2 quotes
12 from one source, or only mentioned as not used
Dispatch recomputation core · 1 source · 1 quote
Gated DeltaNet MoE core · 1 source · 1 quote
Hybrid instruction-and-reasoning MoE core · 1 source · 1 quote
Hybrid Mixture of Experts core · 1 source · 1 quote
MegaMoE used · 1 source · 1 quote
Routed expert modulation used · 1 source · 1 quote
Routed-expert silencing test evaluated · 1 source · 1 quote
MoE layer densification by expert concatenation not used · 1 source · 1 quote

expert routing 15

Top-k expert routing core · 11 sources · 12 quotes
Anticipatory Routing used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Keep Routing core · 1 source · 1 quote
Sigmoid-based expert routing core · 1 source · 1 quote
Token-choice routing with softplus gating core · 1 source · 1 quote
Top-k routing with frozen bias default · 1 source · 1 quote
DP-aware routing used · 1 source · 1 quote
Hash routing used · 1 source · 1 quote
Latent-space routing used · 1 source · 1 quote
Token-choice routing used · 1 source · 1 quote
Loss-spike-triggered Anticipatory Routing optional · 1 source · 1 quote
Random routed-expert selection evaluated · 1 source · 1 quote
Heterogeneity-guided MoE routing not used · 1 source · 1 quote

expert load balancing 25

Auxiliary-loss-free load balancing core · 3 sources · 4 quotes
Quantile Balancing core · 3 sources · 5 quotes
Exact Quantile Balancing core · 2 sources · 3 quotes
Load-error gradient injection core · 2 sources · 3 quotes
Load balancing used · 2 sources · 2 quotes
20 from one source, or only mentioned as not used
Modality-specific load balancing core · 1 source · 2 quotes
Per-expert routing correction bias core · 1 source · 1 quote
Auxiliary-loss load balancing used · 1 source · 1 quote
Disabled router-bias updates used · 1 source · 1 quote
Exact coordinate minimization used · 1 source · 1 quote
Expert bias update factor used · 1 source · 1 quote
Expert Parallelism Load Balancing used · 1 source · 1 quote
Expert-parallel-group load-balancing loss used · 1 source · 1 quote
Histogram-based global quantile estimation used · 1 source · 1 quote
Mixture-of-experts load balancing used · 1 source · 2 quotes
Persistent load balancing used · 1 source · 1 quote
Pooled-global-batch quantile estimation used · 1 source · 1 quote
Recentered global-quantile target update used · 1 source · 1 quote
Two-pass radix selection used · 1 source · 1 quote
Uniform-bin histogram quantile estimation used · 1 source · 1 quote
Round-robin load balancing optional · 1 source · 1 quote
MaxVio evaluated · 1 source · 1 quote
Load-balancing regularization not used · 1 source · 1 quote

shared experts 6

Shared expert core · 13 sources · 13 quotes
MoE without shared experts core · 3 sources · 3 quotes
2 single mentions
MoE with expert sinks core · 1 source · 1 quote
Shared expert isolation core · 1 source · 1 quote

fine-grained experts 8

Expert granularity evaluated · 3 sources · 3 quotes
7 single mentions
Fine-grained expert segmentation core · 1 source · 1 quote
Many small experts used · 1 source · 1 quote
MoE granularity 4 used · 1 source · 1 quote
Coarse-grained MoE configuration evaluated · 1 source · 1 quote
Fine-grained MoE configuration evaluated · 1 source · 1 quote

latent mixture of experts 5

LatentMoE core · 7 sources · 9 quotes
Stable LatentMoE core · 6 sources · 7 quotes
3 single mentions
Hybrid LatentMoE core · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled heredense feed-forward networkmixture of experts
DeepSeek-V4.1-Flash-1017 core——Asymmetric input/output parameter activation coreMixture of experts core—
GLM-5.3-Flash core——Mixture of experts coreTop-k expert routing core—
DeepSeek-V4.1-Flash core——Asymmetric input/output parameter activation coreMixture of experts coreModality-specific load balancing core—
Hy4-preview core——Mixture of experts coreShared expert coreTop-k expert routing core—
Step-5-Preview core——Mixture of experts core—
MiniMax-M3.1-Flash core——Mixture of experts core—
Hy3 core——Mixture of experts coreShared expert coreSparse expert activation coreTop-k expert routing core—
MiMo-V2.6-Flash core——Mixture of experts coreMoE without shared experts core—
NVIDIA-Nemotron-3-Ultra-550B-A55B core——Hybrid LatentMoE coreLatentMoE coreMixture of experts coreExpert Parallelism Load Balancing usedShared expert usedMaxVio evaluatedExpert granularity not used—
DeepSeek-V4-Flash core—SwiGLU usedSwiGLU clamping used—DeepSeekMoE coreMixture of experts coreShared expert coreAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedAnticipatory Routing optionalLoss-spike-triggered Anticipatory Routing optional—
DeepSeek-V3.2 core——Keep Routing core—
DeepSeek-V4-Flash-Vision-Exp core——Mixture of experts core—
DeepSeek-V4-Pro core—SwiGLU usedSwiGLU clamping used—DeepSeekMoE coreMixture of experts coreShared expert coreAnticipatory Routing usedAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedLoss-spike-triggered Anticipatory Routing optional—
GLM-5.2 core——Mixture of experts coreDP-aware routing used—
Gemma 4 31B core——Mixture of experts coreSparse expert activation core—
Inkling core——Mixture of experts coreMoE with expert sinks coreShared expert coreTop-k expert routing coreSigmoid router with auxiliary-loss-free load balancing used—
Kimi K3 core—SiTU-GLU coreSigmoid Tanh Unit (SiTU) used—Auxiliary-loss-free load balancing coreDispatch recomputation coreLatentMoE coreMixture of experts coreOutput-independent MoE gradient reformulation coreQuantile Balancing coreSharded latent weights with fused all-gather GEMM epilogue coreSparse expert activation coreStable LatentMoE coreTop-k routing with frozen bias defaultExact coordinate minimization usedExponential moving average of estimated quantiles usedHistogram-based global quantile estimation usedLatent-space routing usedPooled-global-batch quantile estimation usedUniform-bin histogram quantile estimation used—
Kolibri-1 core—Leading dense feed-forward layers before MoE layers not used—Exact Quantile Balancing coreLoad-error gradient injection coreMixture of experts coreShared expert coreTop-k expert routing coreMany small experts usedMixture-of-experts load balancing usedRecentered global-quantile target update usedTwo-pass radix selection usedCoarse-grained MoE configuration evaluatedExpert granularity evaluatedFine-grained MoE configuration evaluatedRandom routed-expert selection evaluatedRouted-expert silencing test evaluatedMoE layer densification by expert concatenation not used—
LFM3-1.7B core——Mixture of experts coreTop-k routing with normalized sigmoid and adaptive load-balancing bias coreDense-first-layer, sparse-MoE-later-layer placement used—
Laguna-S-2.1 core——Mixture of experts coreSparse expert activation coreToken-choice routing with softplus gating coreAuxiliary-loss load balancing usedRouted expert modulation usedShared expert usedToken-choice routing used—
LingBot-VLA 2.0 core——Auxiliary-loss-free load balancing coreFine-grained expert segmentation coreFine-grained routed experts with a shared expert coreMixture of experts corePer-expert routing correction bias coreShared expert isolation coreSigmoid-based expert routing coreSign-based expert-load correction-bias updates core—
MiMo-V2.5 core—Dense feed-forward network used—Mixture of experts coreExpert bias update factor usedLoad balancing usedTop-k expert routing usedRound-robin load balancing optionalMoE without shared experts not used—
MiMo-V2.5-Pro core——Mixture of experts core—
MiMo-V2.6-Pro core——Mixture of experts coreMoE without shared experts coreSparse expert activation core—
MiniMax-M3 core——Mixture of experts corePersistent load balancing usedShared expert usedTop-k expert routing used—
Mistral-Large-4 core——Granular Mixture-of-Experts with routed and shared experts coreHybrid instruction-and-reasoning MoE coreMixture of experts coreShared expert coreTop-k expert routing coreExpert-parallel-group load-balancing loss usedMoE granularity 4 usedLoad balancing mentioned—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core——LatentMoE coreMixture of experts coreSparse expert activation coreDisabled router-bias updates usedLatent-dimension reduction with increased expert count and active experts used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B core——Mixture of experts coreDisabled router-bias updates used—
OpenVLA / OpenVLA-OFT optionalFeature-wise Linear Modulation (FiLM) optional———
Qwen3.5-397B-A17B core——Hybrid Mixture of Experts coreMixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreMoE with 8 routed and 1 shared expert activated per token coreSparse expert activation core—
Qwen3.6-35B-A3B core——Gated DeltaNet MoE coreMixture of experts core—
Qwen3.8-Flash-Next coreContextual gating for N-gram embedding injection used—SwiGLU-clip not used—Mixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreFrequency-based partitioning of N-gram embedding slots evaluated—
SmolVLA core——Action expert core—
Step-3.7-Flash core——Mixture of experts core—
X-VLA evaluatedParallel MLP-Mixer-based action decoder evaluated——Heterogeneity-guided MoE routing not usedLoad-balancing regularization not used—
gpt-oss-120b core—SwiGLU used—Mixture of experts coreTop-k routing with softmax over selected experts core—
π0.5 core——Action expert core—
↑