Open Models Almanac
Methodsmodel architecturechannel mixermixture of experts

methods · channel mixer

mixture of experts

76 methods filed at this node or below it, from the sources of 36 models.

In practice

3 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelMixture of expertsShared expertAuxiliary-loss-free load balancing (selection bias)
MiniMax-M3.1-Flashin its codein its codein its code
DeepSeek-V4.1-Flash-1017in its codein its codein its code
Step-5-Previewin its codein its codein its code
Kolibri-1in its codein its codein its code
DeepSeek-V4.1-Flashin its codein its codein its code
Hy4-previewin its codein its codein its code
GLM-5.3-Flashin its codein its codein its code
DeepSeek-V4-Flash-Vision-Expin its codein its codein its code
GLM-5.3in its codein its codein its code
DeepSeek-V4-Pro-0813in its codein its codein its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codein its codein its code
DeepSeek-V4-Flash-0731in its codein its codein its code
Laguna-S-2.1in its codein its codein its code
Kimi K3in its codein its codein its code
Inklingin its codein its codein its code
Hy3in its codein its codein its code
GLM-5.2in its codein its codein its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codein its codein its code
MiniMax-M3in its codein its codein its code
Step-3.7-Flashin its codein its codein its code
DeepSeek-V4-Flashin its codein its codein its code
DeepSeek-V4-Proin its codein its codein its code
DeepSeek-V3.2in its codein its codein its code
Mistral-Large-4in its codein its codenot in its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codein its codecode unclear
MiMo-V2.6-Flashin its codenot in its codein its code
MiMo-V2.6-Proin its codenot in its codein its code
Qwen3.8-Flash-Nextin its codein its codenot in its code
MiMo-V2.5in its codenot in its codein its code
MiMo-V2.5-Proin its codenot in its codein its code
Qwen3.6-35B-A3Bin its codein its codenot in its code
Qwen3.5-397B-A17Bin its codein its codenot in its code
gpt-oss-120bin its codenot in its codenot in its code
LFM3-1.7Bnot in its codenot in its codenot in its code
Xiaomi-Robotics-1not in its codenot in its codenot in its code
MolmoAct 2not in its codenot in its codenot in its code
Gemma 4 31Bnot in its codenot in its codenot in its code
X-VLAnot in its codenot in its codenot in its code
OpenVLA / OpenVLA-OFTnot in its codenot in its codenot in its code

In this branch 76

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 17

Mixture of experts core · 113 sources · 135 quotes
Sparse expert activation core · 9 sources · 9 quotes
Action expert core · 2 sources · 2 quotes
Asymmetric input/output parameter activation core · 2 sources · 2 quotes
DeepSeekMoE core · 2 sources · 2 quotes
12 from one source, or only mentioned as not used
Dispatch recomputation core · 1 source · 1 quote
Gated DeltaNet MoE core · 1 source · 1 quote
Hybrid instruction-and-reasoning MoE core · 1 source · 1 quote
Hybrid Mixture of Experts core · 1 source · 1 quote
MegaMoE used · 1 source · 1 quote
Routed expert modulation used · 1 source · 1 quote
Routed-expert silencing test evaluated · 1 source · 1 quote
MoE layer densification by expert concatenation not used · 1 source · 1 quote

expert routing 15

Top-k expert routing core · 11 sources · 12 quotes
Anticipatory Routing used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Keep Routing core · 1 source · 1 quote
Sigmoid-based expert routing core · 1 source · 1 quote
Token-choice routing with softplus gating core · 1 source · 1 quote
Top-k routing with frozen bias default · 1 source · 1 quote
DP-aware routing used · 1 source · 1 quote
Hash routing used · 1 source · 1 quote
Latent-space routing used · 1 source · 1 quote
Token-choice routing used · 1 source · 1 quote
Loss-spike-triggered Anticipatory Routing optional · 1 source · 1 quote
Random routed-expert selection evaluated · 1 source · 1 quote
Heterogeneity-guided MoE routing not used · 1 source · 1 quote

expert load balancing 25

Auxiliary-loss-free load balancing core · 3 sources · 4 quotes
Quantile Balancing core · 3 sources · 5 quotes
Exact Quantile Balancing core · 2 sources · 3 quotes
Load-error gradient injection core · 2 sources · 3 quotes
Load balancing used · 2 sources · 2 quotes
20 from one source, or only mentioned as not used
Modality-specific load balancing core · 1 source · 2 quotes
Per-expert routing correction bias core · 1 source · 1 quote
Auxiliary-loss load balancing used · 1 source · 1 quote
Disabled router-bias updates used · 1 source · 1 quote
Exact coordinate minimization used · 1 source · 1 quote
Expert bias update factor used · 1 source · 1 quote
Expert Parallelism Load Balancing used · 1 source · 1 quote
Expert-parallel-group load-balancing loss used · 1 source · 1 quote
Histogram-based global quantile estimation used · 1 source · 1 quote
Mixture-of-experts load balancing used · 1 source · 2 quotes
Persistent load balancing used · 1 source · 1 quote
Pooled-global-batch quantile estimation used · 1 source · 1 quote
Recentered global-quantile target update used · 1 source · 1 quote
Two-pass radix selection used · 1 source · 1 quote
Uniform-bin histogram quantile estimation used · 1 source · 1 quote
Round-robin load balancing optional · 1 source · 1 quote
MaxVio evaluated · 1 source · 1 quote
Load-balancing regularization not used · 1 source · 1 quote

shared experts 6

Shared expert core · 13 sources · 13 quotes
MoE without shared experts core · 3 sources · 3 quotes
2 single mentions
MoE with expert sinks core · 1 source · 1 quote
Shared expert isolation core · 1 source · 1 quote

fine-grained experts 8

Expert granularity evaluated · 3 sources · 3 quotes
7 single mentions
Fine-grained expert segmentation core · 1 source · 1 quote
Many small experts used · 1 source · 1 quote
MoE granularity 4 used · 1 source · 1 quote
Coarse-grained MoE configuration evaluated · 1 source · 1 quote
Fine-grained MoE configuration evaluated · 1 source · 1 quote

latent mixture of experts 5

LatentMoE core · 7 sources · 9 quotes
Stable LatentMoE core · 6 sources · 7 quotes
3 single mentions
Hybrid LatentMoE core · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled hereexpert routingexpert load balancingshared expertsfine-grained expertslatent mixture of experts
DeepSeek-V4.1-Flash-1017 coreAsymmetric input/output parameter activation coreMixture of experts core——————
GLM-5.3-Flash coreMixture of experts core—Top-k expert routing core—————
DeepSeek-V4.1-Flash coreAsymmetric input/output parameter activation coreMixture of experts core——Modality-specific load balancing core————
Hy4-preview coreMixture of experts core—Top-k expert routing core——Shared expert core———
Step-5-Preview coreMixture of experts core——————
MiniMax-M3.1-Flash coreMixture of experts core——————
Hy3 coreMixture of experts coreSparse expert activation core—Top-k expert routing core——Shared expert core———
MiMo-V2.6-Flash coreMixture of experts core———MoE without shared experts core———
NVIDIA-Nemotron-3-Ultra-550B-A55B coreMixture of experts core——Expert Parallelism Load Balancing usedMaxVio evaluated—Shared expert used—Expert granularity not used—Hybrid LatentMoE coreLatentMoE core—
DeepSeek-V4-Flash coreDeepSeekMoE coreMixture of experts coreMegaMoE used—Hash routing usedAnticipatory Routing optionalLoss-spike-triggered Anticipatory Routing optional—Auxiliary-loss-free load balancing used—Shared expert core———
DeepSeek-V3.2 core—Keep Routing core—————
DeepSeek-V4-Flash-Vision-Exp coreMixture of experts core——————
DeepSeek-V4-Pro coreDeepSeekMoE coreMixture of experts coreMegaMoE used—Anticipatory Routing usedHash routing usedLoss-spike-triggered Anticipatory Routing optional—Auxiliary-loss-free load balancing used—Shared expert core———
GLM-5.2 coreMixture of experts core—DP-aware routing used—————
Gemma 4 31B coreMixture of experts coreSparse expert activation core——————
Inkling coreMixture of experts coreSigmoid router with auxiliary-loss-free load balancing used—Top-k expert routing core——MoE with expert sinks coreShared expert core———
Kimi K3 coreDispatch recomputation coreMixture of experts coreOutput-independent MoE gradient reformulation coreSparse expert activation core—Top-k routing with frozen bias defaultLatent-space routing used—Auxiliary-loss-free load balancing coreQuantile Balancing coreExact coordinate minimization usedExponential moving average of estimated quantiles usedHistogram-based global quantile estimation usedPooled-global-batch quantile estimation usedUniform-bin histogram quantile estimation used———LatentMoE coreSharded latent weights with fused all-gather GEMM epilogue coreStable LatentMoE core—
Kolibri-1 coreMixture of experts coreRouted-expert silencing test evaluatedMoE layer densification by expert concatenation not used—Top-k expert routing coreRandom routed-expert selection evaluated—Exact Quantile Balancing coreLoad-error gradient injection coreMixture-of-experts load balancing usedRecentered global-quantile target update usedTwo-pass radix selection used—Shared expert core—Many small experts usedCoarse-grained MoE configuration evaluatedExpert granularity evaluatedFine-grained MoE configuration evaluated——
LFM3-1.7B coreMixture of experts coreDense-first-layer, sparse-MoE-later-layer placement used—Top-k routing with normalized sigmoid and adaptive load-balancing bias core—————
Laguna-S-2.1 coreMixture of experts coreSparse expert activation coreRouted expert modulation used—Token-choice routing with softplus gating coreToken-choice routing used—Auxiliary-loss load balancing used—Shared expert used———
LingBot-VLA 2.0 coreMixture of experts core—Sigmoid-based expert routing core—Auxiliary-loss-free load balancing corePer-expert routing correction bias coreSign-based expert-load correction-bias updates core—Shared expert isolation core—Fine-grained expert segmentation coreFine-grained routed experts with a shared expert core——
MiMo-V2.5 coreMixture of experts core—Top-k expert routing used—Expert bias update factor usedLoad balancing usedRound-robin load balancing optional—MoE without shared experts not used———
MiMo-V2.5-Pro coreMixture of experts core——————
MiMo-V2.6-Pro coreMixture of experts coreSparse expert activation core———MoE without shared experts core———
MiniMax-M3 coreMixture of experts core—Top-k expert routing used—Persistent load balancing used—Shared expert used———
Mistral-Large-4 coreHybrid instruction-and-reasoning MoE coreMixture of experts core—Top-k expert routing core—Expert-parallel-group load-balancing loss usedLoad balancing mentioned—Shared expert core—Granular Mixture-of-Experts with routed and shared experts coreMoE granularity 4 used——
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreMixture of experts coreSparse expert activation core——Disabled router-bias updates used———LatentMoE coreLatent-dimension reduction with increased expert count and active experts used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreMixture of experts core——Disabled router-bias updates used————
Qwen3.5-397B-A17B coreHybrid Mixture of Experts coreMixture of experts coreSparse expert activation core———MoE with 10 routed and 1 shared expert activated per token coreMoE with 8 routed and 1 shared expert activated per token core———
Qwen3.6-35B-A3B coreGated DeltaNet MoE coreMixture of experts core——————
Qwen3.8-Flash-Next coreMixture of experts coreFrequency-based partitioning of N-gram embedding slots evaluated———MoE with 10 routed and 1 shared expert activated per token core———
SmolVLA coreAction expert core——————
Step-3.7-Flash coreMixture of experts core——————
X-VLA not used—Heterogeneity-guided MoE routing not used—Load-balancing regularization not used————
gpt-oss-120b coreMixture of experts core—Top-k routing with softmax over selected experts core—————
π0.5 coreAction expert core——————
↑