methods · mixture of experts
shared experts
6 methods filed at this node or below it, from the sources of 16 models.
In practice
Checked as “Shared expert” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | adopted | n_shared_experts=1MistralLarge4TextMoE adds a shared_experts MLP sized by moe_intermediate_size * n_shared_experts=1 that every token passes through in addition to routed experts. text_config.n_shared_experts = 1 | code → |
| MiniMax-M3.1-Flash | in its code | not stated | n_shared_experts=1 · shared_intermediate_size=3072Every MoE block also runs a dense shared expert (shared_intermediate_size=3072, n_shared_experts=1) whose output is added to the routed-expert output. text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072 | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | n_shared_experts=1n_shared_experts=1 configures a shared expert that every token passes through alongside the routed experts, and the code explicitly loads/pads shared_experts weights. text_config.n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | not stated | n_shared_experts=1 · moe_shared_expert_intermediate_size=5376The text_config declares 1 shared expert with a nonzero intermediate size, applied alongside the routed experts in MoE layers. llm_config.n_shared_experts = 1llm_config.moe_shared_expert_intermediate_size = 5376 | code → |
| Step-5-Preview | in its code | not stated | share_expert_dim=1536Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed output. text_config.share_expert_dim = 1536 | code → |
| Kolibri-1 | in its code | adopted | shared_expert_intermediate_size=512A shared_experts MLP with nonzero intermediate size (512) is added to every token's output alongside routed experts. shared_expert_intermediate_size = 512 | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | n_shared_experts=1MoE layers include n_shared_experts=1 shared expert(s) that every token passes through, loaded/handled via shared_experts weight paths. text_config.n_shared_experts = 1 | code → |
| Hy4-preview | in its code | adopted | n_shared_experts=1HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed experts. n_shared_experts = 1 | code → |
| GLM-5.3-Flash | in its code | not stated | n_shared_experts=1Glm5NextTextMoE includes a shared_experts MLP (sized by n_shared_experts=1) added to every token's output alongside the routed experts. text_config.n_shared_experts = 1 | code → |
| Qwen3.8-Flash-Next | in its code | not stated | shared_expert_intermediate_size=640Each MoE block has a shared_expert MLP with nonzero shared_expert_intermediate_size applied to every token and gated before being added to routed expert output. text_config.shared_expert_intermediate_size = 640 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | n_shared_experts=1Each MoE block adds a shared DeepseekV4MLP that every token passes through in addition to the routed experts, with n_shared_experts=1. n_shared_experts = 1 | code → |
| GLM-5.3 | in its code | not stated | n_shared_experts=1GlmMoeDsaMoE adds a shared_experts MLP sized by n_shared_experts=1 that every token passes through in addition to routed experts. n_shared_experts = 1 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | n_shared_experts=1Each MoE block also runs a DeepseekV4MLP shared expert that every token passes through, added to the routed output, with n_shared_experts=1 n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | n_shared_experts=1 · moe_shared_expert_intermediate_size=3712Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with n_shared_experts=1. n_shared_experts = 1moe_shared_expert_intermediate_size = 3712 | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | n_shared_experts=1Each MoE block adds the output of a dense shared-expert MLP to every token's routed output, and n_shared_experts=1 in this config. n_shared_experts = 1 | code → |
| Laguna-S-2.1 | in its code | adopted | shared_expert_intermediate_size=1024Each sparse MoE block includes a shared_experts LagunaMLP with intermediate size shared_expert_intermediate_size=1024 added to every token's output. shared_expert_intermediate_size = 1024 | code → |
| Kimi K3 | in its code | not stated | num_shared_experts=2KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts. text_config.num_shared_experts = 2 | code → |
| Inkling | in its code | adopted | n_shared_experts=2 · shared_expert_sink=trueMoE layers include 2 shared 'sink' experts that every token passes through in addition to the routed experts, enabled by n_shared_experts=2 and shared_expert_sink=true. text_config.n_shared_experts = 2text_config.shared_expert_sink = true | code → |
| Hy3 | in its code | adopted | num_shared_experts=1Each MoE layer includes a shared HYV3MLP expert sized by num_shared_experts=1 that every token passes through and adds to the routed output. num_shared_experts = 1 | code → |
| GLM-5.2 | in its code | not stated | n_shared_experts=1GlmMoeDsaMoE adds a shared_experts MLP (sized by n_shared_experts=1) that every token passes through in addition to the routed experts. n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | adopted | n_shared_experts=1 · moe_shared_expert_intermediate_size=10240Every MoE layer additionally runs a shared expert MLP of intermediate size 10240 that all tokens pass through, added to the routed-expert output. n_shared_experts = 1moe_shared_expert_intermediate_size = 10240 | code → |
| MiniMax-M3 | in its code | adopted | n_shared_experts=1 · shared_intermediate_size=3072Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed experts' output. text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072 | code → |
| Step-3.7-Flash | in its code | not stated | share_expert_dim=1280Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally to the routed output. text_config.share_expert_dim = 1280 | code → |
| DeepSeek-V4-Flash | in its code | adopted | n_shared_experts=1Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config confirms it's enabled. n_shared_experts = 1 | code → |
| DeepSeek-V4-Pro | in its code | adopted | n_shared_experts=1Every MoE layer adds one shared DeepseekV4MLP expert that all tokens pass through, with n_shared_experts=1. n_shared_experts = 1 | code → |
| Qwen3.6-35B-A3B | in its code | not stated | shared_expert_intermediate_size=512Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated and added to the routed output. text_config.shared_expert_intermediate_size = 512 | code → |
| Qwen3.5-397B-A17B | in its code | not stated | shared_expert_intermediate_size=1024Each MoE block also includes a shared_expert MLP (sized by shared_expert_intermediate_size=1024) applied to every token and gated, added to the routed-expert output. text_config.shared_expert_intermediate_size = 1024 | code → |
| DeepSeek-V3.2 | in its code | not stated | n_shared_experts=1DeepseekV32MoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) applied to every token in addition to routed experts. n_shared_experts = 1 | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesThere is no MoE mechanism at all, so there is no shared-expert path either. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in this config, and MiMoV2MoE builds no additional always-on expert module. n_shared_experts = null | code → |
| MiMo-V2.6-Pro | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in the config and no shared-expert module is instantiated in MiMoV2MoE. n_shared_experts = null | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesThere is no MoE structure at all, so no shared-expert mechanism or related config keys exist. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesThere is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key exists. | code → |
| MiMo-V2.5 | not in its code | not stated | n_shared_experts=nulln_shared_experts is null and MiMoV2MoE has no shared-expert module in the code. n_shared_experts = null | code → |
| MiMo-V2.5-Pro | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in this config and no shared-expert module is instantiated in MiMoV2MoE. n_shared_experts = null | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesThere is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is disabled. | code → |
| X-VLA | not in its code | not stated | verdict and linesThere is no MoE structure at all, so no shared-expert mechanism exists in this model. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of selected experts | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo shared-expert config keys (n_shared_experts, shared_expert_intermediate_size) or code are present, consistent with there being no MoE at all. | code → |
In this branch 6
Everything filed at this node or below it, with one collapsible heading per child node.
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| Hy4-preview core | Shared expert core— |
| Hy3 core | Shared expert core— |
| MiMo-V2.6-Flash core | MoE without shared experts core— |
| NVIDIA-Nemotron-3-Ultra-550B-A55B used | Shared expert used— |
| DeepSeek-V4-Flash core | Shared expert core— |
| DeepSeek-V4-Pro core | Shared expert core— |
| Inkling core | MoE with expert sinks coreShared expert core— |
| Kolibri-1 core | Shared expert core— |
| Laguna-S-2.1 used | Shared expert used— |
| LingBot-VLA 2.0 core | Shared expert isolation core— |
| MiMo-V2.5 not used | MoE without shared experts not used— |
| MiMo-V2.6-Pro core | MoE without shared experts core— |
| MiniMax-M3 used | Shared expert used— |
| Mistral-Large-4 core | Shared expert core— |
| Qwen3.5-397B-A17B core | MoE with 10 routed and 1 shared expert activated per token coreMoE with 8 routed and 1 shared expert activated per token core— |
| Qwen3.8-Flash-Next core | MoE with 10 routed and 1 shared expert activated per token core— |