general family · filed under model architecture
Shared Experts
Experts in a mixture-of-experts layer that are shared rather than selected from the routed experts; the evidence includes configurations with shared experts processing every token.
Also called Shared Expert, 1 shared expert, Shared expert in MoE, shared expert plus routed experts in MoE, 1 shared expert and 384 routed experts.
- sources
- 7
- models
- 8
- labs adopt it
- 5
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 4core 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (Dai et al., 2024) paper arxiv.orgintroduces shared-expert isolation
In the code
Checked as “Shared expert” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | Glm5NextTextMoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) that every token passes through in addition to routed experts. modeling_glm5_next.py#L197 |
| DeepSeek-V4.1-Flash | in its code | used | Each MoE layer also has n_shared_experts=1 shared expert(s) that all tokens pass through in addition to the routed experts. model.py#L1051 |
| Hy4-preview | in its code | core | HYV4MoE adds a shared_experts HYV4MLP with intermediate size scaled by n_shared_experts=1, applied to every token in addition to the routed experts. modeling_hy_v4.py#L600 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | Each MoE block includes a DeepseekV4MLP shared_experts module (n_shared_experts=1) added unconditionally to the routed output. modeling_deepseek_v4.py#L1129 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | used | Each MoE block includes a shared expert MLP (NemotronHMLP with moe_shared_expert_intermediate_size=10240, n_shared_experts=1) applied to every token in addition to routed experts. modeling_nemotron_h.py#L701 |
| MiMo-V2.6-Flash | not in its code | not stated | n_shared_experts is null in this config, and MiMoV2MoE builds no additional always-on expert module. |
| DeepSeek-V4-Flash | in its code | core | Each SparseMoeBlock adds a DeepseekV4MLP shared expert output to every token's routed output, and n_shared_experts is 1 (nonzero). modeling_deepseek_v4.py#L1139 |
| MiMo-V2.5 | not in its code | not stated | n_shared_experts is null and MiMoV2MoE has no shared-expert module in the code. |
| GLM-5.3 | in its code | not stated | Each MoE layer adds a shared_experts MLP (sized by n_shared_experts=1) that every token passes through in addition to routed experts. modeling_glm_moe_dsa.py#L572 |
| Hy3 | in its code | core | Each MoE layer includes a shared expert MLP with nonzero intermediate size (num_shared_experts=1) added to every token's output modeling_hy_v3.py#L358 |
| GLM-5.2 | in its code | not stated | GlmMoeDsaMoE includes a shared_experts GlmMoeDsaMLP with intermediate size scaled by n_shared_experts=1, applied to every token in addition to routed experts. modeling_glm_moe_dsa.py#L572 |
| MiniMax-M3 | in its code | used | Each MoE block also runs a dense shared expert (shared_intermediate_size, n_shared_experts=1) whose output is added to the routed-expert output. modeling_minimax_m3_vl.py#L255 |
| DeepSeek-V3.2 | in its code | not stated | DeepseekV32MoE adds a shared_experts MLP sized by n_shared_experts=1 that every token passes through in addition to routed experts. modeling_deepseek_v32.py#L601 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | Every MoE block runs a shared DeepseekV4MLP that every token passes through in addition to the routed experts, and n_shared_experts=1 in this config. modeling_deepseek_v4.py#L1129 |
| DeepSeek-V4-Pro | in its code | core | n_shared_experts is 1 and every MoE block adds a DeepseekV4MLP shared expert output to the routed output for every token modeling_deepseek_v4.py#L1139 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | Every SparseMoeBlock adds a DeepseekV4MLP shared_experts pass that every token goes through, and n_shared_experts=1 in config. modeling_deepseek_v4.py#L1139 |
| Gemma 4 31B | not in its code | not stated | No shared-expert mechanism (always-applied expert MLP) exists in the code; only routed experts (Gemma4TextExperts) are implemented, and MoE itself is disabled anyway. |
| Inkling | in its code | not stated | MoE layers include 2 shared 'sink' experts that every token passes through, implemented in InklingSinkExperts and added to the routed output. moe.py#L466 |
| Kimi K3 | in its code | not stated | KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts. modeling_kimi_linear.py#L797 |
| Laguna-S-2.1 | in its code | used | Each sparse MoE block includes a shared_experts MLP with nonzero intermediate size 1024 that every token passes through, added to the routed output. modeling_laguna.py#L234 |
| MiMo-V2.5-Pro | not in its code | not stated | n_shared_experts is null in this config and no shared-expert module is instantiated in MiMoV2MoE. |
| MiMo-V2.6-Pro | not in its code | not stated | n_shared_experts is null in the config and no shared-expert module is instantiated in MiMoV2MoE. |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | NemotronHMoE adds a shared_experts NemotronHMLP with moe_shared_expert_intermediate_size=3712 whose output is added to every token unconditionally, and n_shared_experts=1. modeling_nemotron_h.py#L701 |
| Qwen3.5-397B-A17B | in its code | not stated | A shared_expert MLP with nonzero intermediate size processes every token and is gated and added to the routed expert output. modeling_qwen3_5_moe.py#L909 |
| Qwen3.6-35B-A3B | in its code | not stated | Each MoE block also has a shared_expert MLP with nonzero shared_expert_intermediate_size, gated by a sigmoid and added to the routed expert output for every token. modeling_qwen3_5_moe.py#L909 |
| Qwen3.8-Flash-Next | in its code | not stated | Each MoE block also runs a shared_expert MLP (sized by shared_expert_intermediate_size=640) through every token, gated and added to the routed output. modeling_qwen4_exp.py#L996 |
| Step-3.7-Flash | in its code | not stated | Each MoE block includes a Step3p7MLP shared expert with nonzero share_expert_dim (1280) added unconditionally to the routed output. modeling_step3p7.py#L620 |
| gpt-oss-120b | not in its code | not stated | There is no shared-expert module in GptOssMLP or its config; only routed experts exist. |
Evidence
7 spans quoted from the sources, strongest treatment first.
each containing 256 routed experts and 1 shared expert
coremodel architecturein Hy4-previewTencent Hunyuan
sets fine-grained routed experts and shared experts
coremodel architecturein DeepSeek-V4DeepSeek
192 routed experts (top-8) + 1 shared expert
coremodel architecturein Hy3Tencent
Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.
usedmodel architecturein MiniMax-M3MiniMax
Shared Expert Intermediate Size 10240
usedmodel architecturein Nemotron 3 UltraNVIDIA
Each MoE layer consists of 1 shared expert and 384 routed experts, where the intermediate hidden dimension of each expert is 2304.
usedunclearin DeepSeek-V4.1-FlashDeepSeek
plus a shared expert that processes every token
usedmodel architecturein Laguna XS.2Poolside
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: shared experts.