Mistral-Large-4
Mistral AIOne shared expert always runs alongside the selected experts, for a total of 5 active experts per token.
2 more from this model
and 1 shared expert
128 routed experts (top-4) plus one shared expert
general family · filed under model architecture
A shared expert is an expert in a mixture-of-experts layer that is evaluated for every token.1 It runs alongside routed experts, of which only a few are selected per token, and its output is added to the mixture of the selected routed experts.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
a shared expert, evaluated for every token— tech report
yt = fshared(xt) + ∑︂ e∈It St,efe(xt— tech report
Also called 1 shared expert, one shared expert, Shared expert in mixture-of-experts layers, Shared expert in MoE, shared expert plus routed experts in MoE, 1 shared expert and 384 routed experts.
Checked as “Shared expert” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | adopted | n_shared_experts=1MistralLarge4TextMoE adds a shared_experts MLP sized by moe_intermediate_size * n_shared_experts=1 that every token passes through in addition to routed experts. text_config.n_shared_experts = 1 | code → |
| MiniMax-M3.1-Flash | in its code | not stated | n_shared_experts=1 · shared_intermediate_size=3072Every MoE block also runs a dense shared expert (shared_intermediate_size=3072, n_shared_experts=1) whose output is added to the routed-expert output. text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072 | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | n_shared_experts=1n_shared_experts=1 configures a shared expert that every token passes through alongside the routed experts, and the code explicitly loads/pads shared_experts weights. text_config.n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | not stated | n_shared_experts=1 · moe_shared_expert_intermediate_size=5376The text_config declares 1 shared expert with a nonzero intermediate size, applied alongside the routed experts in MoE layers. llm_config.n_shared_experts = 1llm_config.moe_shared_expert_intermediate_size = 5376 | code → |
| Step-5-Preview | in its code | not stated | share_expert_dim=1536Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed output. text_config.share_expert_dim = 1536 | code → |
| Kolibri-1 | in its code | adopted | shared_expert_intermediate_size=512A shared_experts MLP with nonzero intermediate size (512) is added to every token's output alongside routed experts. shared_expert_intermediate_size = 512 | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | n_shared_experts=1MoE layers include n_shared_experts=1 shared expert(s) that every token passes through, loaded/handled via shared_experts weight paths. text_config.n_shared_experts = 1 | code → |
| Hy4-preview | in its code | adopted | n_shared_experts=1HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed experts. n_shared_experts = 1 | code → |
| GLM-5.3-Flash | in its code | not stated | n_shared_experts=1Glm5NextTextMoE includes a shared_experts MLP (sized by n_shared_experts=1) added to every token's output alongside the routed experts. text_config.n_shared_experts = 1 | code → |
| Qwen3.8-Flash-Next | in its code | not stated | shared_expert_intermediate_size=640Each MoE block has a shared_expert MLP with nonzero shared_expert_intermediate_size applied to every token and gated before being added to routed expert output. text_config.shared_expert_intermediate_size = 640 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | n_shared_experts=1Each MoE block adds a shared DeepseekV4MLP that every token passes through in addition to the routed experts, with n_shared_experts=1. n_shared_experts = 1 | code → |
| GLM-5.3 | in its code | not stated | n_shared_experts=1GlmMoeDsaMoE adds a shared_experts MLP sized by n_shared_experts=1 that every token passes through in addition to routed experts. n_shared_experts = 1 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | n_shared_experts=1Each MoE block also runs a DeepseekV4MLP shared expert that every token passes through, added to the routed output, with n_shared_experts=1 n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | n_shared_experts=1 · moe_shared_expert_intermediate_size=3712Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with n_shared_experts=1. n_shared_experts = 1moe_shared_expert_intermediate_size = 3712 | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | n_shared_experts=1Each MoE block adds the output of a dense shared-expert MLP to every token's routed output, and n_shared_experts=1 in this config. n_shared_experts = 1 | code → |
| Laguna-S-2.1 | in its code | adopted | shared_expert_intermediate_size=1024Each sparse MoE block includes a shared_experts LagunaMLP with intermediate size shared_expert_intermediate_size=1024 added to every token's output. shared_expert_intermediate_size = 1024 | code → |
| Kimi K3 | in its code | not stated | num_shared_experts=2KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts. text_config.num_shared_experts = 2 | code → |
| Inkling | in its code | adopted | n_shared_experts=2 · shared_expert_sink=trueMoE layers include 2 shared 'sink' experts that every token passes through in addition to the routed experts, enabled by n_shared_experts=2 and shared_expert_sink=true. text_config.n_shared_experts = 2text_config.shared_expert_sink = true | code → |
| Hy3 | in its code | adopted | num_shared_experts=1Each MoE layer includes a shared HYV3MLP expert sized by num_shared_experts=1 that every token passes through and adds to the routed output. num_shared_experts = 1 | code → |
| GLM-5.2 | in its code | not stated | n_shared_experts=1GlmMoeDsaMoE adds a shared_experts MLP (sized by n_shared_experts=1) that every token passes through in addition to the routed experts. n_shared_experts = 1 | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | adopted | n_shared_experts=1 · moe_shared_expert_intermediate_size=10240Every MoE layer additionally runs a shared expert MLP of intermediate size 10240 that all tokens pass through, added to the routed-expert output. n_shared_experts = 1moe_shared_expert_intermediate_size = 10240 | code → |
| MiniMax-M3 | in its code | adopted | n_shared_experts=1 · shared_intermediate_size=3072Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed experts' output. text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072 | code → |
| Step-3.7-Flash | in its code | not stated | share_expert_dim=1280Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally to the routed output. text_config.share_expert_dim = 1280 | code → |
| DeepSeek-V4-Flash | in its code | adopted | n_shared_experts=1Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config confirms it's enabled. n_shared_experts = 1 | code → |
| DeepSeek-V4-Pro | in its code | adopted | n_shared_experts=1Every MoE layer adds one shared DeepseekV4MLP expert that all tokens pass through, with n_shared_experts=1. n_shared_experts = 1 | code → |
| Qwen3.6-35B-A3B | in its code | not stated | shared_expert_intermediate_size=512Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated and added to the routed output. text_config.shared_expert_intermediate_size = 512 | code → |
| Qwen3.5-397B-A17B | in its code | not stated | shared_expert_intermediate_size=1024Each MoE block also includes a shared_expert MLP (sized by shared_expert_intermediate_size=1024) applied to every token and gated, added to the routed-expert output. text_config.shared_expert_intermediate_size = 1024 | code → |
| DeepSeek-V3.2 | in its code | not stated | n_shared_experts=1DeepseekV32MoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) applied to every token in addition to routed experts. n_shared_experts = 1 | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesThere is no MoE mechanism at all, so there is no shared-expert path either. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in this config, and MiMoV2MoE builds no additional always-on expert module. n_shared_experts = null | code → |
| MiMo-V2.6-Pro | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in the config and no shared-expert module is instantiated in MiMoV2MoE. n_shared_experts = null | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesThere is no MoE structure at all, so no shared-expert mechanism or related config keys exist. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesThere is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key exists. | code → |
| MiMo-V2.5 | not in its code | not stated | n_shared_experts=nulln_shared_experts is null and MiMoV2MoE has no shared-expert module in the code. n_shared_experts = null | code → |
| MiMo-V2.5-Pro | not in its code | not stated | n_shared_experts=nulln_shared_experts is null in this config and no shared-expert module is instantiated in MiMoV2MoE. n_shared_experts = null | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesThere is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is disabled. | code → |
| X-VLA | not in its code | not stated | verdict and linesThere is no MoE structure at all, so no shared-expert mechanism exists in this model. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of selected experts | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo shared-expert config keys (n_shared_experts, shared_expert_intermediate_size) or code are present, consistent with there being no MoE at all. | code → |
13 quotes from 13 sources, one card per model, each opening with its strongest.
One shared expert always runs alongside the selected experts, for a total of 5 active experts per token.
and 1 shared expert
128 routed experts (top-4) plus one shared expert
each containing 256 routed experts and 1 shared expert
every token activates the top-8 routed experts along with the shared expert
each token is routed to 6 of 256 experts, plus 2 shared experts active on every token
plus 2 shared experts active on every token
192 routed experts (top-8) + 1 shared expert
sets fine-grained routed experts and shared experts
plus a shared expert that processes every token
Shared Expert Intermediate Size 10240
Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.
Picked by hand, not extracted: where to read more, not evidence for anything on this page.