Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsshared experts

general family · filed under model architecture

Shared expert

A shared expert is an expert in a mixture-of-experts layer that is evaluated for every token.1 It runs alongside routed experts, of which only a few are selected per token, and its output is added to the mixture of the selected routed experts.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. a shared expert, evaluated for every token — tech report
  2. yt = fshared(xt) + ∑︂ e∈It St,efe(xt — tech report

Also called 1 shared expert, one shared expert, Shared expert in mixture-of-experts layers, Shared expert in MoE, shared expert plus routed experts in MoE, 1 shared expert and 384 routed experts.

sources
13
models
10
labs adopt it
8
strongest
core

In practice

Checked as “Shared expert” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codeadopted
n_shared_experts=1

MistralLarge4TextMoE adds a shared_experts MLP sized by moe_intermediate_size * n_shared_experts=1 that every token passes through in addition to routed experts.

text_config.n_shared_experts = 1
self.shared_experts = MistralLarge4TextMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_mistral_large4.py · L601–603
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_mistral_large4.py · L611
MiniMax-M3.1-Flashin its codenot stated
n_shared_experts=1 · shared_intermediate_size=3072

Every MoE block also runs a dense shared expert (shared_intermediate_size=3072, n_shared_experts=1) whose output is added to the routed-expert output.

text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072
self.shared_experts = MiniMaxM3VLDenseMLP(config, intermediate_size=config.shared_intermediate_size)
modeling_minimax_m3_vl.py · L248
hidden_states = hidden_states * self.routed_scaling_factor
        hidden_states = hidden_states + shared_output
modeling_minimax_m3_vl.py · L258–259
DeepSeek-V4.1-Flash-1017in its codenot stated
n_shared_experts=1

n_shared_experts=1 configures a shared expert that every token passes through alongside the routed experts, and the code explicitly loads/pads shared_experts weights.

text_config.n_shared_experts = 1
if ".shared_experts." in name:
                if is_pp_missing_parameter(name, self):
                    continue
                experts = self.layers[extract_layer_index(name)].ffn.experts
model.py · L1395–1398
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codenot stated
n_shared_experts=1 · moe_shared_expert_intermediate_size=5376

The text_config declares 1 shared expert with a nonzero intermediate size, applied alongside the routed experts in MoE layers.

llm_config.n_shared_experts = 1llm_config.moe_shared_expert_intermediate_size = 5376
Step-5-Previewin its codenot stated
share_expert_dim=1536

Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed output.

text_config.share_expert_dim = 1536
self.share_expert = Step5MLP(
            config=config,
            hidden_size=self.hidden_size,
            intermediate_size=config.share_expert_dim,
            hidden_act="silu",
model.py · L1753–1757
Kolibri-1in its codeadopted
shared_expert_intermediate_size=512

A shared_experts MLP with nonzero intermediate size (512) is added to every token's output alongside routed experts.

shared_expert_intermediate_size = 512
        self.shared_experts = Kolibri1MLP(config, intermediate_size=config.shared_expert_intermediate_size)
modeling_kolibri1.py · L305
        hidden_states = hidden_states + self.shared_experts(residuals)
modeling_kolibri1.py · L313
DeepSeek-V4.1-Flashin its codenot stated
n_shared_experts=1

MoE layers include n_shared_experts=1 shared expert(s) that every token passes through, loaded/handled via shared_experts weight paths.

text_config.n_shared_experts = 1
            if ".shared_experts." in name:
                if is_pp_missing_parameter(name, self):
                    continue
                experts = self.layers[extract_layer_index(name)].ffn.experts
model.py · L1395–1398
Hy4-previewin its codeadopted
n_shared_experts=1

HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed experts.

n_shared_experts = 1
self.shared_experts = HYV4MLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_hy_v4.py · L588–590
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_hy_v4.py · L598
GLM-5.3-Flashin its codenot stated
n_shared_experts=1

Glm5NextTextMoE includes a shared_experts MLP (sized by n_shared_experts=1) added to every token's output alongside the routed experts.

text_config.n_shared_experts = 1
        self.shared_experts = Glm5NextTextMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_glm5_next.py · L195–197
        hidden_states = hidden_states + self.shared_experts(residuals)
modeling_glm5_next.py · L205
Qwen3.8-Flash-Nextin its codenot stated
shared_expert_intermediate_size=640

Each MoE block has a shared_expert MLP with nonzero shared_expert_intermediate_size applied to every token and gated before being added to routed expert output.

text_config.shared_expert_intermediate_size = 640
self.shared_expert = Qwen4ExpTextMLP(config, intermediate_size=config.shared_expert_intermediate_size)
        self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)
modeling_qwen4_exp.py · L973–974
DeepSeek-V4-Flash-Vision-Expin its codenot stated
n_shared_experts=1

Each MoE block adds a shared DeepseekV4MLP that every token passes through in addition to the routed experts, with n_shared_experts=1.

n_shared_experts = 1
self.shared_experts = DeepseekV4MLP(config)
modeling_deepseek_v4.py · L1137
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
GLM-5.3in its codenot stated
n_shared_experts=1

GlmMoeDsaMoE adds a shared_experts MLP sized by n_shared_experts=1 that every token passes through in addition to routed experts.

n_shared_experts = 1
self.shared_experts = GlmMoeDsaMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_glm_moe_dsa.py · L560–562
DeepSeek-V4-Pro-0813in its codenot stated
n_shared_experts=1

Each MoE block also runs a DeepseekV4MLP shared expert that every token passes through, added to the routed output, with n_shared_experts=1

n_shared_experts = 1
self.shared_experts = DeepseekV4MLP(config)
modeling_deepseek_v4.py · L1137
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
n_shared_experts=1 · moe_shared_expert_intermediate_size=3712

Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with n_shared_experts=1.

n_shared_experts = 1moe_shared_expert_intermediate_size = 3712
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L731
hidden_states = hidden_states + self.shared_experts(residuals)
        return hidden_states
modeling_nemotron_h.py · L753–754
DeepSeek-V4-Flash-0731in its codenot stated
n_shared_experts=1

Each MoE block adds the output of a dense shared-expert MLP to every token's routed output, and n_shared_experts=1 in this config.

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Laguna-S-2.1in its codeadopted
shared_expert_intermediate_size=1024

Each sparse MoE block includes a shared_experts LagunaMLP with intermediate size shared_expert_intermediate_size=1024 added to every token's output.

shared_expert_intermediate_size = 1024
self.shared_experts = LagunaMLP(config, intermediate_size=config.shared_expert_intermediate_size)
modeling_laguna.py · L223
hidden_states = hidden_states + shared_output
modeling_laguna.py · L235
Kimi K3in its codenot stated
num_shared_experts=2

KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts.

text_config.num_shared_experts = 2
if config.num_shared_experts is not None:
            intermediate_size = config.moe_intermediate_size * config.num_shared_experts
            self.shared_experts = KimiMLP(
                config=config, intermediate_size=intermediate_size,
            )
modeling_kimi_linear.py · L797–801
if self.config.num_shared_experts is not None:
            y = y + self.shared_experts(identity)
modeling_kimi_linear.py · L836–837
Inklingin its codeadopted
n_shared_experts=2 · shared_expert_sink=true

MoE layers include 2 shared 'sink' experts that every token passes through in addition to the routed experts, enabled by n_shared_experts=2 and shared_expert_sink=true.

text_config.n_shared_experts = 2text_config.shared_expert_sink = true
assert config.n_shared_experts > 0 and config.shared_expert_sink
        assert config.inference_moe_w13_interleaved
        n_routed = config.n_routed_experts
        n_shared = config.n_shared_experts
moe.py · L421–424
Hy3in its codeadopted
num_shared_experts=1

Each MoE layer includes a shared HYV3MLP expert sized by num_shared_experts=1 that every token passes through and adds to the routed output.

num_shared_experts = 1
shared_intermediate = config.moe_intermediate_size * config.num_shared_experts
        self.shared_experts = HYV3MLP(config, intermediate_size=shared_intermediate)
modeling_hy_v3.py · L347–348
GLM-5.2in its codenot stated
n_shared_experts=1

GlmMoeDsaMoE adds a shared_experts MLP (sized by n_shared_experts=1) that every token passes through in addition to the routed experts.

n_shared_experts = 1
self.shared_experts = GlmMoeDsaMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_glm_moe_dsa.py · L560–562
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codeadopted
n_shared_experts=1 · moe_shared_expert_intermediate_size=10240

Every MoE layer additionally runs a shared expert MLP of intermediate size 10240 that all tokens pass through, added to the routed-expert output.

n_shared_experts = 1moe_shared_expert_intermediate_size = 10240
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L731
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_nemotron_h.py · L753
MiniMax-M3in its codeadopted
n_shared_experts=1 · shared_intermediate_size=3072

Each MoE block runs a dense shared expert of shared_intermediate_size=3072 (n_shared_experts=1) whose output is added to the routed experts' output.

text_config.n_shared_experts = 1text_config.shared_intermediate_size = 3072
self.shared_experts = MiniMaxM3VLDenseMLP(config, intermediate_size=config.shared_intermediate_size)
modeling_minimax_m3_vl.py · L248
hidden_states = hidden_states + shared_output
modeling_minimax_m3_vl.py · L259
Step-3.7-Flashin its codenot stated
share_expert_dim=1280

Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally to the routed output.

text_config.share_expert_dim = 1280
self.shared_experts = Step3p7MLP(config, layer_idx, is_shared_expert=True)
modeling_step3p7.py · L610
hidden_states = hidden_states * self.routed_scaling_factor
        hidden_states = hidden_states + shared_output
modeling_step3p7.py · L621–622
DeepSeek-V4-Flashin its codeadopted
n_shared_experts=1

Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config confirms it's enabled.

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
DeepSeek-V4-Proin its codeadopted
n_shared_experts=1

Every MoE layer adds one shared DeepseekV4MLP expert that all tokens pass through, with n_shared_experts=1.

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Qwen3.6-35B-A3Bin its codenot stated
shared_expert_intermediate_size=512

Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated and added to the routed output.

text_config.shared_expert_intermediate_size = 512
self.shared_expert = Qwen3_5MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size)
        self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)
modeling_qwen3_5_moe.py · L889–890
Qwen3.5-397B-A17Bin its codenot stated
shared_expert_intermediate_size=1024

Each MoE block also includes a shared_expert MLP (sized by shared_expert_intermediate_size=1024) applied to every token and gated, added to the routed-expert output.

text_config.shared_expert_intermediate_size = 1024
self.shared_expert = Qwen3_5MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size)
        self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)
modeling_qwen3_5_moe.py · L889–890
shared_expert_output = F.sigmoid(self.shared_expert_gate(hidden_states_reshaped)) * shared_expert_output

        expert_output = expert_output + shared_expert_output
modeling_qwen3_5_moe.py · L899–901
DeepSeek-V3.2in its codenot stated
n_shared_experts=1

DeepseekV32MoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) applied to every token in addition to routed experts.

n_shared_experts = 1
self.shared_experts = DeepseekV32MLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_deepseek_v32.py · L587–589
LFM3-1.7Bnot in its codenot stated
verdict and lines

There is no MoE mechanism at all, so there is no shared-expert path either.

MiMo-V2.6-Flashnot in its codenot stated
n_shared_experts=null

n_shared_experts is null in this config, and MiMoV2MoE builds no additional always-on expert module.

n_shared_experts = null
MiMo-V2.6-Pronot in its codenot stated
n_shared_experts=null

n_shared_experts is null in the config and no shared-expert module is instantiated in MiMoV2MoE.

n_shared_experts = null
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

There is no MoE structure at all, so no shared-expert mechanism or related config keys exist.

MolmoAct 2not in its codenot stated
verdict and lines

There is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key exists.

MiMo-V2.5not in its codenot stated
n_shared_experts=null

n_shared_experts is null and MiMoV2MoE has no shared-expert module in the code.

n_shared_experts = null
MiMo-V2.5-Pronot in its codenot stated
n_shared_experts=null

n_shared_experts is null in this config and no shared-expert module is instantiated in MiMoV2MoE.

n_shared_experts = null
Gemma 4 31Bnot in its codenot stated
verdict and lines

There is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is disabled.

X-VLAnot in its codenot stated
verdict and lines

There is no MoE structure at all, so no shared-expert mechanism exists in this model.

gpt-oss-120bnot in its codenot stated
verdict and lines

No shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of selected experts

self.router = GptOssTopKRouter(config)
        self.experts = GptOssExperts(config)
modeling_gpt_oss.py · L136–137
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No shared-expert config keys (n_shared_experts, shared_expert_intermediate_size) or code are present, consistent with there being no MoE at all.

Documented in

Tencent (Hunyuan)
Aleph Alpha
Mistral AI
Thinking Machines Lab
MiniMax
Poolside

Evidence

13 quotes from 13 sources, one card per model, each opening with its strongest.

How sources treat it
used 3core 10

Mistral-Large-4

Mistral AIadopted3 sources

One shared expert always runs alongside the selected experts, for a total of 5 active experts per token.

coremodel architecturein Mistral Large 4Mistral AI
2 more from this model

and 1 shared expert

coremodel architecturein Mistral Large 4Mistral AI

128 routed experts (top-4) plus one shared expert

coreunclearin Mistral Large 4Mistral AI

MiniMax-M3

MiniMaxadopted1 source

Each MoE layer uses 128 routed experts, 1 shared expert, and top-4 routed expert selection.

usedmodel architecturein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑