general family · filed under model architecture
Mixture of Experts
A model architecture using a mixture of experts; the evidence does not specify a more particular routing or activation mechanism.
Also called 198B total params / 11B activated params, 552B-parameter MoE, Mixture of Experts (MoE), Mixture-of-Experts (MoE) architecture, Mixture-of-Experts (MoE) architectures, MoE.
- sources
- 88
- models
- 24
- labs adopt it
- 13
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017) paper arxiv.orgintroduces the sparsely-gated MoE layer
- Mixture of Experts (MoE) - LLM Architecture Gallery explainer sebastianraschka.com
In the code
Checked as “Mixture of experts” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | core | Glm5NextTextMoE routes each token to num_experts_per_tok=8 of n_routed_experts=288 expert MLPs via a learned sigmoid router (Glm5NextTextTopkRouter) in sparse mlp_layer_types layers. modeling_glm5_next.py#L178 |
| DeepSeek-V4.1-Flash | in its code | core | Feed-forward layers (DeepseekV4MoE) route each token to a learned top-k subset of n_routed_experts routed experts. model.py#L118 |
| Hy4-preview | in its code | core | HYV4TopkRouter routes each token to num_experts_per_tok of n_routed_experts expert MLPs via sigmoid-scored top-k selection, used in every layer where mlp_layer_types is "sparse". modeling_hy_v4.py#L543 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | Feed-forward layers use DeepseekV4SparseMoeBlock with a learned TopKRouter (or hash router) selecting num_experts_per_tok=6 of n_routed_experts=256 experts per token. modeling_deepseek_v4.py#L1086 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | core | MoE layers route tokens to a subset of 512 routed experts via a top-k router selecting 22 experts per token. modeling_nemotron_h.py#L730 |
| MiMo-V2.6-Flash | in its code | core | MoE feed-forward layers route each token via a sigmoid-scored top-8-of-256 router (MiMoV2MoEGate/MiMoV2MoE) applied on layers flagged in moe_layer_freq. modeling_mimo_v2.py#L191 |
| DeepSeek-V4-Flash | in its code | core | Feed-forward layers route tokens to top num_experts_per_tok=6 of n_routed_experts=256 experts via a learned TopKRouter (or hash router for early layers). modeling_deepseek_v4.py#L1086 |
| MiMo-V2.5 | in its code | core | MoE layers route each token via a sigmoid-scored router (noaux_tc) to 8 of 256 routed experts per token, per moe_layer_freq. modeling_mimo_v2.py#L191 |
| GLM-5.3 | in its code | not stated | Sparse MLP layers route each token via a learned TopkRouter to 8 of 256 routed experts (n_routed_experts=256, num_experts_per_tok=8). modeling_glm_moe_dsa.py#L513 |
| Hy3 | in its code | core | Feed-forward layers (all but the first) route tokens to 8 of 192 experts via a learned sigmoid router modeling_hy_v3.py#L384 |
| GLM-5.2 | in its code | core | MoE decoder layers route each token via GlmMoeDsaTopkRouter to 8 of 256 routed experts (GlmMoeDsaExperts) on layers marked 'sparse'. modeling_glm_moe_dsa.py#L592 |
| MiniMax-M3 | in its code | core | Feed-forward layers marked 'sparse' in mlp_layer_types route each token to num_experts_per_tok of num_local_experts via a sigmoid top-k router. modeling_minimax_m3_vl.py#L243 |
| DeepSeek-V3.2 | in its code | not stated | DeepseekV32MoE routes tokens to n_routed_experts=256 experts via a top-k sigmoid router (num_experts_per_tok=8) in every layer beyond first_k_dense_replace. modeling_deepseek_v32.py#L542 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | core | Feed-forward layers route tokens to a subset of n_routed_experts=256 expert MLPs with num_experts_per_tok=6 chosen by a learned TopKRouter (or hash router for early layers). modeling_deepseek_v4.py#L1086 |
| DeepSeek-V4-Pro | in its code | core | Feed-forward MoE layers route tokens to top-6 of 384 routed experts via a learned router modeling_deepseek_v4.py#L1086 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | The SparseMoeBlock routes tokens to a top-k of n_routed_experts=384 expert MLPs via a learned router (TopKRouter or HashRouter for early layers). modeling_deepseek_v4.py#L1086 |
| Gemma 4 31B | not in its code | core | MoE code (router + experts) exists but enable_moe_block is false and num_experts/top_k_experts are null in this config, so no routing occurs. modeling_gemma4.py#L1376 |
| Inkling | in its code | core | Feed-forward layers past dense_mlp_idx route each token to a top-k subset of 256 routed experts chosen by a sigmoid gate. moe.py#L442 |
| Kimi K3 | in its code | core | KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace. modeling_kimi_linear.py#L786 |
| Laguna-S-2.1 | in its code | core | LagunaSparseMoeBlock routes tokens to top-10 of 256 experts via a learned sigmoid router, applied in layers marked sparse. modeling_laguna.py#L242 |
| MiMo-V2.5-Pro | in its code | core | Feed-forward layers route tokens to 8-of-384 experts chosen by a learned sigmoid router (noaux_tc top-k selection). modeling_mimo_v2.py#L186 |
| MiMo-V2.6-Pro | in its code | core | MoE feed-forward layers route tokens to 8 of 384 experts per token via a learned sigmoid router (MiMoV2MoEGate), active per moe_layer_freq. modeling_mimo_v2.py#L191 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | core | NemotronHTopkRouter routes each token to num_experts_per_tok=6 of n_routed_experts=128 expert MLPs via sigmoid scores and top-k selection. modeling_nemotron_h.py#L759 |
| Qwen3.5-397B-A17B | in its code | core | Each MoE block routes tokens via a learned TopKRouter to a subset (10 of 512) of expert MLPs. modeling_qwen3_5_moe.py#L897 |
| Qwen3.6-35B-A3B | in its code | core | Qwen3_5MoeSparseMoeBlock routes each token to num_experts_per_tok=8 of num_experts=256 expert MLPs via a learned TopKRouter. modeling_qwen3_5_moe.py#L897 |
| Qwen3.8-Flash-Next | in its code | core | Qwen4ExpTextSparseMoeBlock routes each token via a top-k router (Qwen4ExpTextTopKRouter) to a subset of the 512 routed experts. modeling_qwen4_exp.py#L973 |
| Step-3.7-Flash | in its code | core | Step3p7SparseMoeBlock routes tokens via Step3p7TopKRouter's top-k selection over 288 routed experts with 8 selected per token. modeling_step3p7.py#L606 |
| gpt-oss-120b | in its code | core | Each token is routed via top-k (4 of 128) learned router to expert MLPs. modeling_gpt_oss.py#L128 |
Evidence
101 spans quoted from the sources, strongest treatment first.
MiMo-V2.6-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used during inference.
Inkling is a mixture-of-experts system with 975 billion total parameters
NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of always-on agents.
Architecture: Mixture-of-Experts (MoE)
Qwen3.5-397B-A17B is a 397 billion parameter Mixture-of-Experts model with only 17 billion active parameters per token.
Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model
a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model
a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model
scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts
Architecture: Sparse MoE (Mixture of Experts), 1.02T total / 42B activated parameters
native omnimodal MoE model
DeepSeek V4 is a Mixture-of-Experts (MoE) language model series
breakthrough hybrid mixture-of-experts architecture
198B total params / 11B activated params
The gpt-oss models are autoregressive Mixture-of-Experts (MoE) transformers
open-weight multimodal mixture-of-experts model
The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE
The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE, each containing 256 routed experts and 1 shared expert
Featuring both Dense and Mixture-of-Experts (MoE) architectures
Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model.
Hy3’s architecture contains a sparse MoE with 192 experts and top-8 routing.
109B-parameter Mixture of Experts (MoE) model trained from scratch
with approximately 550 billion total parameters and 55 billion active
A 66-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) feed-forward backbone
Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model
activates just 16 of its 896 experts per token
550 billion total and 55 billion active parameter Mixture-of-Experts
Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Nemotron 3 Ultra uses a MoE Hybrid Mamba-Attention architecture
Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
It is a mixture-of-experts model with 320B total parameters and 18B active per token
Inkling is "a Mixture-of-Experts transformer with 975B total parameters, 41B active"
MoE experts: 512 experts, 10 routed + 1 shared activated per token
The model employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture, utilizing interleaved Mamba-2 and MoE layers
It is a Mixture-of-Experts (MoE) model with 8B activated parameters per token.
MiMo-V2.5-Pro is an open-source Mixture-of-Experts (MoE) language model
Qwen3.8-Flash-Next is a sparse mixture-of-experts model
MiMo-V2.5-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used during inference.
Employs Mixture-of-Experts Hybrid Mamba-Attention architecture.
552B-parameter MoE.
NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA
Inkling is a Mixture-of-Experts transformer with 975B total parameters, 41B active.
Hy4 preview is a mixture-of-experts model from Tencent
Both models are mixture-of-experts: gpt-oss-120b activates 5.1B parameters per token
Both big models are sparse mixture-of-experts
we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with support for contexts of up to one million tokens.
DeepSeek-V4-Pro carries 1.6 trillion parameters with 49 billion activated per token.
The MoE framework that manages 896 experts with 16 active per token.
Qwen3.8-Flash-Next is a multimodal, ultra-sparse Mixture-of-Experts model.
The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.
integrates a linear attention mechanism with a sparse mixture-of-experts model
117B parameters with 5.1B active parameters
sparse mixture-of-experts
its Mixture-of-Experts (MoE) architecture means only approximately 40 billion parameters are active
Inkling is a 975B-parameter sparse Mixture-of-Experts model with 41B active parameters
DeepSeek-V4-Pro: 1.6T total / 49B active params
a sparse Mixture-of-Experts (MoE) feed-forward backbone
117B-parameter Mixture-of-Experts (MoE) language model ... activates 5.1B parameters per forward pass
multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token
Built on a Mixture of Experts (MoE) architecture
Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture
Mixture-of-Experts language model
two strong Mixture-of-Experts (MoE) language models
M.1 and XS.2 are Mixture-of-Experts foundation models.
Model type: Mixture of Experts (MoE)
In an MoE model: On the other hand, the router chooses only a small subset of experts.
Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model
GLM-5 scales to 256 experts and reduces its layer count to 80 to minimize expert parallelism communication overhead.
while the MoE experts are compressed to W4A8 (INT4)
It is a MoE model with a total of 295 billion parameters and 21 billion active parameters
Mixture Of Experts - Number of Experts: 256 - Number of Activated Experts: 8 Routed + 1 Shared
Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters
Mixture-of-Experts Hybrid Mamba-Attention architecture
Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Nemotron 3 Ultra uses a MoE Hybrid Mamba-Attention architecture
two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated)
Built on a Mixture-of-Experts architecture with 309B total parameters and 15B activated per token
Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning.
By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.
Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token
Architecture: Sparse MoE (Mixture of Experts)
Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters
DeepSeek V4.1 Flash is a sparse mixture-of-experts model from DeepSeek
Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model
Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model
Architecture: Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters
employs a sparse Mixture of Experts (MoE) architecture
It is a sparse mixture-of-experts model with 13B active parameters out of 284B total.
Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model
MiMo-V2-Flash is a Mixture-of-Experts (MoE) model with 309B total parameters
MoE models suffer from inconsistent expert routing across rollout and training due to numerical precision issues
Mixture-of-Experts language model
Laguna S 2.1 is a 118B total parameter Mixture-of-Experts (MoE) model with 8B activated parameters per token
MiMo-V2.5 is a 310B-parameter Sparse MoE model (15B active)
activating only 3.8 billion of its total parameters during inference to deliver exceptionally fast tokens-per-second
The MoE contains 64 experts with top-4 expert routing
the Gemma 4 model suite features dense and Mixture-of-Experts architectures
The MoE backbone likely stays.
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model
The model runs on a Mixture-of-Experts (MoE) architecture with 1 trillion total parameters and 42 billion active parameters per inference pass.
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts.