Open Models Almanac
Methodsmodel architecturechannel mixermixture of experts

general family · filed under model architecture

Mixture of experts

A Mixture-of-Experts layer replaces the feed-forward network in transformer blocks with multiple expert networks.1 A router selects a small set of experts for each token, so only part of the layer's parameters are activated per token.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we instantiate sparse MoE layers inside the action expert, which replace the feed-forward network (FFN) in the action expert — From Foundation to Application: Improving VLA Models in Practice
  2. is the selected top- routed expert set — From Foundation to Application: Improving VLA Models in Practice
  3. only routed experts are activated for each token — From Foundation to Application: Improving VLA Models in Practice

Also called Mixture of Experts (MoE), Mixture-of-Experts (MoE) architecture, Mixture-of-Experts (MoE) architectures, MoE, MoE model, Model-of-Experts (MoE) architecture.

sources
113
models
32
labs adopt it
17
strongest
core

In practice

Checked as “Mixture of experts” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codeadopted
n_routed_experts=128 · num_experts_per_tok=4

MistralLarge4TextMoE routes tokens to n_routed_experts=128 experts via a top-k router selecting num_experts_per_tok=4 experts per token, applied in all 'sparse' MLP layers.

text_config.n_routed_experts = 128text_config.num_experts_per_tok = 4
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_mistral_large4.py · L543–544
MiniMax-M3.1-Flashin its codeadopted
num_local_experts=128 · num_experts_per_tok=4

A sigmoid-scoring TopKRouter selects num_experts_per_tok=4 of num_local_experts=128 routed experts per token in the sparse MLP blocks.

text_config.num_local_experts = 128text_config.num_experts_per_tok = 4
_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L236–237
DeepSeek-V4.1-Flash-1017in its codeadopted
n_routed_experts=384 · num_experts_per_tok=6

Feed-forward layers route tokens to a subset (num_experts_per_tok=6) of n_routed_experts=384 expert MLPs chosen by a learned router/gate, implemented via DeepseekV4MoE/RoutedExperts.

text_config.n_routed_experts = 384text_config.num_experts_per_tok = 6
config = vllm_config.model_config.hf_config
        n_routed_experts = config.n_routed_experts
        n_activated_experts = config.num_experts_per_tok
model.py · L139–141
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codeadopted
n_routed_experts=512 · num_experts_per_tok=22

The text_config defines 512 routed experts with 22 selected per token, used by the 'moe' blocks in layers_block_type.

llm_config.n_routed_experts = 512llm_config.num_experts_per_tok = 22
Step-5-Previewin its codeadopted
num_experts_per_tok=8 · moe_num_experts=352 · moe_top_k=8

Router (GateLinear) selects moe_top_k=8 of moe_num_experts=352 experts per token for layers listed in moe_layer_list.

text_config.moe_num_experts = 352text_config.moe_top_k = 8text_config.num_experts_per_tok = 8
self.gate = GateLinear(
            config.hidden_size,
            config.moe_num_experts,
            bias=False,
            out_dtype=torch.float32,
            params_dtype=torch.float32,
            prefix=f"{prefix}.gate",
        )
model.py · L1700–1707
Kolibri-1in its codeadopted
num_experts=384 · num_experts_per_tok=6

Each token is routed via Kolibri1TopKRouter to 6 of 384 expert MLPs in Kolibri1SparseMoeBlock.

num_experts = 384num_experts_per_tok = 6
        router_indices = torch.topk(router_logits + self.e_score_correction_bias, self.top_k, dim=-1)[1]
        router_scores = router_logits.gather(1, router_indices).sigmoid()
modeling_kolibri1.py · L288–289
MiMo-V2.6-Flashin its codeadopted
n_routed_experts=256 · num_experts_per_tok=8

MoE feed-forward layers route each token via a sigmoid-scored top-8-of-256 router (MiMoV2MoEGate/MiMoV2MoE) applied on layers flagged in moe_layer_freq.

n_routed_experts = 256num_experts_per_tok = 8
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L191–193
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
MiMo-V2.6-Proin its codeadopted
n_routed_experts=384 · num_experts_per_tok=8 · moe_layer_freq=[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…

MoE feed-forward layers route tokens to 8 of 384 experts per token via a learned sigmoid router (MiMoV2MoEGate), active per moe_layer_freq.

n_routed_experts = 384num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L191–193
DeepSeek-V4.1-Flashin its codeadopted
n_routed_experts=384 · num_experts_per_tok=6

Feed-forward layers route tokens to a subset of 384 routed experts (6 per token) via a learned router/gate, implemented in DeepseekV4MoE.

text_config.n_routed_experts = 384text_config.num_experts_per_tok = 6
        config = vllm_config.model_config.hf_config
        n_routed_experts = config.n_routed_experts
        n_activated_experts = config.num_experts_per_tok
model.py · L139–141
Hy4-previewin its codeadopted
n_routed_experts=256 · num_experts_per_tok=8

Most decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts, per mlp_layer_types='sparse'.

n_routed_experts = 256num_experts_per_tok = 8
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L531–532
self.mlp = HYV4MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else HYV4MLP(config)
modeling_hy_v4.py · L673
GLM-5.3-Flashin its codeadopted
n_routed_experts=288 · num_experts_per_tok=8

Glm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all mlp_layer_types=='sparse' layers.

text_config.n_routed_experts = 288text_config.num_experts_per_tok = 8
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm5_next.py · L176–177
Qwen3.8-Flash-Nextin its codeadopted
num_experts=512 · num_experts_per_tok=10

The MLP block routes each token to num_experts_per_tok of num_experts experts via a learned TopKRouter.

text_config.num_experts = 512text_config.num_experts_per_tok = 10
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen4_exp.py · L960
DeepSeek-V4-Flash-Vision-Expin its codeadopted
n_routed_experts=256 · num_experts_per_tok=6

Feed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash router for bootstrap layers).

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
modeling_deepseek_v4.py · L1094
GLM-5.3in its codenot stated
n_routed_experts=256 · num_experts_per_tok=8 · first_k_dense_replace=3

MoE layers (mlp_layer_types="sparse" for most of the 78 layers) route tokens via GlmMoeDsaTopkRouter to 8 of 256 routed experts.

n_routed_experts = 256num_experts_per_tok = 8first_k_dense_replace = 3
self.mlp = GlmMoeDsaMoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else GlmMoeDsaMLP(config)
modeling_glm_moe_dsa.py · L580
DeepSeek-V4-Pro-0813in its codenot stated
n_routed_experts=384 · num_experts_per_tok=6

Feed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6 experts per token

n_routed_experts = 384num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codeadopted
n_routed_experts=128 · num_experts_per_tok=6

NemotronHMoE routes tokens via NemotronHTopkRouter to a subset (top-6 of 128) of NemotronHExperts.

n_routed_experts = 128num_experts_per_tok = 6
self.top_k = config.num_experts_per_tok
        self.num_experts = config.num_local_experts
        self.hidden_dim = config.hidden_size
        self.weight = nn.Parameter(torch.zeros(self.num_experts, self.hidden_dim))
modeling_nemotron_h.py · L760–763
DeepSeek-V4-Flash-0731in its codenot stated
n_routed_experts=256 · num_experts_per_tok=6

Each MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for the bootstrap layers).

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Laguna-S-2.1in its codeadopted
num_experts=256 · num_experts_per_tok=10

LagunaSparseMoeBlock routes tokens via LagunaTopKRouter top-k=10 over num_experts=256 experts for layers marked 'sparse' in mlp_layer_types.

num_experts = 256num_experts_per_tok = 10
_, selected_experts = torch.topk(scores_for_selection, self.top_k, dim=-1)
        routing_weights = routing_scores.gather(-1, selected_experts)
modeling_laguna.py · L170–171
Kimi K3in its codeadopted
num_experts=896 · num_experts_per_token=16 · first_k_dense_replace=1

KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace.

text_config.num_experts = 896text_config.num_experts_per_token = 16text_config.first_k_dense_replace = 1
self.experts = nn.ModuleList(
            [
                KimiBlockSparseMLP(
                    config,
                    hidden_size=self.moe_hidden_size,
                    intermediate_size=config.moe_intermediate_size,
                )
                for _ in range(config.num_experts)
            ],
        )
        self.gate = KimiMoEGate(config)
modeling_kimi_linear.py · L786–796
Inklingin its codeadopted
n_routed_experts=256 · num_experts_per_tok=6

Feed-forward layers beyond dense_mlp_idx route tokens to a learned sigmoid gate selecting top-6 of 256 routed experts via InklingMoE/FusedMoE.

text_config.n_routed_experts = 256text_config.num_experts_per_tok = 6
self.experts = FusedMoEFactory(
            num_experts=num_experts,
            top_k=config.num_experts_per_tok,
            hidden_size=config.hidden_size,
moe.py · L442–445
Hy3in its codeadopted
num_experts=192 · num_experts_per_tok=8

Feed-forward layers route tokens to 8 of 192 experts via a learned sigmoid router (HYV3TopKRouter) in all but the first dense layer.

num_experts = 192num_experts_per_tok = 8
self.mlp = HYV3MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else HYV3MLP(config)
modeling_hy_v3.py · L373
GLM-5.2in its codeadopted
n_routed_experts=256 · num_experts_per_tok=8

MoE layers (mlp_layer_types == "sparse") route tokens via GlmMoeDsaTopkRouter to top-8 of 256 routed experts.

n_routed_experts = 256num_experts_per_tok = 8
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm_moe_dsa.py · L501–502
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codeadopted
n_routed_experts=512 · num_experts_per_tok=22

MoE blocks route tokens to a subset of 512 routed experts chosen per-token by NemotronHTopkRouter with top-k of 22.

n_routed_experts = 512num_experts_per_tok = 22
self.top_k = config.num_experts_per_tok
        self.num_experts = config.num_local_experts
modeling_nemotron_h.py · L760–761
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
modeling_nemotron_h.py · L789
MiniMax-M3in its codeadopted
num_local_experts=128 · num_experts_per_tok=4

MoE feed-forward layers route each token to 4 of 128 experts via a learned sigmoid-scored router, enabled on most layers via mlp_layer_types.

text_config.num_local_experts = 128text_config.num_experts_per_tok = 4
_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L236–237
Step-3.7-Flashin its codeadopted
use_moe=true · moe_num_experts=288 · moe_top_k=8

Step3p7SparseMoeBlock routes tokens via Step3p7TopKRouter choosing top-8 of 288 experts, enabled on layers 3-44 per moe_layers_enum.

text_config.use_moe = truetext_config.moe_num_experts = 288text_config.moe_top_k = 8
_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_step3p7.py · L596–597
DeepSeek-V4-Flashin its codeadopted
n_routed_experts=256 · num_experts_per_tok=6

The SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via DeepseekV4TopKRouter or a hash-based router.

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
DeepSeek-V4-Proin its codeadopted
n_routed_experts=384 · num_experts_per_tok=6

Feed-forward layers route tokens to n_routed_experts=384 expert MLPs via a learned TopKRouter/HashRouter selecting num_experts_per_tok=6 experts.

n_routed_experts = 384num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
modeling_deepseek_v4.py · L1094
MiMo-V2.5in its codeadopted
n_routed_experts=256 · num_experts_per_tok=8 · moe_layer_freq=[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…

MoE layers route each token via a sigmoid-scored router (noaux_tc) to 8 of 256 routed experts per token, per moe_layer_freq.

n_routed_experts = 256num_experts_per_tok = 8moe_layer_freq = [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L191–193
MiMo-V2.5-Proin its codeadopted
n_routed_experts=384 · num_experts_per_tok=8

Feed-forward layers route tokens to 8-of-384 experts chosen by a learned sigmoid router (noaux_tc top-k selection).

n_routed_experts = 384num_experts_per_tok = 8
self.experts = nn.ModuleList(
            [MiMoV2MLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(config.n_routed_experts)]
        )
modeling_mimo_v2.py · L186–188
Qwen3.6-35B-A3Bin its codeadopted
num_experts=256 · num_experts_per_tok=8

Qwen3_5MoeTopKRouter routes each token to num_experts_per_tok=8 of num_experts=256 expert MLPs via softmax top-k routing.

text_config.num_experts = 256text_config.num_experts_per_tok = 8
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen3_5_moe.py · L877
Qwen3.5-397B-A17Bin its codeadopted
num_experts=512 · num_experts_per_tok=10

Qwen3_5MoeSparseMoeBlock routes tokens to a top-k subset of 512 experts via a learned router (Qwen3_5MoeTopKRouter).

text_config.num_experts = 512text_config.num_experts_per_tok = 10
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen3_5_moe.py · L877
DeepSeek-V3.2in its codenot stated
n_routed_experts=256 · num_experts_per_tok=8

DeepseekV32MoE routes tokens to num_experts_per_tok of n_routed_experts experts via a learned sigmoid router with grouped top-k selection.

n_routed_experts = 256num_experts_per_tok = 8
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_deepseek_v32.py · L528–529
gpt-oss-120bin its codeadopted
num_local_experts=128 · num_experts_per_tok=4

A learned router selects top-4 of 128 experts per token for the feed-forward computation

num_local_experts = 128num_experts_per_tok = 4
router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L127
LFM3-1.7Bnot in its codeadopted
verdict and lines

No router or expert MLP modules exist; Lfm3MLP is a single dense SwiGLU feed-forward applied in every decoder layer.

class Lfm3MLP(nn.Module):
modeling_lfm3.py · L125
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No router, num_experts, or expert MLP module exists; Qwen3VLTextMLP is a single dense SwiGLU MLP per layer.

MolmoAct 2not in its codenot stated
verdict and lines

The feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no num_experts/num_experts_per_tok config keys exist.

class LanguageModelMLP(nn.Module):
    def __init__(
        self,
        input_dim: int,
        intermediate_size: int,
        hidden_act: str,
        device: Union[str, torch.device] = None,
    ):
modeling_molmoact2.py · L2336–2343
Gemma 4 31Bnot in its codeadopted
num_experts=null · enable_moe_block=false · top_k_experts=null

The MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it is switched off for this config.

text_config.enable_moe_block = falsetext_config.num_experts = nulltext_config.top_k_experts = null
self.enable_moe_block = config.enable_moe_block
        if self.enable_moe_block:
            self.router = Gemma4TextRouter(config)
            self.experts = Gemma4TextExperts(config)
modeling_gemma4.py · L1360–1363
X-VLAnot in its codenot stated
verdict and lines

Feed-forward blocks (fc1/fc2 in Florence2EncoderLayer/DecoderLayer, and Mlp in transformer.py) are single dense MLPs with no router or expert selection.

self.fc1 = nn.Linear(self.embed_dim, config.encoder_ffn_dim)
        self.fc2 = nn.Linear(config.encoder_ffn_dim, self.embed_dim)
modeling_florence2.py · L1239–1240
OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

No num_experts/n_routed_experts/num_local_experts config or MoE routing code is present; the feed-forward network is the standard dense Llama MLP.

text_config.model_type = "llama"

Documented in

Evidence

135 quotes from 113 sources, one card per model, each opening with its strongest.

How sources treat it
used 9core 126

Inkling

Thinking Machines Labadopted7 sources

A 66-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) feed-forward backbone

coremodel architecturein InklingThinking Machines Lab
6 more from this model

Inkling is a Mixture-of-Experts transformer with 975B total parameters, 41B active.

coremodel architecturein InklingThinking Machines Lab

Inkling is a 975B-parameter sparse Mixture-of-Experts model with 41B active parameters

coremodel architecturein InklingThinking Machines Lab

Inkling is "a Mixture-of-Experts transformer with 975B total parameters, 41B active"

coremodel architecturein InklingThinking Machines Lab

Inkling is a mixture-of-experts system with 975 billion total parameters

coremodel architecturein InklingThinking Machines Lab

a sparse Mixture-of-Experts (MoE) feed-forward backbone

coremodel architecturein InklingThinking Machines Lab

open-weight multimodal mixture-of-experts model

coremodel architecturein InklingThinking Machines Lab

Hy3

Tencent (Hunyuan)adopted7 sources

Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture

coremodel architecturein Hy3Tencent
7 more from this model

It is a MoE model with a total of 295 billion parameters and 21 billion active parameters

coremodel architecturein Hy3Tencent

Hy3’s architecture contains a sparse MoE with 192 experts and top-8 routing.

coremodel architecturein Hy3Tencent

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model

coremodel architecturein Hy3Tencent Hy Team

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model

coremodel architecturein Hy3Tencent Hy Team

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model.

coremodel architecturein Hy3Tencent

Built on a Mixture of Experts (MoE) architecture

coremodel architecturein Hy3Tencent

Mixture-of-Experts language model

coremodel architecturein Hy3Tencent

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted7 sources

The model employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture, utilizing interleaved Mamba-2 and MoE layers

coremodel architecturein Nemotron 3 UltraNVIDIA
13 more from this model

Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA

550 billion total and 55 billion active parameter Mixture-of-Experts

coremodel architecturein Nemotron 3 UltraNVIDIA

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model

coremodel architecturein Nemotron 3 UltraNVIDIA

Nemotron 3 Ultra uses a MoE Hybrid Mamba-Attention architecture

coremodel architecturein Nemotron 3 UltraNVIDIA

Nemotron 3 Ultra uses a MoE Hybrid Mamba-Attention architecture

coremodel architecturein Nemotron 3 UltraNVIDIA

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model

usedmodel architecturein Nemotron 3 UltraNVIDIA

with approximately 550 billion total parameters and 55 billion active

coremodel architecturein Nemotron 3 UltraNVIDIA

Employs Mixture-of-Experts Hybrid Mamba-Attention architecture.

coremodel architecturein Nemotron 3 UltraNVIDIA

Mixture-of-Experts Hybrid Mamba-Attention architecture

coremodel architecturein Nemotron 3 UltraNVIDIA

breakthrough hybrid mixture-of-experts architecture

coremodel architecturein Nemotron 3 familyNVIDIA

interleaved Mamba-2 and MoE layers

coremodel architecturein Nemotron 3 UltraNVIDIA

Mistral-Large-4

Mistral AIadopted5 sources

The first decoder layer uses a dense feed-forward network. The other 71 layers use MoE blocks

coremodel architecturein Mistral Large 4Mistral AI
4 more from this model

a 52B active, 1T total parameter multimodal mixture-of-experts model

coremodel architecturein Mistral Large 4Mistral AI

Open-weight hybrid instruct-and-reasoning MoE with multimodal input

coreunclearin Mistral Large 4Mistral AI

a 1 trillion-parameter natively multimodal Mixture-of-Experts model

coremodel architecturein Mistral Large 4Mistral AI

Mistral Large 4 is Mistral AI's frontier mixture-of-experts multimodal model

coreunclearin Mistral Large 4Mistral AI

Its 120B-parameter mixture-of-experts (MoE) architecture activates 12B parameters per token

coremodel architecturein Nemotron 3.5 Super VLNVIDIA
4 more from this model

activate only a fraction of their total parameters for each token

coremodel architecturein NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 LightningNVIDIA

Nemotron Hybrid MoE with 512 routed experts per MoE layer

coremodel architecturein Nemotron 3.5 Super VLNVIDIA

512 routed experts per MoE layer

coremodel architecturein Nemotron 3.5 Super VLNVIDIA

an open multimodal mixture-of-experts model from NVIDIA

coremodel architecturein Nemotron 3.5 Super VLNVIDIA

Laguna-S-2.1

Poolsideadopted5 sources

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts (MoE) model with 8B activated parameters per token

coremodel architecturein Laguna S 2.1Poolside
4 more from this model

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token

coremodel architecturein Laguna S 2.1Poolside

M.1 and XS.2 are Mixture-of-Experts foundation models.

coremodel architecturein LAGUNA M.1 and LAGUNA XS.2Poolside

It is a Mixture-of-Experts (MoE) model with 8B activated parameters per token.

coremodel architecturein Laguna S 2.1Poolside

Model type: Mixture of Experts (MoE)

coremodel architecturein Laguna S 2.1Poolside

Kimi K3

Moonshot AIadopted5 sources

We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework.

coremodel architecturein Kimi K3Moonshot AI
4 more from this model

scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts

coremodel architecturein Kimi K3Moonshot AI

The MoE framework that manages 896 experts with 16 active per token.

coremodel architecturein Kimi K3Moonshot AI

Architecture: Mixture-of-Experts (MoE)

coremodel architecturein Kimi K3Moonshot AI

activates just 16 of its 896 experts per token

coremodel architecturein Kimi K3Moonshot AI

Step-3.7-Flash

StepFunadopted5 sources

Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model

coremodel architecturein Step 3.7 FlashStepFun
6 more from this model

Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model

coremodel architecturein Step 3.7 FlashStepFun

Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model

coremodel architecturein Step 3.7 FlashStepFun

a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model

coremodel architecturein Step 3.7 FlashStepFun

a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model

coremodel architecturein Step 3.7 FlashStepFun

198B total params / 11B activated params

coremodel architecturein Step 3.7 FlashStepFun

employs a sparse Mixture of Experts (MoE) architecture

coremodel architecturein Step 3.7 FlashStepFun

MiMo-V2.5

Xiaomiadopted5 sources

MoE models suffer from inconsistent expert routing across rollout and training due to numerical precision issues

coremodel architecturein MiMo-V2-FlashXiaomi
7 more from this model

Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters

coremodel architecturein MiMo-V2.5Xiaomi

Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters

coremodel architecturein MiMo-V2.5Xiaomi

MiMo-V2-Flash is a Mixture-of-Experts (MoE) model with 309B total parameters

coremodel architecturein MiMo-V2-FlashXiaomi

MiMo-V2.5 is a 310B-parameter Sparse MoE model (15B active)

coremodel architecturein MiMo-V2.5Xiaomi

Each MoE layer comprises 256 experts in total, with 8 activated per token, and contains no shared experts.

usedmodel architecturein MiMo-V2-FlashXiaomi

Architecture: Sparse MoE (Mixture of Experts)

coremodel architecturein MiMo-V2.5Xiaomi

native omnimodal MoE model

coremodel architecturein MiMo-V2.5Xiaomi

Gemma 4 31B

Google DeepMindadopted5 sources

Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning.

coremodel architecturein Gemma 4Google DeepMind
4 more from this model

By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest.

coremodel architecturein Gemma 4Google DeepMind

activating only 3.8 billion of its total parameters during inference to deliver exceptionally fast tokens-per-second

usedmodel architecturein Gemma 4 26BGoogle

the Gemma 4 model suite features dense and Mixture-of-Experts architectures

usedmodel architecturein Gemma 4Google DeepMind

Featuring both Dense and Mixture-of-Experts (MoE) architectures

coremodel architecturein Gemma 4Google DeepMind

DeepSeek-V4.1-Flash

DeepSeekadopted4 sources

The model uses 1 shared expert and 384 routed experts per MoE layer, activating 6 routed experts per token.

coremodel architecturein DeepSeek-V4.1-FlashDeepSeek
3 more from this model

DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model

coreunclearin DeepSeek-V4.1-FlashDeepSeek

552B-parameter MoE.

coremodel architecturein DeepSeek-V4.1-FlashDeepSeek

DeepSeek V4.1 Flash is a sparse mixture-of-experts model from DeepSeek

coremodel architecturein DeepSeek-V4.1-FlashDeepSeek

Hy4-preview

Tencent (Hunyuan)adopted4 sources

The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE, each containing 256 routed experts and 1 shared expert

coremodel architecturein Hy4-previewTencent Hy Team
5 more from this model

The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE

coremodel architecturein Hy4-previewTencent Hy Team

each containing 256 routed experts and 1 shared expert

coremodel architecturein Hy4-previewTencent Hy Team

Hy4 preview is a mixture-of-experts model from Tencent

coreunclearin Hy4-previewTencent

256 routed experts (top-8) and 1 shared expert

coremodel architecturein Hy4-previewTencent Hy Team

Mixture-of-Experts language model

coremodel architecturein Hy4-previewTencent Hy Team

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted4 sources

MoE experts: 512 experts, 10 routed + 1 shared activated per token

coremodel architecturein Qwen3.8-Flash-NextQwen Team
3 more from this model

Qwen3.8-Flash-Next is a sparse mixture-of-experts model

coreunclearin Qwen3.8-Flash-NextQwen Team

a multimodal MoE model

coremodel architecturein Qwen3.8-Flash-NextQwen Team

Qwen3.8-Flash-Next is a multimodal, ultra-sparse Mixture-of-Experts model.

coremodel architecturein Qwen3.8-Flash-NextQwen Team

DeepSeek-V4-Pro

DeepSeekadopted4 sources

two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated)

coremodel architecturein DeepSeek-V4 seriesDeepSeek
3 more from this model

DeepSeek V4 is a Mixture-of-Experts (MoE) language model series

coremodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V4-Pro carries 1.6 trillion parameters with 49 billion activated per token.

coremodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V4-Pro: 1.6T total / 49B active params

coremodel architecturein DeepSeek-V4DeepSeek

MiMo-V2.5-Pro

Xiaomiadopted4 sources

MiMo-V2.5-Pro is an open-source Mixture-of-Experts (MoE) language model

coremodel architecturein MiMo-V2.5-ProXiaomi
3 more from this model

MiMo-V2.5 is a 310B-parameter Sparse MoE model (15B active)

coremodel architecturein MiMo-V2.5Xiaomi

MiMo-V2.5-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used during inference.

coremodel architecturein MiMo-V2.5-ProXiaomi

The model runs on a Mixture-of-Experts (MoE) architecture with 1 trillion total parameters and 42 billion active parameters per inference pass.

usedmodel architecturein MiMo-V2.5-ProXiaomi

gpt-oss-120b

OpenAIadopted4 sources

117B parameters with 5.1B active parameters

coremodel architecturein gpt-ossOpenAI
3 more from this model

117B-parameter Mixture-of-Experts (MoE) language model ... activates 5.1B parameters per forward pass

coremodel architecturein gpt-oss-120bOpenAI

Both models are mixture-of-experts: gpt-oss-120b activates 5.1B parameters per token

coremodel architecturein gpt-oss-120b & gpt-oss-20bOpenAI

The gpt-oss models are autoregressive Mixture-of-Experts (MoE) transformers

coremodel architecturein gpt-oss-120b & gpt-oss-20bOpenAI

Step-5-Preview

StepFunadopted3 sources

Step 5 Preview is a sparse Mixture-of-Experts (MoE) model with 600B total parameters and 27B active parameters.

coremodel architecturein Step 5 PreviewStepFun
2 more from this model

built on a sparse Mixture-of-Experts architecture (27B active / 600B total parameters)

coremodel architecturein Step 5 PreviewStepFun

Step 5 Preview is a sparse Mixture-of-Experts (MoE) model

coremodel architecturein Step 5 PreviewStepFun

Kolibri-1

Aleph Alphaadopted3 sources

Its mixture-of-experts architecture activates only a small fraction of its parameters for each token, keeping compute per token low while retaining the capacity of a much larger model.

coremodel architecturein KolibriAleph Alpha
5 more from this model

Kolibri is a decoder-only causal Transformer ... with sparse MoE layers of FFNs

coremodel architecturein KolibriAleph Alpha

Kolibri is the second model in the Kolibri family, an English–German MoE

coreunclearin KolibriAleph Alpha

Sparse MoE architectures are able to increase model capacity at matched compute by routing each token through a subset of experts.

usedunclearin KolibriAleph Alpha

Kolibri is an English-German Mixture-of-Experts Transformer

coremodel architecturein KolibriAleph Alpha

Kolibri therefore uses MoE layers in all 50 blocks.

coremodel architecturein KolibriAleph Alpha

MiMo-V2.6-Pro

Xiaomiadopted3 sources

Architecture: Sparse MoE (Mixture of Experts), 1.02T total / 42B activated parameters

coremodel architecturein MiMo-V2.6-Pro-RLXiaomi
2 more from this model

MiMo-V2.6-Pro is a Mixture of Experts (MoE) model with 1.0 trillion total parameters, but only 42 billion active parameters are used during inference.

coremodel architecturein MiMo-V2.6-ProXiaomi

Both big models are sparse mixture-of-experts

coreunclearin MiMo-V2.6-ProXiaomi

MiMo-V2.6-Flash

Xiaomiadopted3 sources

Architecture: Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters

coremodel architecturein MiMo-V2.6-Flash-RLXiaomi
2 more from this model

Built on a Mixture-of-Experts architecture with 309B total parameters and 15B activated per token

coremodel architecturein MiMo-V2.6-FlashXiaomi

Both big models are sparse mixture-of-experts

coreunclearin MiMo-V2.6-ProXiaomi

NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of always-on agents.

coremodel architecturein Nemotron 3.5 LightningNVIDIA
3 more from this model

In an MoE model: On the other hand, the router chooses only a small subset of experts.

coreunclearin Nemotron 3.5 LightningNVIDIA

NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA

coreunclearin Nemotron 3.5 LightningNVIDIA

Published runtime configuration also exposes 128 routed experts plus a shared expert, with six routed experts selected per token.

usedunclearin Nemotron 3.5 LightningNVIDIA

DeepSeek-V4-Flash

DeepSeekadopted3 sources

DeepSeek V4 is a Mixture-of-Experts (MoE) language model series

coremodel architecturein DeepSeek-V4DeepSeek
2 more from this model

two strong Mixture-of-Experts (MoE) language models

coremodel architecturein DeepSeek-V4-FlashDeepSeek

DeepSeek-V4-Pro: 1.6T total / 49B active params

coremodel architecturein DeepSeek-V4DeepSeek

Qwen3.5-397B-A17B

Alibaba (Qwen)adopted3 sources

Qwen3.5-397B-A17B is a 397 billion parameter Mixture-of-Experts model with only 17 billion active parameters per token.

coremodel architecturein Qwen3.5-397B-A17BAlibaba
2 more from this model

integrates a linear attention mechanism with a sparse mixture-of-experts model

coremodel architecturein Qwen3.5-397B-A17BQwen Team

sparse mixture-of-experts

coremodel architecturein Qwen3.5-397B-A17BAlibaba

GLM-5.3-Flash

Z.ai (Zhipu AI)adopted2 sources

multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token

coremodel architecturein GLM-5.3-FlashZ.ai
1 more from this model

It is a mixture-of-experts model with 320B total parameters and 18B active per token

coremodel architecturein GLM-5.3-FlashZ.ai

DeepSeek-V4-Flash-Vision-Exp

DeepSeekadopted2 sources

The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.

coremodel architecturein DeepSeek-V4-Flash-Vision-ExpDeepSeek
1 more from this model

It is a sparse mixture-of-experts model with 13B active parameters out of 284B total.

coremodel architecturein DeepSeek-V4-Flash-Vision-ExpDeepSeek

LingBot-VLA 2.0

Robbyant (Ant Group)adopted2 sources

we instantiate sparse MoE layers inside the action expert, which replace the feed-forward network (FFN) in the action expert.

coreunclearin LingBot-VLA 2.0Robbyant
2 more from this model

we introduce a token-level loss-free MoE architecture for large-scale multi-embodiment VLA pretraining.

coreunclearin LingBot-VLA 2.0Robbyant

LingBot-VLA-V2 uses sparse MoE layers inside the action expert.

coremodel architecturein LingBot-VLA 2.0Robbyant

GLM-5.2

Z.ai (Zhipu AI)adopted2 sources

GLM-5 scales to 256 experts and reduces its layer count to 80 to minimize expert parallelism communication overhead.

coremodel architecturein GLM-5Z.ai
2 more from this model

while the MoE experts are compressed to W4A8 (INT4)

coremodel architecturein GLM-5Z.ai

its Mixture-of-Experts (MoE) architecture means only approximately 40 billion parameters are active

coremodel architecturein GLM-5.2Z.ai

MiniMax-M3

MiniMaxadopted2 sources

109B-parameter Mixture of Experts (MoE) model trained from scratch

coremodel architecturein MiniMax-M3MiniMax
2 more from this model

The MoE contains 64 experts with top-4 expert routing

usedmodel architecturein MiniMax Sparse AttentionMiniMax

The MoE backbone likely stays.

usedmodel architecturein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑