A Mixture-of-Experts layer replaces the feed-forward network in transformer blocks with multiple expert networks.1 A router selects a small set of experts for each token, so only part of the layer's parameters are activated per token.2,3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
MistralLarge4TextMoE routes tokens to n_routed_experts=128 experts via a top-k router selecting num_experts_per_tok=4 experts per token, applied in all 'sparse' MLP layers.
Feed-forward layers route tokens to a subset (num_experts_per_tok=6) of n_routed_experts=384 expert MLPs chosen by a learned router/gate, implemented via DeepseekV4MoE/RoutedExperts.
MoE feed-forward layers route each token via a sigmoid-scored top-8-of-256 router (MiMoV2MoEGate/MiMoV2MoE) applied on layers flagged in moe_layer_freq.
Most decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts, per mlp_layer_types='sparse'.
Glm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all mlp_layer_types=='sparse' layers.
Feed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash router for bootstrap layers).
Feed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6 experts per token
Each MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for the bootstrap layers).
KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace.
The SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via DeepseekV4TopKRouter or a hash-based router.
The feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no num_experts/num_experts_per_tok config keys exist.
The MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it is switched off for this config.
Feed-forward blocks (fc1/fc2 in Florence2EncoderLayer/DecoderLayer, and Mlp in transformer.py) are single dense MLPs with no router or expert selection.
V4.1 Flash is a sparse mixture-of-experts model from DeepSeek, and the first built on the company's Causal Encoder-Decoder (CED) architecture. It activates 8B parameters on input and 16B on output from a 552B-parameter backbone, an asymmetric split
Fine-grained expert segmentation and shared expert isolation allow universal priors and specialized embodiment/task patterns to coexist under the same active compute budget.
Furthermore, we adopt fine-grained expert segmentation and shared expert isolation, in which a lightweight shared expert preserves universal priors, while a set of routed experts provides specialized modeling capacity.
Fine-grained expert segmentation and shared expert isolation allow universal priors and specialized embodiment/task patterns to coexist under the same active compute budget.
We also apply short convolutions at two points — after the key and value projections in each attention layer, and on the attention and MLP residual branch outputs before they rejoin the main residual stream.
By decoupling expert load balancing from the primary action-learning objective, and employing sigmoid-based routing confidence to allow each token to independently activate multiple experts, our design enables sparse experts to capture general representations and control logic shared across embodiments.
sparse MoE MLPs in most layers. For stability, the first two layers remain dense while all subsequent layers include an MoE block. Experts themselves are SwiGLU MLPs. Each MoE layer has 32 experts and selects the Top-k experts per token with a normalized sigmoid router and adaptive routing biases for load balancing
Mixture-of-Experts: Each MoE block consists of a fixed number of experts (128 for gpt-oss-120b and 32 for gpt-oss-20b), as well as a standard linear router projection which maps residual activations to scores for each expert. For both models, we select the top- experts for each token given by the router, and weight the output of each expert by the softmax of the router projection over only the selected experts.
The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE, each containing 256 routed experts and 1 shared expert
coremodel architecturein Hy4-previewTencent Hy Team
two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated)
Its mixture-of-experts architecture activates only a small fraction of its parameters for each token, keeping compute per token low while retaining the capacity of a much larger model.
NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of always-on agents.