Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

methods · mixture of experts

expert load balancing

25 methods filed at this node or below it, from the sources of 14 models.

In practice

Checked as “Auxiliary-loss-free load balancing (selection bias)” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
MiniMax-M3.1-Flashin its codenot stated
use_routing_bias=true

The router adds a learned e_score_correction_bias to the sigmoid scores used only for top-k selection, not for the final weighting.

text_config.use_routing_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L235–237
DeepSeek-V4.1-Flash-1017in its codenot stated
topk_method=noaux_tc

topk_method is set to noaux_tc (DeepSeek-V3-style auxiliary-loss-free balancing) and the loader maps a gate bias to e_score_correction_bias used only for expert selection.

text_config.topk_method = "noaux_tc"
if name.endswith(".ffn.gate.bias"):
                    name = name.replace(
                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
                    )
dspark.py · L519–522
Step-5-Previewin its codenot stated
use_moe_router_bias=true

A per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true.

text_config.use_moe_router_bias = true
self.router_bias = nn.Parameter(
            torch.zeros(config.moe_num_experts, dtype=torch.float32),
            requires_grad=False,
        )
model.py · L1710–1713
e_score_correction_bias=self.router_bias,
            routed_scaling_factor=config.moe_router_scaling_factor,
model.py · L1777–1778
Kolibri-1in its codenot stated
verdict and lines

The router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits).

        self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_kolibri1.py · L273
        router_indices = torch.topk(router_logits + self.e_score_correction_bias, self.top_k, dim=-1)[1]
        router_scores = router_logits.gather(1, router_indices).sigmoid()
modeling_kolibri1.py · L288–289
MiMo-V2.6-Flashin its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
MiMo-V2.6-Proin its codenot stated
topk_method=noaux_tc

The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L176
DeepSeek-V4.1-Flashin its codenot stated
topk_method=noaux_tc

topk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced in weight remapping.

text_config.topk_method = "noaux_tc"
                if name.endswith(".ffn.gate.bias"):
                    name = name.replace(
                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
                    )
dspark.py · L519–522
Hy4-previewin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_hy_v4.py · L515–516
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L531–532
GLM-5.3-Flashin its codenot stated
topk_method=noaux_tc

Glm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc.

text_config.topk_method = "noaux_tc"
        scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_glm5_next.py · L160–161
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm5_next.py · L176–177
DeepSeek-V4-Flash-Vision-Expin its codenot stated
topk_method=noaux_tc

The TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method.

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
GLM-5.3in its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (topk_method "noaux_tc"), not for weighting outputs (weights are gathered from the un-biased scores).

topk_method = "noaux_tc"
scores_for_choice = scores + self.e_score_correction_bias
modeling_glm_moe_dsa.py · L486
topk_weights = scores.gather(1, topk_indices)
modeling_glm_moe_dsa.py · L502
DeepSeek-V4-Pro-0813in its codenot stated
topk_method=noaux_tc

The router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
        group_scores = (
            scores_for_choice.view(-1, self.num_group, self.num_experts // self.num_group)
            .topk(2, dim=-1)[0]
            .sum(dim=-1)
        )
modeling_nemotron_h.py · L773–779
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
DeepSeek-V4-Flash-0731in its codenot stated
topk_method=noaux_tc

The router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Laguna-S-2.1in its codenot stated
verdict and lines

The router adds a learned e_score_correction_bias to routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias-based balancing.

scores_for_selection = routing_scores + self.e_score_correction_bias.to(routing_scores.dtype)
        _, selected_experts = torch.topk(scores_for_selection, self.top_k, dim=-1)
        routing_weights = routing_scores.gather(-1, selected_experts)
modeling_laguna.py · L169–171
Kimi K3in its codeadopted
topk_method=noaux_tc

KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism.

text_config.topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Parameter(
            torch.empty(self.num_experts),
        )
modeling_kimi_linear.py · L693–695
scores_for_choice = scores + self.e_score_correction_bias.unsqueeze(0)
modeling_kimi_linear.py · L723
Inklingin its codenot stated
use_gate_bias=true

The gate adds a learned per-expert bias to the sigmoid routing scores used only for top-k selection, controlled by use_gate_bias=true (DeepSeek-style selection-bias load balancing).

text_config.use_gate_bias = true
sel = tl.where(mask_r, tl.sigmoid(logits), float("-inf"))
    if HAS_BIAS:
        bias = tl.load(bias_ptr + offs, mask=mask_r, other=0.0).to(tl.float32)
        sel = tl.where(mask_r, sel + bias, float("-inf"))
moe.py · L110–113
Hy3in its codenot stated
moe_router_enable_expert_bias=true

The router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing.

moe_router_enable_expert_bias = true
scores_for_choice = routing_weights + e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_hy_v3.py · L288–290
GLM-5.2in its codenot stated
topk_method=noaux_tc

The router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method "noaux_tc".

topk_method = "noaux_tc"
scores_for_choice = scores + self.e_score_correction_bias
modeling_glm_moe_dsa.py · L486
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are gathered from the unbiased scores).

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_nemotron_h.py · L773–774
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
MiniMax-M3in its codenot stated
use_routing_bias=true

The router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores.

text_config.use_routing_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L235–237
Step-3.7-Flashin its codenot stated
use_moe_router_bias=true

The router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts.

text_config.use_moe_router_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_step3p7.py · L595–597
DeepSeek-V4-Flashin its codeadopted
topk_method=noaux_tc

The TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
        weights = weights / (weights.sum(dim=-1, keepdim=True) + 1e-20)
modeling_deepseek_v4.py · L1094–1096
DeepSeek-V4-Proin its codeadopted
topk_method=noaux_tc

The router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching DeepSeek-V3's bias-based auxiliary-loss-free balancing, enabled via topk_method noaux_tc.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
MiMo-V2.5in its codenot stated
topk_method=noaux_tc

The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
MiMo-V2.5-Proin its codenot stated
topk_method=noaux_tc

The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L159
DeepSeek-V3.2in its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights come from the unbiased scores.

topk_method = "noaux_tc"
scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_deepseek_v32.py · L512–513
Mistral-Large-4not in its codenot stated
verdict and lines

The router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned per-expert selection bias (no e_score_correction_bias in config or code).

scores_for_choice = scores.masked_fill(~score_mask.bool(), 0.0)
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
modeling_mistral_large4.py · L542–543
LFM3-1.7Bnot in its codenot stated
verdict and lines

No router or expert-selection bias exists since the model has no mixture-of-experts layers.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load balancing cannot be confirmed.

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

The router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to the scores.

router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
        router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen4_exp.py · L959–960
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No router or e_score_correction_bias exists since the model has no mixture-of-experts feed-forward layers.

MolmoAct 2not in its codenot stated
verdict and lines

No router or e_score_correction_bias mechanism exists since the model has no MoE layers.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

The router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism.

router_logits = F.linear(hidden_states, self.weight)  # (seq_len, num_experts)
        router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
modeling_qwen3_5_moe.py · L875–876
Gemma 4 31Bnot in its codenot stated
verdict and lines

The router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway.

top_k_weights, top_k_index = torch.topk(
            router_probabilities,
            k=self.config.top_k_experts,
            dim=-1,
        )  # both [B*S, K]
modeling_gemma4.py · L1324–1328
Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

The router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias added to logits.

router_logits = F.linear(hidden_states, self.weight)  # (seq_len, num_experts)
        router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
modeling_qwen3_5_moe.py · L875–876
X-VLAnot in its codenot stated
verdict and lines

No router or expert-selection bias mechanism exists since the model has no mixture-of-experts layers.

gpt-oss-120bnot in its codenot stated
verdict and lines

The router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present

router_logits = F.linear(hidden_states, self.weight, self.bias)  # (num_tokens, num_experts)
        router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L126–127
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No topk_method, e_score_correction_bias, or router bias mechanism is present since there is no MoE router in this model.

In this branch 25

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 25

Auxiliary-loss-free load balancing core · 3 sources · 4 quotes
Quantile Balancing core · 3 sources · 5 quotes
Exact Quantile Balancing core · 2 sources · 3 quotes
Load-error gradient injection core · 2 sources · 3 quotes
Load balancing used · 2 sources · 2 quotes
20 from one source, or only mentioned as not used
Modality-specific load balancing core · 1 source · 2 quotes
Per-expert routing correction bias core · 1 source · 1 quote
Auxiliary-loss load balancing used · 1 source · 1 quote
Disabled router-bias updates used · 1 source · 1 quote
Exact coordinate minimization used · 1 source · 1 quote
Expert bias update factor used · 1 source · 1 quote
Expert Parallelism Load Balancing used · 1 source · 1 quote
Expert-parallel-group load-balancing loss used · 1 source · 1 quote
Histogram-based global quantile estimation used · 1 source · 1 quote
Mixture-of-experts load balancing used · 1 source · 2 quotes
Persistent load balancing used · 1 source · 1 quote
Pooled-global-batch quantile estimation used · 1 source · 1 quote
Recentered global-quantile target update used · 1 source · 1 quote
Two-pass radix selection used · 1 source · 1 quote
Uniform-bin histogram quantile estimation used · 1 source · 1 quote
Round-robin load balancing optional · 1 source · 1 quote
MaxVio evaluated · 1 source · 1 quote
Load-balancing regularization not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑