methods · mixture of experts
expert load balancing
25 methods filed at this node or below it, from the sources of 14 models.
In practice
Checked as “Auxiliary-loss-free load balancing (selection bias)” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| MiniMax-M3.1-Flash | in its code | not stated | use_routing_bias=trueThe router adds a learned e_score_correction_bias to the sigmoid scores used only for top-k selection, not for the final weighting. text_config.use_routing_bias = true | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | topk_method=noaux_tctopk_method is set to noaux_tc (DeepSeek-V3-style auxiliary-loss-free balancing) and the loader maps a gate bias to e_score_correction_bias used only for expert selection. text_config.topk_method = "noaux_tc" | code → |
| Step-5-Preview | in its code | not stated | use_moe_router_bias=trueA per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true. text_config.use_moe_router_bias = true | code → |
| Kolibri-1 | in its code | not stated | verdict and linesThe router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits). | code → |
| MiMo-V2.6-Flash | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| MiMo-V2.6-Pro | in its code | not stated | topk_method=noaux_tcThe MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | topk_method=noaux_tctopk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced in weight remapping. text_config.topk_method = "noaux_tc" | code → |
| Hy4-preview | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style. | code → |
| GLM-5.3-Flash | in its code | not stated | topk_method=noaux_tcGlm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc. text_config.topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | topk_method=noaux_tcThe TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method. topk_method = "noaux_tc" | code → |
| GLM-5.3 | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (topk_method "noaux_tc"), not for weighting outputs (weights are gathered from the un-biased scores). topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | topk_method=noaux_tcThe router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method topk_method = "noaux_tc" | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores. | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | topk_method=noaux_tcThe router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method. topk_method = "noaux_tc" | code → |
| Laguna-S-2.1 | in its code | not stated | verdict and linesThe router adds a learned e_score_correction_bias to routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias-based balancing. | code → |
| Kimi K3 | in its code | adopted | topk_method=noaux_tcKimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism. text_config.topk_method = "noaux_tc" | code → |
| Inkling | in its code | not stated | use_gate_bias=trueThe gate adds a learned per-expert bias to the sigmoid routing scores used only for top-k selection, controlled by use_gate_bias=true (DeepSeek-style selection-bias load balancing). text_config.use_gate_bias = true | code → |
| Hy3 | in its code | not stated | moe_router_enable_expert_bias=trueThe router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing. moe_router_enable_expert_bias = true | code → |
| GLM-5.2 | in its code | not stated | topk_method=noaux_tcThe router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method "noaux_tc". topk_method = "noaux_tc" | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are gathered from the unbiased scores). | code → |
| MiniMax-M3 | in its code | not stated | use_routing_bias=trueThe router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores. text_config.use_routing_bias = true | code → |
| Step-3.7-Flash | in its code | not stated | use_moe_router_bias=trueThe router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts. text_config.use_moe_router_bias = true | code → |
| DeepSeek-V4-Flash | in its code | adopted | topk_method=noaux_tcThe TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method. topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Pro | in its code | adopted | topk_method=noaux_tcThe router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching DeepSeek-V3's bias-based auxiliary-loss-free balancing, enabled via topk_method noaux_tc. topk_method = "noaux_tc" | code → |
| MiMo-V2.5 | in its code | not stated | topk_method=noaux_tcThe gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| MiMo-V2.5-Pro | in its code | not stated | topk_method=noaux_tcThe MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting. topk_method = "noaux_tc" | code → |
| DeepSeek-V3.2 | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights come from the unbiased scores. topk_method = "noaux_tc" | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThe router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned per-expert selection bias (no e_score_correction_bias in config or code). | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo router or expert-selection bias exists since the model has no mixture-of-experts layers. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load balancing cannot be confirmed. | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesThe router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to the scores. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo router or e_score_correction_bias exists since the model has no mixture-of-experts feed-forward layers. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo router or e_score_correction_bias mechanism exists since the model has no MoE layers. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesThe router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesThe router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesThe router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias added to logits. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo router or expert-selection bias mechanism exists since the model has no mixture-of-experts layers. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesThe router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo topk_method, e_score_correction_bias, or router bias mechanism is present since there is no MoE router in this model. | code → |
In this branch 25
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 25
20 from one source, or only mentioned as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.