specific method · filed under model architecture
Auxiliary-loss-free load balancing
A load-balancing approach distinguished from auxiliary-loss-based routing by using expert-wise correction biases rather than an auxiliary balancing loss.
Also called auxiliary-loss-free strategy, Auxiliary-loss-free MoE load balancing, Auxiliary-loss-free routing.
- sources
- 3
- models
- 4
- labs adopt it
- 2
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 3core 2
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
In the code
Checked as “Auxiliary-loss-free load balancing (selection bias)” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | The router adds e_score_correction_bias to sigmoid scores solely for top-k expert selection (topk_method noaux_tc), matching DeepSeek-V3's bias-based load balancing. modeling_glm5_next.py#L162 |
| DeepSeek-V4.1-Flash | in its code | core | topk_method is 'noaux_tc' (DeepSeek-V3 aux-loss-free scheme) and the router's e_score_correction_bias is loaded and mapped from the gate's bias in the weight loader. model.py#L1179 |
| Hy4-preview | in its code | not stated | The router adds an e_score_correction_bias buffer to the sigmoid scores solely for top-k expert selection, while the actual output weighting uses the unbiased scores gathered afterward, DeepSeek-V3 style. modeling_hy_v4.py#L528 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | The router adds e_score_correction_bias to the scores only for top-k selection (topk_method noaux_tc), while the weights used for output combination come from the unbiased scores. modeling_deepseek_v4.py#L1086 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not stated | The router adds e_score_correction_bias to the sigmoid scores solely to determine top-k expert selection (via topk_group/group masking), while the unbiased scores are gathered for the actual output weights. modeling_nemotron_h.py#L743 |
| MiMo-V2.6-Flash | in its code | not stated | The router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores. modeling_mimo_v2.py#L164 |
| DeepSeek-V4-Flash | in its code | used | The router adds e_score_correction_bias to scores solely for top-k selection (not for the output weights, which are gathered from the unbiased scores), matching DeepSeek-V3's noaux_tc method. modeling_deepseek_v4.py#L1086 |
| MiMo-V2.5 | in its code | not stated | The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores. modeling_mimo_v2.py#L164 |
| GLM-5.3 | in its code | not stated | The router adds an e_score_correction_bias buffer to sigmoid scores only for top-k/group selection (topk_method noaux_tc), while the actual weights used for combining are taken from the unbiased scores. modeling_glm_moe_dsa.py#L498 |
| Hy3 | in its code | not stated | The router adds an e_score_correction_bias to routing scores only for top-k selection, not for the output weighting, matching moe_router_enable_expert_bias: true modeling_hy_v3.py#L299 |
| GLM-5.2 | in its code | not stated | GlmMoeDsaTopkRouter adds e_score_correction_bias to sigmoid scores only to select the top-k/top-group experts (topk_method 'noaux_tc'), not to weight outputs, which are gathered from the un-biased scores. modeling_glm_moe_dsa.py#L498 |
| MiniMax-M3 | in its code | not stated | The router adds an e_score_correction_bias only to the scores used for top-k selection, not to the routing weights used for output scaling, matching DeepSeek-V3 style bias. modeling_minimax_m3_vl.py#L242 |
| DeepSeek-V3.2 | in its code | not stated | The router adds e_score_correction_bias to sigmoid scores only for top-k expert selection (topk_method noaux_tc), matching DeepSeek-V3's bias-based load balancing. modeling_deepseek_v32.py#L526 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | The router adds an e_score_correction_bias buffer to scores only for top-k selection (not for the weights used afterward), matching DeepSeek-V3's noaux_tc scheme, and topk_method is set to noaux_tc in this config. modeling_deepseek_v4.py#L1086 |
| DeepSeek-V4-Pro | in its code | used | The router adds an e_score_correction_bias buffer to scores only for top-k expert selection, matching DeepSeek-V3's noaux_tc scheme, and topk_method is set to noaux_tc modeling_deepseek_v4.py#L1080 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | The TopKRouter adds e_score_correction_bias only to the scores used for top-k selection (not to the output weights), matching DeepSeek-V3's noaux_tc bias mechanism, with topk_method set to noaux_tc. modeling_deepseek_v4.py#L1086 |
| Gemma 4 31B | not in its code | not stated | The router uses a plain softmax + topk with a per-expert output scale, not a selection-only bias like e_score_correction_bias, and no such config key is present. modeling_gemma4.py#L1340 |
| Inkling | in its code | not stated | The router adds a per-expert selection bias (use_gate_bias / self.bias) to the sigmoid scores used only for top-k selection, not for output weighting, matching aux-loss-free bias-based balancing. moe.py#L110 |
| Kimi K3 | in its code | core | KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism. modeling_kimi_linear.py#L693 |
| Laguna-S-2.1 | in its code | not stated | The router adds e_score_correction_bias only to the scores used for top-k selection, not to the routing weights used to combine expert outputs. modeling_laguna.py#L180 |
| MiMo-V2.5-Pro | in its code | not stated | The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting. modeling_mimo_v2.py#L159 |
| MiMo-V2.6-Pro | in its code | not stated | The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores. modeling_mimo_v2.py#L164 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | The router adds e_score_correction_bias to sigmoid scores only for selecting top-k experts, while output weighting uses the unbiased scores. modeling_nemotron_h.py#L743 |
| Qwen3.5-397B-A17B | not in its code | not stated | The router only uses softmax + topk with no e_score_correction_bias or expert-selection bias term; load balancing uses the classic auxiliary loss function instead. modeling_qwen3_5_moe.py#L896 |
| Qwen3.6-35B-A3B | not in its code | not stated | The router computes plain softmax top-k without any e_score_correction_bias-style selection bias; load balancing instead uses an auxiliary loss (router_aux_loss_coef). modeling_qwen3_5_moe.py#L896 |
| Qwen3.8-Flash-Next | not in its code | not stated | The router uses plain softmax top-k selection with no e_score_correction_bias or similar selection-only bias term; only an auxiliary load-balancing loss coefficient exists. |
| Step-3.7-Flash | in its code | not stated | The router adds a learned e_score_correction_bias to routing scores only for the top-k selection (not for the weighting), matching DeepSeek-V3-style bias-based balancing, enabled via use_moe_router_bias. modeling_step3p7.py#L605 |
| gpt-oss-120b | not in its code | not stated | The router uses a standard softmax top-k with a learned bias term added before topk, but there is no e_score_correction_bias style selection-only bias; instead an auxiliary load-balancing loss (router_aux_loss_coef) is used for training. modeling_gpt_oss.py#L127 |
Evidence
5 spans quoted from the sources, strongest treatment first.
Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].
coremodel architecturein Kimi K3Moonshot AI
we extend auxiliary-loss-free load balancing (Wang et al., 2024a)
coreoptimizationin DeepSeek-V4.1-FlashDeepSeek
we also employ the auxiliary-loss-free strategy
usedtraining objectivein DeepSeek-V4DeepSeek
For auxiliary-loss-free load balancing, we set the bias update speed to 0.001.
usedmodel architecturein DeepSeek-V4DeepSeek
For auxiliary-loss-free load balancing, we set the bias update speed to 0.001 for both image and text tokens, while retaining a small sequence-level balance loss with a loss weight of 0.0001 to avoid extreme imbalance within single sequences.
usedunclearin DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: expert load balancing.
Quantile BalancingHistogram-based quantile estimationAuxiliary-loss load balancingExact coordinate minimizationExpert bias update factorExpert Parallelism Load BalancingExponential moving average of estimated quantilesformHCLoad balancingMaxVioModality-specific auxiliary-loss-free load balancingPersistent load balancingPooled-global-batch quantile estimationRound-robin load balancing