Kimi K3
Moonshot AIUnlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].
specific method · filed under model architecture
Auxiliary-loss-free load balancing balances expert load in MoE routing without an auxiliary loss.1 It adds an expert-specific bias to the router scores used for top-k selection, but omits the bias from the mixture weights so router gradients are unchanged.2,3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30— k3 tech report
Load balancing is imple- mented by adding an expert-specific bias bj to the router score used for Top- k selection— k3 tech report
Because b is omitted from pi,j, it regulates dispatch without altering the mixture weights or the gradient-based optimization of the router— k3 tech report
Also called auxiliary-loss-free strategy, Auxiliary-loss-free MoE load balancing, Auxiliary-loss-free routing.
Checked as “Auxiliary-loss-free load balancing (selection bias)” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| MiniMax-M3.1-Flash | in its code | not stated | use_routing_bias=trueThe router adds a learned e_score_correction_bias to the sigmoid scores used only for top-k selection, not for the final weighting. text_config.use_routing_bias = true | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | topk_method=noaux_tctopk_method is set to noaux_tc (DeepSeek-V3-style auxiliary-loss-free balancing) and the loader maps a gate bias to e_score_correction_bias used only for expert selection. text_config.topk_method = "noaux_tc" | code → |
| Step-5-Preview | in its code | not stated | use_moe_router_bias=trueA per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true. text_config.use_moe_router_bias = true | code → |
| Kolibri-1 | in its code | not stated | verdict and linesThe router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits). | code → |
| MiMo-V2.6-Flash | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| MiMo-V2.6-Pro | in its code | not stated | topk_method=noaux_tcThe MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | topk_method=noaux_tctopk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced in weight remapping. text_config.topk_method = "noaux_tc" | code → |
| Hy4-preview | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style. | code → |
| GLM-5.3-Flash | in its code | not stated | topk_method=noaux_tcGlm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc. text_config.topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | topk_method=noaux_tcThe TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method. topk_method = "noaux_tc" | code → |
| GLM-5.3 | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (topk_method "noaux_tc"), not for weighting outputs (weights are gathered from the un-biased scores). topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | topk_method=noaux_tcThe router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method topk_method = "noaux_tc" | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores. | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | topk_method=noaux_tcThe router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method. topk_method = "noaux_tc" | code → |
| Laguna-S-2.1 | in its code | not stated | verdict and linesThe router adds a learned e_score_correction_bias to routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias-based balancing. | code → |
| Kimi K3 | in its code | adopted | topk_method=noaux_tcKimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism. text_config.topk_method = "noaux_tc" | code → |
| Inkling | in its code | not stated | use_gate_bias=trueThe gate adds a learned per-expert bias to the sigmoid routing scores used only for top-k selection, controlled by use_gate_bias=true (DeepSeek-style selection-bias load balancing). text_config.use_gate_bias = true | code → |
| Hy3 | in its code | not stated | moe_router_enable_expert_bias=trueThe router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing. moe_router_enable_expert_bias = true | code → |
| GLM-5.2 | in its code | not stated | topk_method=noaux_tcThe router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method "noaux_tc". topk_method = "noaux_tc" | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not stated | verdict and linesThe router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are gathered from the unbiased scores). | code → |
| MiniMax-M3 | in its code | not stated | use_routing_bias=trueThe router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores. text_config.use_routing_bias = true | code → |
| Step-3.7-Flash | in its code | not stated | use_moe_router_bias=trueThe router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts. text_config.use_moe_router_bias = true | code → |
| DeepSeek-V4-Flash | in its code | adopted | topk_method=noaux_tcThe TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method. topk_method = "noaux_tc" | code → |
| DeepSeek-V4-Pro | in its code | adopted | topk_method=noaux_tcThe router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching DeepSeek-V3's bias-based auxiliary-loss-free balancing, enabled via topk_method noaux_tc. topk_method = "noaux_tc" | code → |
| MiMo-V2.5 | in its code | not stated | topk_method=noaux_tcThe gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores. topk_method = "noaux_tc" | code → |
| MiMo-V2.5-Pro | in its code | not stated | topk_method=noaux_tcThe MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting. topk_method = "noaux_tc" | code → |
| DeepSeek-V3.2 | in its code | not stated | topk_method=noaux_tcThe router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights come from the unbiased scores. topk_method = "noaux_tc" | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThe router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned per-expert selection bias (no e_score_correction_bias in config or code). | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo router or expert-selection bias exists since the model has no mixture-of-experts layers. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load balancing cannot be confirmed. | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesThe router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to the scores. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo router or e_score_correction_bias exists since the model has no mixture-of-experts feed-forward layers. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo router or e_score_correction_bias mechanism exists since the model has no MoE layers. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesThe router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesThe router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesThe router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias added to logits. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo router or expert-selection bias mechanism exists since the model has no mixture-of-experts layers. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesThe router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo topk_method, e_score_correction_bias, or router bias mechanism is present since there is no MoE router in this model. | code → |
4 quotes from 3 sources, one card per model, each opening with its strongest.
Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].
we adopt an auxiliary-loss-free strategy inspired by DeepSeek-V3 [19] to promote load balancing within the MoE architecture.
For auxiliary-loss-free load balancing, we set the bias update speed to 0.001.
we also employ the auxiliary-loss-free strategy
Picked by hand, not extracted: where to read more, not evidence for anything on this page.