Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

specific method · filed under model architecture

Auxiliary-loss-free load balancing

Auxiliary-loss-free load balancing balances expert load in MoE routing without an auxiliary loss.1 It adds an expert-specific bias to the router scores used for top-k selection, but omits the bias from the mixture weights so router gradients are unchanged.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30 — k3 tech report
  2. Load balancing is imple- mented by adding an expert-specific bias bj to the router score used for Top- k selection — k3 tech report
  3. Because b is omitted from pi,j, it regulates dispatch without altering the mixture weights or the gradient-based optimization of the router — k3 tech report

Also called auxiliary-loss-free strategy, Auxiliary-loss-free MoE load balancing, Auxiliary-loss-free routing.

sources
3
models
4
labs adopt it
3
strongest
core

In practice

Checked as “Auxiliary-loss-free load balancing (selection bias)” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
MiniMax-M3.1-Flashin its codenot stated
use_routing_bias=true

The router adds a learned e_score_correction_bias to the sigmoid scores used only for top-k selection, not for the final weighting.

text_config.use_routing_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L235–237
DeepSeek-V4.1-Flash-1017in its codenot stated
topk_method=noaux_tc

topk_method is set to noaux_tc (DeepSeek-V3-style auxiliary-loss-free balancing) and the loader maps a gate bias to e_score_correction_bias used only for expert selection.

text_config.topk_method = "noaux_tc"
if name.endswith(".ffn.gate.bias"):
                    name = name.replace(
                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
                    )
dspark.py · L519–522
Step-5-Previewin its codenot stated
use_moe_router_bias=true

A per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true.

text_config.use_moe_router_bias = true
self.router_bias = nn.Parameter(
            torch.zeros(config.moe_num_experts, dtype=torch.float32),
            requires_grad=False,
        )
model.py · L1710–1713
e_score_correction_bias=self.router_bias,
            routed_scaling_factor=config.moe_router_scaling_factor,
model.py · L1777–1778
Kolibri-1in its codenot stated
verdict and lines

The router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits).

        self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_kolibri1.py · L273
        router_indices = torch.topk(router_logits + self.e_score_correction_bias, self.top_k, dim=-1)[1]
        router_scores = router_logits.gather(1, router_indices).sigmoid()
modeling_kolibri1.py · L288–289
MiMo-V2.6-Flashin its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to scores only for top-k selection (noaux_tc topk_method), while topk_weight for weighting is gathered from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
MiMo-V2.6-Proin its codenot stated
topk_method=noaux_tc

The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method=noaux_tc) while the actual weighting uses the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L176
DeepSeek-V4.1-Flashin its codenot stated
topk_method=noaux_tc

topk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced in weight remapping.

text_config.topk_method = "noaux_tc"
                if name.endswith(".ffn.gate.bias"):
                    name = name.replace(
                        ".ffn.gate.bias", ".ffn.gate.e_score_correction_bias"
                    )
dspark.py · L519–522
Hy4-previewin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_hy_v4.py · L515–516
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L531–532
GLM-5.3-Flashin its codenot stated
topk_method=noaux_tc

Glm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc.

text_config.topk_method = "noaux_tc"
        scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_glm5_next.py · L160–161
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm5_next.py · L176–177
DeepSeek-V4-Flash-Vision-Expin its codenot stated
topk_method=noaux_tc

The TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method.

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
GLM-5.3in its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (topk_method "noaux_tc"), not for weighting outputs (weights are gathered from the un-biased scores).

topk_method = "noaux_tc"
scores_for_choice = scores + self.e_score_correction_bias
modeling_glm_moe_dsa.py · L486
topk_weights = scores.gather(1, topk_indices)
modeling_glm_moe_dsa.py · L502
DeepSeek-V4-Pro-0813in its codenot stated
topk_method=noaux_tc

The router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
        group_scores = (
            scores_for_choice.view(-1, self.num_group, self.num_experts // self.num_group)
            .topk(2, dim=-1)[0]
            .sum(dim=-1)
        )
modeling_nemotron_h.py · L773–779
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
DeepSeek-V4-Flash-0731in its codenot stated
topk_method=noaux_tc

The router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Laguna-S-2.1in its codenot stated
verdict and lines

The router adds a learned e_score_correction_bias to routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias-based balancing.

scores_for_selection = routing_scores + self.e_score_correction_bias.to(routing_scores.dtype)
        _, selected_experts = torch.topk(scores_for_selection, self.top_k, dim=-1)
        routing_weights = routing_scores.gather(-1, selected_experts)
modeling_laguna.py · L169–171
Kimi K3in its codeadopted
topk_method=noaux_tc

KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism.

text_config.topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Parameter(
            torch.empty(self.num_experts),
        )
modeling_kimi_linear.py · L693–695
scores_for_choice = scores + self.e_score_correction_bias.unsqueeze(0)
modeling_kimi_linear.py · L723
Inklingin its codenot stated
use_gate_bias=true

The gate adds a learned per-expert bias to the sigmoid routing scores used only for top-k selection, controlled by use_gate_bias=true (DeepSeek-style selection-bias load balancing).

text_config.use_gate_bias = true
sel = tl.where(mask_r, tl.sigmoid(logits), float("-inf"))
    if HAS_BIAS:
        bias = tl.load(bias_ptr + offs, mask=mask_r, other=0.0).to(tl.float32)
        sel = tl.where(mask_r, sel + bias, float("-inf"))
moe.py · L110–113
Hy3in its codenot stated
moe_router_enable_expert_bias=true

The router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing.

moe_router_enable_expert_bias = true
scores_for_choice = routing_weights + e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_hy_v3.py · L288–290
GLM-5.2in its codenot stated
topk_method=noaux_tc

The router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method "noaux_tc".

topk_method = "noaux_tc"
scores_for_choice = scores + self.e_score_correction_bias
modeling_glm_moe_dsa.py · L486
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot stated
verdict and lines

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are gathered from the unbiased scores).

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_nemotron_h.py · L773–774
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
MiniMax-M3in its codenot stated
use_routing_bias=true

The router adds e_score_correction_bias to the sigmoid scores solely to pick top-k experts, while actual routing weights are taken from the unbiased scores.

text_config.use_routing_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_minimax_m3_vl.py · L235–237
Step-3.7-Flashin its codenot stated
use_moe_router_bias=true

The router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts.

text_config.use_moe_router_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_step3p7.py · L595–597
DeepSeek-V4-Flashin its codeadopted
topk_method=noaux_tc

The TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
        weights = weights / (weights.sum(dim=-1, keepdim=True) + 1e-20)
modeling_deepseek_v4.py · L1094–1096
DeepSeek-V4-Proin its codeadopted
topk_method=noaux_tc

The router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching DeepSeek-V3's bias-based auxiliary-loss-free balancing, enabled via topk_method noaux_tc.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
MiMo-V2.5in its codenot stated
topk_method=noaux_tc

The gate adds e_score_correction_bias to scores only for top-k selection (noaux_tc), while the actual weights used to combine expert outputs come from the unbiased sigmoid scores.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L164
_, topk_idx = torch.topk(tmp_scores, k=self.top_k, dim=-1, sorted=False)
            topk_weight = scores.gather(1, topk_idx)
modeling_mimo_v2.py · L175–176
MiMo-V2.5-Proin its codenot stated
topk_method=noaux_tc

The MoE gate adds e_score_correction_bias to routing scores only for top-k selection (topk_method noaux_tc), not for weighting.

topk_method = "noaux_tc"
scores_for_choice = scores.view(bsz * seq_len, -1) + self.e_score_correction_bias.unsqueeze(0)
modeling_mimo_v2.py · L159
DeepSeek-V3.2in its codenot stated
topk_method=noaux_tc

The router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights come from the unbiased scores.

topk_method = "noaux_tc"
scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_deepseek_v32.py · L512–513
Mistral-Large-4not in its codenot stated
verdict and lines

The router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned per-expert selection bias (no e_score_correction_bias in config or code).

scores_for_choice = scores.masked_fill(~score_mask.bool(), 0.0)
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
modeling_mistral_large4.py · L542–543
LFM3-1.7Bnot in its codenot stated
verdict and lines

No router or expert-selection bias exists since the model has no mixture-of-experts layers.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load balancing cannot be confirmed.

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

The router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to the scores.

router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
        router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen4_exp.py · L959–960
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No router or e_score_correction_bias exists since the model has no mixture-of-experts feed-forward layers.

MolmoAct 2not in its codenot stated
verdict and lines

No router or e_score_correction_bias mechanism exists since the model has no MoE layers.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

The router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism.

router_logits = F.linear(hidden_states, self.weight)  # (seq_len, num_experts)
        router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
modeling_qwen3_5_moe.py · L875–876
Gemma 4 31Bnot in its codenot stated
verdict and lines

The router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway.

top_k_weights, top_k_index = torch.topk(
            router_probabilities,
            k=self.config.top_k_experts,
            dim=-1,
        )  # both [B*S, K]
modeling_gemma4.py · L1324–1328
Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

The router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias added to logits.

router_logits = F.linear(hidden_states, self.weight)  # (seq_len, num_experts)
        router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
modeling_qwen3_5_moe.py · L875–876
X-VLAnot in its codenot stated
verdict and lines

No router or expert-selection bias mechanism exists since the model has no mixture-of-experts layers.

gpt-oss-120bnot in its codenot stated
verdict and lines

The router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present

router_logits = F.linear(hidden_states, self.weight, self.bias)  # (num_tokens, num_experts)
        router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L126–127
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No topk_method, e_score_correction_bias, or router bias mechanism is present since there is no MoE router in this model.

Documented in

Moonshot AI
Robbyant (Ant Group)

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 2

Kimi K3

Moonshot AIadopted1 source

Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].

coremodel architecturein Kimi K3Moonshot AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑