Open Models Almanac

Channel mixer · C3

Expert load balancing

Balance over the whole batch lets experts specialize and leaves GPUs waiting; balance inside each micro-batch is faster and costs quality; a selection bias drops the loss and can quietly override the router

Illustration of a row of identical wooden counters receding in perspective, each with a short, equal queue of round off-white tokens. Above each counter a gold weight hangs from a long ink-blue rail, each at a different height. One counter near the front has a coral lamp lit over it, and a group of teal tokens waits in its queue.

During reinforcement learning on MiMo-V2.6-Pro, the busiest expert in one layer went from 6 times the mean load to 16 times in 20 training steps, and the share of experts below a tenth of the mean rose from 0.5% to 22% (V2.6 report §5.4). The same report’s reinforcement-learning runs also died out of memory, from a different problem: “an expert-parallel (EP) rank received over 30× the mean token load despite a relatively balanced full batch”.

The first failure is a router abandoning experts. The second is a router that looks fine over a whole batch and still swamps one GPU for one step. Balancing methods differ in which tokens they count over and in how they act on the count, and the measured differences follow the first of those more.

Background

A mixture-of-experts (MoE) layer replaces a transformer block’s feed-forward unit with EE smaller ones, the experts FiF_i, of which each token uses kk (C2 builds it). A router gives token xx one logit uiu_i per expert, a score function hh (a sigmoid in most models here, a softmax in some) makes scores si=h(ui)s_i = h(u_i), and a per-expert bias bib_i joins them only to choose the set:

T(x)=top-k of (si+bi),y=∑i∈T(x)giFi(x)+S(x),gi=ρ si∑l∈T(x)sl.T(x) = \text{top-}k \text{ of } (s_i + b_i), \qquad y = \sum_{i \in T(x)} g_i F_i(x) + S(x), \qquad g_i = \frac{\rho\, s_i}{\sum_{l \in T(x)} s_l}.

Here ρ\rho is a constant and SS an optional shared expert every token uses. The bias stays out of the weights gig_i, and an expert trains only on tokens that select it. Sizes such as “1B” count parameters.

Under expert parallelism (EP), experts sit on different GPU processes, called ranks, and tokens travel to them in an all-to-all exchange. Most systems here are dropless: an overloaded expert takes all its tokens, so overload becomes waiting or an out-of-memory error rather than lost tokens.

Figure 1.One MoE layer under expert parallelism. The bias takes part in choosing experts and not in weighting them. Ranks exchange tokens synchronously, so the layer waits for the busiest one, and only the routed part of the step waits.

Three batch sizes matter. A sequence is one row of a batch, often several documents packed together. A micro-batch is what goes through one forward and backward pass together: 1 to 4 sequences per GPU in Qiu et al. (2025), about one per rank in Kimi K3’s design, going by its S×KS \times K tokens per rank (our reading). The global batch is everything behind one optimizer step, all ranks and all accumulation passes. Waiting happens per micro-batch and per layer, while a global-batch bias changes once per optimizer step.

Balance has a standard measure, MaxVio. For cic_i tokens assigned to expert ii out of BB tokens,

MaxVio=max⁡icikB/E−1.\text{MaxVio} = \max_i \frac{c_i}{kB/E} - 1.

Zero means even load, and 1 means the busiest expert has twice its share. The ceiling is E/k−1E/k - 1: 55 for K3 (E=896E = 896, k=16k = 16), computed. Wang et al. (2024) define it. Counted over a global batch it is Global MaxVio, over one micro-batch Local MaxVio, and a report may average over layers or take the worst, so each number below says which. It looks only at the top; dead experts need another count, such as MiMo’s share below a tenth of the mean.

Expert imbalance reaches the clock through three links. A rank’s load sums over the experts it hosts, so the excess shrinks with more experts per rank unless hot experts sit together: Aleph Alpha’s Kolibri pretrains with 48 per rank (384 over its EP 8), and DeepSeek-V3’s decode has one. Only the routed part of the step waits, and with all experts on one device nothing does. And each layer waits on its own busiest rank, so step time goes as a sum of per-layer maxima (our reading of the last two).

Why a router collapses

Shazeer et al. (2017), who made the gate sparse, saw it first: “This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network.” With no balancing their model reached a test perplexity (a held-out loss, lower is better) of 39.8 with the busiest expert at 17.80 times the mean. With any of their balancing losses it reached 35.6 to 35.7, and at most 1.47 times.

Later models show less of a quality gap: on Wang’s 1B model, no balancing leaves MaxVio near 5 and perplexity near 9.54, read off their figure. The other costs remain. A dead expert wastes memory (Marin, an open training project: “Many experts immediately die out”), a hot one stalls its rank, and under dropless EP buffers must fit the peak: K3’s report gives S×K×RS \times K \times R as the worst case for a copy-free buffer in DeepEP, for sequence length SS, top-KK and RR ranks, where even load needs S×KS \times K.

Kolibri’s report states the two goals together: “Load balancing ensures sufficient learning signal for every expert and efficient expert-parallel execution. The former requires balance across the global optimiser-step batch; the latter requires balance within each local microbatch.”

The auxiliary loss

The first fix adds a loss. Switch Transformers (Fedus et al., 2021) write it as

Laux=α E∑i=1EfiPi,L_{\text{aux}} = \alpha\, E \sum_{i=1}^{E} f_i P_i,

where fif_i is the fraction of tokens routed to expert ii and PiP_i the mean router probability for it. The count passes no gradient, so PiP_i carries the signal. GShard (Lepikhin et al., 2020) gives the reason for the shape: “we want to minimize mean square of ce/Sc_e/S”, with one factor replaced by the differentiable mean gate. Since E∑fi2=1+CV2(f)E\sum f_i^2 = 1 + \text{CV}^2(f) when the fif_i sum to one, as at Switch’s top-1, this is a surrogate for the squared coefficient of variation Shazeer penalized.

Switch says the loss “is minimized under a uniform distribution”. As a joint function of ff and PP that’s loose, and the exact statement is about the gradient training uses. With ff constant and softmax probabilities pp over BB tokens (derivation ours),

∂∂ut,j∑ifiPi=1B pt,j(fj−∑ifi pt,i).\frac{\partial}{\partial u_{t,j}} \sum_i f_i P_i = \frac{1}{B}\, p_{t,j}\Big(f_j - \sum_i f_i\, p_{t,i}\Big).

Each token moves probability off experts more loaded than its own probability-weighted average. When ff is uniform the bracket is zero whatever PP is, so the loss asks for even counts, not flat probabilities, and goes silent once counts are even. Neitemeier et al. (2026) state this as the straight-through gradient of the squared distance between load shares and uniform: the backward pass treats the count as if it were PP. It needs probabilities that sum to one. On a raw sigmoid (Wang’s Eq. 2) the gradient is fj σ′(ut,j)/Bf_j\,\sigma'(u_{t,j})/B, which pushes every score down even at even load; DeepSeek-V3’s Eq. 19 normalizes to remove this.

Constants don’t carry between papers, since the forms differ by factors of EE and kk. Switch “found 10−210^{-2} balanced load quickly without interfering with training loss” on a top-1 softmax router. Wang’s 1B model, top-6 sigmoid, shows a cost: read off their figure, α\alpha of 0, 10−410^{-4}, 10−310^{-3} and 10−210^{-2} give MaxVio near 5.1, 2.2, 0.72 and 0.1, and perplexity near 9.54, 9.53, 9.56 and 9.69. Whether the raw-sigmoid term explains that last step is untested.

Later models often keep a loss as a safety net. DeepSeek-V3 sets α=0.0001\alpha = 0.0001 “just to avoid extreme imbalance within any single sequence”, and MiMo-V2-Flash’s pretraining uses 10−510^{-5}. No report removes it to measure what it does. A loss also promises no bound, so early systems capped each expert and let overflow tokens skip the layer, “typically < 1%” of them in Switch. DeepSeek-V3 “does not drop any tokens during training”. The inference code we can read has no cap, which says nothing about training stacks.

What you count over

The term fif_i needs a set of tokens, and the choice turned out to matter more than the coefficient. Qiu et al. (2025) trained a 3.4B model on 120B tokens with sequences of 4,096 tokens, counting over different numbers of sequences:

Rule, sequences counted overAvg. perplexityMMLU (a benchmark, higher is better)
Loss, micro-batch (4)8.16741.63
Loss, partial (32)8.06242.08
Loss, global batch (512)8.03843.48
Bias, micro-batch (4)8.52141.30
Bias, global batch (512)8.08042.74

The bias rows anticipate the next section, and use a sigmoid gate where the loss rows use softmax. Three coefficients from 0.001 to 0.008 at micro-batch scope stayed between 8.154 and 8.167. In a 400B-token run, going from 2 sequences to 8 gained more than raising kk from 4 to 6 did at 50% more active expert FLOPs.

DeepSeek-V3’s report built a batch-wise auxiliary loss to separate scope from channel: validation losses on 1B were 2.258 for a sequence-wise loss, 2.253 for the bias and 2.253 for the batch-wise loss, and on 3B 2.085, 2.080 and 2.080. “The key distinction … lies in their balancing scope.” So at batch scope the channel hardly matters in these two tests (Qiu’s 8.038 and 8.080 differ in gate too). At small scope it does, and the bias is the worst row.

An identity shows why tuning can’t catch up (ours). For MM equal micro-batches with load shares FmF^m, mean Fˉ\bar F and uniform QQ,

1M∑m12∥Fm−Q∥2=12∥Fˉ−Q∥2⏟global imbalance+1M∑m12∥Fm−Fˉ∥2⏟skew between micro-batches.\frac1M\sum_m \tfrac12\|F^m-Q\|^2 = \underbrace{\tfrac12\|\bar F-Q\|^2}_{\text{global imbalance}} + \underbrace{\frac1M\sum_m\tfrac12\|F^m-\bar F\|^2}_{\text{skew between micro-batches}}.

A micro-batch loss is the global loss plus a penalty on micro-batches differing from each other, with one coefficient on both. Qiu’s account of why that second term hurts is data: “one micro-batch is usually formed with packed and truncated sequences from one specific domain”, and tokens of a code micro-batch can’t favour the experts good at code. A control supports it: counting the same small number of tokens drawn at random from the whole batch scored 8.041, against global’s 8.038 and micro’s 8.167. Its authors measured specialization mainly by selection frequencies, “without conducting more rigorous validation”, and the size depends on the data pipeline: Kolibri deliberately mixes its micro-batches. Selection also sets early: on the 400B runs, switching from 2 sequences to 512 at step 10,000 of 100,000 reached 7.283, against 7.383 for never and 7.199 for always.

Domains in the data
Domains in one micro-batch
Router
Placement
Global MaxVio
0
Local MaxVio
0
ranks
busiest rank at 1× its share

Every micro-batch holds every domain, so tokens stay with their own experts and load is even at both scopes.

Global MaxVio
0
Local MaxVio
0
ranks
busiest rank at 1× its share

This micro-batch holds every domain, so it is already even and forcing it changes nothing.

Global MaxVio
0
Local MaxVio
1
ranks
busiest rank at 1× its share

Tokens stay with their own domain's experts, so only 50% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
1
ranks
busiest rank at 2× its share

Tokens stay with their own domain's experts, so only 50% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
0
ranks
busiest rank at 1× its share

Even load, at a price: 50% of this micro-batch's token slots now land on experts of a domain it doesn't hold. That is the skew specialization needs.

Global MaxVio
0
Local MaxVio
3
ranks
busiest rank at 1× its share

Tokens stay with their own domain's experts, so only 25% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
3
ranks
busiest rank at 4× its share

Tokens stay with their own domain's experts, so only 25% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
0
ranks
busiest rank at 1× its share

Even load, at a price: 75% of this micro-batch's token slots now land on experts of a domain it doesn't hold. That is the skew specialization needs.

Global MaxVio
0
Local MaxVio
7
ranks
busiest rank at 1× its share

Tokens stay with their own domain's experts, so only 12.5% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
7
ranks
busiest rank at 4× its share

Tokens stay with their own domain's experts, so only 12.5% of the experts work on this micro-batch.

Global MaxVio
0
Local MaxVio
0
ranks
busiest rank at 1× its share

Even load, at a price: 87.5% of this micro-batch's token slots now land on experts of a domain it doesn't hold. That is the skew specialization needs.

Figure 2.A toy router with fully specialized experts. The global batch is even, and each micro-batch holds only some domains, so its experts are loaded up to D/m times their share. How that reaches a GPU depends on placement. Real routers aren't this clean, and how single-domain a micro-batch is depends on the data pipeline.

Global scope costs speed. On a 43B model, a global loss (128 sequences with an accumulation buffer) ran at 1.64 s per iteration against 1.55 for 8 sequences, 5.8% slower: about 1% from communicating counts and “the remainder mainly due to local expert load imbalance under the dropless strategy”. Adding a micro-batch loss at 1% of the global weight brought it to 1.59 s, with perplexity 5.795 between the two runs’ 5.779 and 5.862. This is the only timing of global against micro-batch scope. It compares two imbalanced runs, so it doesn’t price imbalance itself.

Other scopes target the machine. DeepSeekMoE argues that for computation bottlenecks “it becomes unnecessary to enforce strict balance constraints at the expert level”, and DeepSeek-V2 weighted a device-level loss above the expert-level one (0.05 against 0.003). Mistral Large 4 counts over each EP group, “the balancing metric responsible for slowdowns – expert inbound tokens”, reports that “Pretraining comparisons favored this objective” without numbers, and warns of coupling to the partitioning. Poolside’s Laguna XS.2 uses Qiu’s loss “across the non-padding tokens of the global batch”, for a reason: on Laguna M.1, identical padding representations meant “all padding tokens in a batch are routed to the same expert at the same time”.

The selection bias

Wang et al. (2024) removed the loss and added a number per expert that only affects choice. After each batch,

bi←bi+γ sign(kBE−ci),b_i \leftarrow b_i + \gamma\, \text{sign}\Big(\frac{kB}{E} - c_i\Big),

so a busy expert’s bias falls by γ\gamma and an idle one’s rises by γ\gamma. “It is not added to the gi,tg_{i,t} that weights the output of the selected experts.” The update uses the previous batch: if a token’s route depended on counts over the current batch, it would depend on later tokens that generation can’t see. DeepSeek-V3 adopted this with γ=0.001\gamma = 0.001 and reports better results “on most of the evaluation benchmarks” at 15.7B and 228.7B. DeSieno’s “conscience” for competitive learning (1988, as described by Villmann and Claussen, 2006) is an ancestor no MoE paper we read cites: a per-unit bias inside the winner rule, set from the unit’s win rate.

Figure 3.Wang et al.'s 1B model, MaxVio averaged over layers on a validation set. Raising α narrows the imbalance and, past a point, raises perplexity. Values for the loss other than α = 0.001 are read off the paper's figure. Their loss is computed per sequence and the bias per batch, so this compares scope as well as channel.

Wang’s 1B model went from MaxVio 0.72 to 0.04 and perplexity 9.56 to 9.50, and a 3B model from 0.52 to 0.04 and 7.97 to 7.92. The loss they beat counts per sequence, so this moves scope and channel together.

Suppose we route BB tokens to maximize total score, with kk experts per token and kB/EkB/E tokens per expert. Put a Lagrange multiplier bib_i on each expert’s load constraint. Each token then takes its kk largest st,i+bis_{t,i} + b_i, and we minimize over bb the convex function

Φ(b)=∑tTopKSumk(st+b)−kBE∑ibi,∂Φ∂bi=ci(b)−kBE,\Phi(b) = \sum_t \text{TopKSum}_k(s_t + b) - \frac{kB}{E} \sum_i b_i, \qquad \frac{\partial \Phi}{\partial b_i} = c_i(b) - \frac{kB}{E},

where TopKSumk\text{TopKSum}_k adds a token’s kk largest values (Neitemeier’s App. B; a subgradient where tokens tie). The gradient is the load error, and adding a constant to every bib_i changes nothing. K3’s report names the update: “A SignSGD step on this objective recovers the fixed-step sign update”. The bias is minus a price in Bertsekas’s auction algorithm, which BASE layers (Lewis et al., 2021) used directly for routing, and Wang’s linear variant is Robbins and Monro’s (1951) stochastic approximation with constant gain (both mappings ours). Global scope means one price vector shared by all micro-batches, which is why “token-independent expert biases cannot ensure microbatch-level” balance (Neitemeier).

Because the gate is left alone, this adds no gradient term. It still changes what is trained: the task loss reaches only selected logits (C2), and the bias decides which those are and which tokens each expert sees. K3’s “without altering … the gradient-based optimization of the router” goes too far on that account.

What the step size costs

γ\gamma is a learning rate for Φ\Phi: too low adapts slowly, too high oscillates. Wang found this tame on 1B, with perplexity 9.51, 9.50 and 9.51 at γ=10−4,10−3,10−2\gamma = 10^{-4}, 10^{-3}, 10^{-2}, and slow convergence at one end and “undesirable fluctuations” at the other. V3 sets γ\gamma to 0 for its last 500B of 14.8T tokens, MiMo-V2-Flash drops it to 10−510^{-5} for its long-context stage, and Arcee’s Trinity Large adds momentum and a soft clamp (SMEBU). K3 argues the rule breaks near a thousand experts: “balancing the load of nearly 10310^3 experts exceeds the regime in which existing auxiliary-loss-free bias updates remain well behaved”. It prints no measurement, so the measured cost of the step size is small on 1B and untested at scale.

Sign step γ
Computed rule
first below 0.7
step 663
mean MaxVio, steps 500 to 749
1.422
back below 0.7 after the change
not by step 1,500
first below 0.7
step 221
mean MaxVio, steps 500 to 749
0.371
back below 0.7 after the change
step 1,217 (467 steps later)
first below 0.7
step 78
mean MaxVio, steps 500 to 749
0.386
back below 0.7 after the change
step 908 (158 steps later)
first below 0.7
step 23
mean MaxVio, steps 500 to 749
0.478
back below 0.7 after the change
step 798 (48 steps later)
first below 0.7
step 9
mean MaxVio, steps 500 to 749
0.834
back below 0.7 after the change
step 769 (19 steps later)
first below 0.7
step 1
mean MaxVio, steps 500 to 749
0.499
back below 0.7 after the change
step 751 (1 step later)
first below 0.7
step 1
mean MaxVio, steps 500 to 749
0.417
back below 0.7 after the change
step 759 (9 steps later)
Figure 4.A toy router with 32 experts and 256 tokens per step, so sampling noise is large, as it is in serving and in small micro-batches. Settled small steps sit near the noise of random routing and take hundreds of steps to arrive. A quantile update arrives at once at about 1.4 times that noise, and averaging its estimates, which K3's report suggests, brings it down at some cost in reaction time.

Quantile balancing

The bias can be computed instead of walked towards, which is quantile balancing (QB). For token tt, let the cutoff τt\tau_t be the (k+1)(k+1)-th largest biased score. For a token that selects expert ii, τt−st,i\tau_t - s_{t,i} is the bias at which it would just drop out. Taking the k/Ek/E quantile of these values across the batch, and centering, gives

b^i=quantilek/E{τt−st,i}t,bi←b^i−1E∑jb^j.\hat b_i = \text{quantile}_{k/E}\{\tau_t - s_{t,i}\}_t, \qquad b_i \leftarrow \hat b_i - \frac{1}{E}\sum_j \hat b_j.

Centering changes no decision. The rule has no step size and still acts on the next step, for causality. K3 derives it as a minimizer of a two-variable dual along one coordinate, with the cutoffs held still: “QB jumps directly to the exact coordinate minimizer of the same dual objective”. For an overloaded expert it is also the exact minimizer of Φ\Phi along its own coordinate, the other biases held. An underloaded expert has to beat the kk-th largest score, which is above τt\tau_t, so its own update raises its bias and undershoots; it reaches its share as the overloaded release tokens (ours). One update isn’t a full solve, as Neitemeier warns. On the toy below, repeating it on one batch takes MaxVio from 1.70 through 0.30 and 0.13 to 0.05 (computed).

Update
MaxVio
1.70
loads range
4 to 162
MaxVio
0.30
loads range
45 to 78
MaxVio
0.13
loads range
51 to 68
MaxVio
0.05
loads range
51 to 63
MaxVio
0.05
loads range
53 to 63
Figure 5.One token's scores, with its cutoff, and all experts' loads, as one toy batch is balanced repeatedly. Real training does one update per optimizer step on fresh tokens, so each update also meets new data.

The method is Jianlin Su’s (2026, a blog post we could not fetch; K3, Neitemeier and Marin all credit it). Marin ran it on a 32B model with 64 experts over 326B tokens, with “zero loss spikes”, and K3 uses it at 896. A quantile from one batch passes that batch’s sampling error straight through, about 2\sqrt{2} times the floor if batches are independent (ours; the figure above shows it), and K3 mentions a moving average that “can improve load balance still further”, without saying it uses one. The larger cost is that the rule needs a distribution spread over ranks.

Three ways to get the quantile

Gathering all margins on Neitemeier’s 7.5B test model would mean 4.5 GiB per layer per step (9.4M tokens × 256 experts × 2 bytes for BF16, a 16-bit float, computed). A global sum is an all-reduce, whose cost follows what is summed, and quantiles don’t add.

Marin averages per-rank quantiles (its code takes jax.lax.pmean of each device’s value). This needs a small all-reduce and no bins, but if ranks hold different data the mean of quantiles isn’t the quantile of the mix.

K3 pools a histogram. Each rank bins each expert’s margins into BhB_h buckets and one all-reduce sums the counts, so the result is the pooled batch up to one bin width, w=(bmax⁡−bmin⁡+2)/Bhw = (b_{\max} - b_{\min} + 2)/B_h. The “+2” comes from sigmoid scores lying in (0,1)(0,1), so the estimator depends on the bias space. With Bh=1000B_h = 1000, K3 reports “at most a few 10−310^{-3}, and we observe no measurable residual load imbalance”. At 1000 bins its layer sends 896,000 counts (computed).

Kolibri’s Exact Quantile Balancing (EQB, Neitemeier et al., 2026) selects by radix. A first all-reduce of 256 bins per expert finds the high byte of the BF16 quantile, and a second, among margins in the winning bin, finds the low byte. That is 2×256×E2 \times 256 \times E int32 per layer whatever BB is: 0.75 MiB for Kolibri’s E=384E = 384, 37.5 MiB per step over 50 layers (computed). Under gradient accumulation the second pass must rescan margins from every micro-batch, so they have to be kept or recomputed (our inference). “Exact” means the exact order statistic of rounded values, and BF16 spacing grows with the margin: 0.0078 near 1, 0.125 from 16.

How different the ranks are
Histogram bins
Figure 6.A toy expert whose margins differ by rank, as they would if ranks held different domains. Averaging quantiles gives it twice its target load at the default skew. Measured gaps in real training are far smaller, and no source compares a histogram with the exact quantile on one model.

Neitemeier’s test model has E=256E = 256, k=6k = 6 and two dense layers, and runs 100B tokens on one seed; BPB is bits per byte on held-out sets, lower is better, and MaxVio is the worst layer’s:

MethodGlobal MaxVioLocal MaxVioMean BPB
Rank-averaged QB0.925.960.6782
EQB0.745.380.6570

The baseline is rank averaging, so read this as pooled against averaged; nobody has compared a histogram with exact on one model. Marin’s team, on 64 GPUs, ran the histogram against averaging and saw “very slightly better balancing but no change in loss” with about 1% slowdown, and noted that 64 “may not be representative” (issue 7891). So one seed supports a quality gain, and one small run doesn’t show it.

EQB’s 0.74 is far from zero. Independent tokens would give Global MaxVio about 0.007 here (computed from 2ln⁡E⋅E/(kB)\sqrt{2\ln E \cdot E/(kB)}), but tokens in a sequence aren’t independent. If its 64 data-parallel ranks give about 64 interchangeable micro-batches, Local MaxVio over 64\sqrt{64} predicts about 0.7, close to the table (ours, rough).

Repairing the micro-batch

Global prices leave local imbalance by construction: in the table the busiest expert has more than 6 times its share in the worst layer.

One repair adds a small gradient on the router. Load-Error Injection (LEI, Neitemeier et al.) adds each expert’s local excess ei=ci/(kBl/E)−1e_i = c_i/(k B_l/E) - 1 over a micro-batch of BlB_l tokens directly to the gradient of its score, so descent lowers an overloaded expert’s score:

∂L∂st,i←∂L∂st,i+λ tanh⁡ξξ ei,ξ=∥e∥∞c.\frac{\partial L}{\partial s_{t,i}} \leftarrow \frac{\partial L}{\partial s_{t,i}} + \lambda\, \frac{\tanh \xi}{\xi}\, e_i, \qquad \xi = \frac{\|e\|_\infty}{c}.

The factor caps the added term at λc\lambda c, because “unbounded auxiliary score gradients caused attention logit instability for both the GShard loss and LEI”. Forward routing and weights don’t change. Any gradient added this way is the gradient of a loss with stop-gradient counts, so LEI is Wang’s raw-sigmoid form with its constant term removed and a clip added, which is why it doesn’t shrink scores (ours). Kolibri ships λ=10−5\lambda = 10^{-5}, c=1c = 1.

Added to EQB, same model and run lengthGlobal MaxVioLocal MaxVioMean BPB
Nothing0.745.380.6570
Normalized GShard loss0.744.300.6758
GShard loss0.714.710.6853
Normalized LEI0.603.520.6720

At 100B tokens, every local repair costs BPB. At 500B the picture changes metric and direction: average accuracy runs 48.25, 50.53, 48.56, 48.00 as λ\lambda rises from 0 to 8×10−58 \times 10^{-5}, the best accuracy going with the worst Global MaxVio. One seed can’t separate that from noise, and the paper adds that “Local MaxVio is a proxy for dispatch cost rather than measured EP throughput”.

The other repair leaves the router alone and moves experts. With copies of a hot expert on several GPUs, its tokens can be split among them while the router stays free to skew. No output changes, so this may use the current batch without breaking causality. DeepSeek-V3 serves this way, with “high-load experts” detected online “e.g., every 10 minutes”: one extra expert per GPU on top of 8 in prefill, and in decode 64 of 320 GPUs for redundant and shared experts. DeepSeek’s EPLB library packs such copies. K3’s MoonEP plans per micro-batch in training so that every rank receives “exactly S×KS \times K tokens”, reserving up to E/RE/R redundant experts per rank, which can double a rank’s routed weights (computed from its bound), and prints no throughput. UltraEP (2026), a separate system, measures it: 1.49× throughput over no balancing, averaged across training and prefill, up to 256 GPUs. It adds that placement “cannot replace routing losses because they do not serve the same modeling objectives”. Copies address the stall and leave an expert’s training signal as it was, and replication is what lets a model keep the router global (our reading).

Serving changes the numbers. A decode step has one token per active sequence, so at K3’s shape uniform independent routing gives MaxVio 1.8 at 256 sequences and 0.8 at 1,024 from sampling alone (simulated), and a frozen bias can’t help. Prefill is compute-bound, and UltraEP finds it “the primary source of expert load imbalance”. For “memory-bound decode” it finds compute imbalance “largely diluted by memory access latency”, though DeepSeek’s decode has one expert per GPU and replicates anyway; no source resolves this.

When balanced routing is useless

Kolibri has no dense first layers, and its pretraining stayed at Global MaxVio 0.31 averaged over layers. Its first two layers reached 2.39, the worst in the model, with a moderate dead-expert fraction. Its report then checks the top-kk intervention rate, the share of a token’s router-preferred experts that the bias replaces: 98.5% in those layers towards the end of training. At one checkpoint, randomizing or silencing their routed experts changed the loss on 65,536 held-out tokens by an amount “indistinguishable from zero”. “Balance metrics can look healthy while the bias overrides the router’s own preferences, which leaves the routing effectively random.” No other report prints this metric, so its absence elsewhere isn’t evidence of health.

The 98.5% sits next to 1−k/E=98.44%1 - k/E = 98.44\%, but that level doesn’t indicate randomness. If the router’s own choice is one fixed set FF and load is exactly even, the overlap between FF and the selected set averages 1Bk∑e∈Fce=k/E\frac{1}{Bk}\sum_{e \in F} c_e = k/E (ours). So balance plus fixed favourites gives that rate whatever routing is worth. Kolibri’s layer 48 has a high rate too, and “perturbing it raises the loss clearly”.

Router's fixed favourites, sd
Token-specific signal

Load is even and the bias replaces 4% of the router's choices. Useful selections are at about chance: the routing is random as far as usefulness goes.

Load is even and the bias replaces 4% of the router's choices. Useful selections are at 4.1 times chance: the routing is useful.

Load is even and the bias replaces 4% of the router's choices. Useful selections are at 12.6 times chance: the routing is useful.

Load is even and the bias replaces 55% of the router's choices. Useful selections are at about chance: the routing is random as far as usefulness goes.

Load is even and the bias replaces 53% of the router's choices. Useful selections are at 4.1 times chance: the routing is useful.

Load is even and the bias replaces 33% of the router's choices. Useful selections are at 12.6 times chance: the routing is useful.

Load is even and the bias replaces 86% of the router's choices. Useful selections are at about chance: the routing is random as far as usefulness goes.

Load is even and the bias replaces 86% of the router's choices. Useful selections are at 4.1 times chance: the routing is useful.

Load is even and the bias replaces 79% of the router's choices. Useful selections are at 12.7 times chance: the routing is useful.

Load is even and the bias replaces 92% of the router's choices. Useful selections are at about chance: the routing is random as far as usefulness goes.

Load is even and the bias replaces 92% of the router's choices. Useful selections are at 4.1 times chance: the routing is useful.

Load is even and the bias replaces 91% of the router's choices. Useful selections are at 12.6 times chance: the routing is useful.

Figure 7.A toy router with a fixed favourite set and a varying amount of token-specific signal, with the bias fitted on the tokens it is scored on, so load is nearly even here where Kolibri's layers were not. Intervention near the reference appears with useful routing and with random routing alike, as Kolibri's report found when perturbing its layers 0, 1 and 48. The toy shows that this can happen, not why Kolibri's early layers took this form.

Selection is the sign of st,e+be−τts_{t,e} + b_e - \tau_t. A balancing bias lines up each expert’s k/Ek/E quantile of margins, which removes most of its main effect, so selection runs on the token-by-expert interaction (our reading). If that residual is mostly noise, the expert set is too. A bias doesn’t cause low signal, and Kolibri has no control run with another balancer. It does remove pressure to fix it: the task loss reaches selected logits only, so nothing penalizes a preference the bias overrides. In a toy of ours such favourites persisted, and grew from a flat start under logit noise. If selection becomes independent of xx, the experts see the same data, stop differing, and the router’s task gradient vanishes (our reading).

Kolibri doesn’t renormalize, so an expert forced in against its score gets a small weight, which may be why silencing them costs nothing; a renormalized router would give them full weight (ours). A loss would tell the router only if it counts the router’s own choice, as V3’s Eq. 18 prints. One that counts dispatched load goes silent once the bias evens it, by the formula above, and Kolibri already had LEI on. No source tests the unbiased count here.

Early layers are where others give up on routing. DeepSeekMoE keeps its first layer dense since “the load balance status converges especially slower for the first layer”, Laguna keeps one, Neitemeier’s test model two (so its tables don’t test this failure), and DeepSeek-V4 uses a frozen token table (C2). Marin says quantile balancing meant it “didn’t need leading dense layers”. Kolibri concluded the same from a 327.6B-token ablation before its full run called that “into question”: the problem “only emerged after longer training durations”, and densifying the layers during fine-tuning gave “no clear downstream improvement”.

Per-modality balancing

Prices assume one token population. DeepSeek-V4.1’s report: “Image and text tokens exhibit distinct representation distributions … Balancing their aggregate load may therefore obscure modality-specific imbalance”. It keeps separate biases per modality, updated independently at the same 0.001, and each token uses its modality’s bias for selection only. The report gives no measurement. In vLLM’s code a row uses bias_vl in place of the text bias when its token id falls in a five-id image range. This is scope again, applied to one known split of the data, and the image bias updates on fewer tokens, so it is noisier (ours).

Freezing

Balancing is a training-stage decision. K3’s “final bias is frozen at inference”. V3 freezes for its last 500B tokens. Kolibri turns load balancing off for fine-tuning and its RL recipe lists no balancing loss. Nemotron 3.5 disables bias updates and the balance loss during quantization-aware distillation, as part of freezing everything but the quantized weights. MiMo-V2.6 froze the router itself after the run in the opening: restoring pre-RL router parameters at step 20 “recovers to near-initial levels while benchmark performance remains unchanged”, and the frozen run kept peak load near 5.5×.

A frozen bias then meets new data. Nemotron’s report finds MaxVio lower on training than validation tokens throughout pretraining (≈1.3 against ≈5 on Nano and ≈2 against ≈6 on Super, on its ratio scale), which suggests routing generalizes imperfectly; it doesn’t test a frozen bias. V3 lists “domain-shift-induced load imbalance during inference” among the costs of letting experts specialize, which is what replicated experts are for.

Comparison

MethodActs throughCounted overYou setLocal imbalance leftSent per layer per stepKnown costExamples
Capacity and dropForward pathBatch or groupCapacity factorCappedCountsDropped tokensSwitch, DeepSeek-V2
Auxiliary lossGradientSequence, micro-batchα\alphaReducedNoneQuality at small scope across three α\alpha; no boundSwitch, V3’s safety net
Global-batch lossGradientGlobal batchα\alphaLarge; stallsEE counts5.8% slower in the one timingQiu, Laguna XS.2
EP-group lossGradientEP groupα\alphaReducedCounts in groupCoupled to partitioningMistral Large 4
Fixed-step biasSelectionGlobal batch (worst in Qiu at micro-batch)γ\gamma, scheduleLarge; stallsEE countsSlow or wobbling; can overrideDeepSeek-V3, V4, MiMo-V2-Flash
QB, rank-averagedSelectionPer rank, then meanNothingLarge; stallsEE quantilesDepends on partition; noise; can overrideMarin
QB, histogramSelectionGlobal batchBinsLarge; stallsBhEB_h E countsBin width; noise; can overrideK3
EQBSelectionGlobal batchNothingLarge; stalls512E512 E int32, two all-reducesRounding; noise; rescan; can overrideKolibri
LEI (on a bias)GradientMicro-batchλ\lambda, clipReducedNoneBPB at 100B, unclear at 500BKolibri
Modality biasesSelectionGlobal batch per modalityAs the biasLarge; stallsTwiceAssumes a known splitDeepSeek-V4.1
Replicated expertsPlacementRankCopies, plannerNone at rank levelWeights or planMemory; prediction or planningV3 serving, EPLB, MoonEP

Where the models in the map sit

Of the 35 models this series tracks, 33 have an MoE layer. By what their code does at selection, 28 from 11 labs add a selection-only bias (27 to scores, Kolibri to logits). The code check says present for 27 of those and unclear for Nemotron 3.5 Super-VL, where our run holds only a wrapper file and the checkpoint has the tensors. Four have a Switch-style loss function and no selection-only bias, and Mistral Large 4 has neither. No shipped code holds an update rule, so code shows a slot, the checkpoint a final value, and only a report what filled it.

Two checkpoint readings add to that (ours). The spread, max minus min, is what selection sees, and a bias that balances a sigmoid router stays below 1 unless two experts are always picked together: if bu−bv≥1b_u - b_v \ge 1 then uu is chosen whenever vv is, so equal shares would mean they only ever appear as a pair. And any token that takes vv over uu needs sv−su>bu−bvs_v - s_u > b_u - b_v, so a spread near 1 says some of the router’s scores have saturated (Step-5-preview reaches 0.948). Selection and the loss both ignore the mean, so it only records the update rule: the sign rule isn’t zero-sum, since usually more experts are under target than over, so its mean climbs, while linear and centered rules hold it at zero. Trinity’s report centers a fixed step, and V3’s describes one where V3.2’s mean is zero, so a zero mean doesn’t prove a quantile.

“Spread” is the median layer [widest], in score units unless marked. “Mean” is positive when every layer’s is, up to the maximum shown. The last column names the report behind each entry.

ModelBias added toSpreadMeanLoss function in code, coefficient in configReport on training
Kimi K3Sigmoid scores0.341 [0.847]≈0NoneQB, histogram (1000 bins in App. D, “a few hundred” in §2.3.3); MoonEP
Kolibri-1Logits3.449 [23.547] logits≈0NoneEQB, LEI λ=10−5\lambda = 10^{-5}; off in fine-tuning
DeepSeek-V3.2Sigmoid scores0.044 [0.221]≈0NoneV3 report: step 0.001, sequence loss 0.0001
DeepSeek-V4-Flash, -0731Root-softplus scores0.465 [1.688]up to 27Present, default 0.001V4 report (Flash): step 0.001, sequence loss 0.0001
DeepSeek-V4-Pro, -0813Root-softplus scores0.442 [1.634]up to 48Present, default 0.001V4 report (Pro): same
DeepSeek-V4-Flash-vision-expRoot-softplus scores; image bias in weights, unused by its code0.398 [1.517]; image 0.562 [2.734]up to 38Present, default 0.001None found
DeepSeek-V4.1-Flash, -1017Root-softplus scores, plus image bias0.519 [2.221]; image 0.796 [2.939]up to 58None in our run’s filesV4.1 report (Flash): per modality, step 0.001, sequence loss 0.0001
MiMo-V2.5, V2.5-ProSigmoid scores0.393 [0.855], 0.104 [0.586]up to 13, 91NoneMiMo-V2-Flash report: step 0.001, sequence loss 10−510^{-5}
MiMo-V2.6-Flash, -ProSigmoid scores0.356 [0.723], 0.283 [0.769]up to 13, 0.5NoneV2.6 report: router frozen in RL
Nemotron 3 UltraSigmoid scores0.166 [0.659]up to 57NoneNot stated
Nemotron 3.5 Lightning, Super-VLSigmoid scores0.111 [0.199], 0.142 [0.539]up to 47, 46None read3.5 report, distilling Lightning and Super Base: 10−410^{-4}, 10−510^{-5}; QAD: bias updates off
GLM-5.2, 5.3Sigmoid scores0.319 [0.643]up to 34NoneNot stated
GLM-5.3-FlashSigmoid scores0.439 [0.584]up to 14Present, 0.001Not stated
MiniMax-M3, M3.1-FlashSigmoid scores0.380 [0.590], 0.182 [0.413]up to 20, 37Present, default 0.001Not stated
Step-3.7-FlashSigmoid scores0.547 [0.881]up to 10NoneNot stated
Step-5-previewSigmoid scores0.573 [0.948]≈0None in our run’s filesNot stated
Hy3, Hy4-previewSigmoid scores0.072 [0.230], 0.101 [0.237]≈0NoneNot stated
InklingSigmoid scores0.543 [0.937]up to 5.4None in our run’s filesBlog: “auxiliary-loss-free”
Laguna-S-2.1Sigmoid scoresAll zeros0Present, 0.0XS.2 report: Qiu’s loss, global batch
gpt-oss-120bLogits, also in weights (not selection-only)0.473 [0.711] logits≈0Present, 0.9Not stated
Qwen3.5, 3.6, 3.8-FlashNonePresent, 0.001Not stated (Qiu et al. is from this team)
Mistral Large 4NoneNoneEP-group loss, coefficient not given

Five models share one DeepSeek-V4 file, and seven models from four labs share a router body. Eight of the 14 labs with an MoE layer say how they balance, or part of it. MiniMax, OpenAI, Qwen (for its models), StepFun, Tencent and Z.ai don’t.

Easy to get wrong

MaxVio numbers don’t compare across reports. Wang and Kolibri average over layers, and Neitemeier takes the worst. Wang counts a validation set, Neitemeier and Kolibri a training step or micro-batch, and Nemotron a mix of training and validation tokens. Nemotron 3 Ultra also defines it without the minus one, so its 1.2 is 0.2 elsewhere and its ≈12 is 11. Three reports credit V3 for the metric, whose report never uses the word.

“Exact” hasn’t been shown to beat Kimi’s histogram, and “fixed step fails near a thousand experts” is K3’s claim without a measurement. Kolibri’s “without a tunable balancing hyperparameter” describes EQB’s update, while its recipe has λ\lambda and the clip.

A code check saying present means a slot exists, and Laguna’s holds zeros. Several flags are read by nothing: moe_router_enable_expert_bias, use_routing_bias and Step-3.7’s use_moe_router_bias. Nemotron, Hy4, Laguna and Kolibri have a bias and no config sign of it.

The stock Transformers loss function isn’t the loss a report describes: it pools all layers into one count per expert index and selects its own top-kk on an unbiased softmax, even for a sigmoid router with a bias.

Check yourself

  1. A layer has E=256E = 256 and k=8k = 8. On a batch of 64,000 tokens, the busiest expert gets 4,000. What is MaxVio, and what is the ceiling?
    AnswerEven share is 64,000×8/256=2,00064{,}000 \times 8 / 256 = 2{,}000, so MaxVio is 4000/2000−1=14000/2000 - 1 = 1. The ceiling is E/k−1=31E/k - 1 = 31. If every token picks the same 8 experts, each receives all 64,000, which is 32 times the share.
  2. Why can tuning α\alpha at micro-batch scope fail to reach what a global-batch loss reaches?
    AnswerThe micro-batch objective equals the global one plus a penalty on micro-batches differing from each other, with one coefficient on both. Lowering α\alpha eases the penalty and loses global balance at the same rate. Qiu et al. find 8.154 to 8.167 across three coefficients at 4 sequences, and 8.038 at 512.
  3. With sign updates at γ=0.001\gamma = 0.001, at least how many steps does it take to open a 0.2 gap between two experts? What does one quantile update do instead?
    AnswerEach expert moves by at most γ\gamma in opposite directions, so the gap grows by at most 0.0020.002 per step: 100 steps. A quantile update moves an overloaded expert to where it would have its share in one step, given the others’ biases. An underloaded expert’s update undershoots, and it finishes filling as tokens are released, so a few updates are needed in all, each on the last step’s tokens.
  4. A layer has MaxVio 0.1 and a top-kk intervention rate of 1−k/E1 - k/E. Is it useless?
    AnswerNot necessarily. Balance plus a fixed favourite set gives that rate whatever routing is worth. The test is what happens to the loss when the layer’s routed experts are randomized or silenced: Kolibri’s layers 0 and 1 didn’t change the loss, and its layer 48 did.

In the map

Methods

Taxonomy nodes

Further reading

↑