During reinforcement learning on MiMo-V2.6-Pro, the busiest expert in one layer went from 6 times the mean load to 16 times in 20 training steps, and the share of experts below a tenth of the mean rose from 0.5% to 22% (V2.6 report §5.4). The same report’s reinforcement-learning runs also died out of memory, from a different problem: “an expert-parallel (EP) rank received over 30× the mean token load despite a relatively balanced full batch”.
The first failure is a router abandoning experts. The second is a router that looks fine over a whole batch and still swamps one GPU for one step. Balancing methods differ in which tokens they count over and in how they act on the count, and the measured differences follow the first of those more.
Background
A mixture-of-experts (MoE) layer replaces a transformer block’s feed-forward unit with smaller ones, the experts , of which each token uses (C2 builds it). A router gives token one logit per expert, a score function (a sigmoid in most models here, a softmax in some) makes scores , and a per-expert bias joins them only to choose the set:
Here is a constant and an optional shared expert every token uses. The bias stays out of the weights , and an expert trains only on tokens that select it. Sizes such as “1B” count parameters.
Under expert parallelism (EP), experts sit on different GPU processes, called ranks, and tokens travel to them in an all-to-all exchange. Most systems here are dropless: an overloaded expert takes all its tokens, so overload becomes waiting or an out-of-memory error rather than lost tokens.
Three batch sizes matter. A sequence is one row of a batch, often several documents packed together. A micro-batch is what goes through one forward and backward pass together: 1 to 4 sequences per GPU in Qiu et al. (2025), about one per rank in Kimi K3’s design, going by its tokens per rank (our reading). The global batch is everything behind one optimizer step, all ranks and all accumulation passes. Waiting happens per micro-batch and per layer, while a global-batch bias changes once per optimizer step.
Balance has a standard measure, MaxVio. For tokens assigned to expert out of tokens,
Zero means even load, and 1 means the busiest expert has twice its share. The ceiling is : 55 for K3 (, ), computed. Wang et al. (2024) define it. Counted over a global batch it is Global MaxVio, over one micro-batch Local MaxVio, and a report may average over layers or take the worst, so each number below says which. It looks only at the top; dead experts need another count, such as MiMo’s share below a tenth of the mean.
Expert imbalance reaches the clock through three links. A rank’s load sums over the experts it hosts, so the excess shrinks with more experts per rank unless hot experts sit together: Aleph Alpha’s Kolibri pretrains with 48 per rank (384 over its EP 8), and DeepSeek-V3’s decode has one. Only the routed part of the step waits, and with all experts on one device nothing does. And each layer waits on its own busiest rank, so step time goes as a sum of per-layer maxima (our reading of the last two).
Why a router collapses
Shazeer et al. (2017), who made the gate sparse, saw it first: “This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network.” With no balancing their model reached a test perplexity (a held-out loss, lower is better) of 39.8 with the busiest expert at 17.80 times the mean. With any of their balancing losses it reached 35.6 to 35.7, and at most 1.47 times.
Later models show less of a quality gap: on Wang’s 1B model, no balancing leaves MaxVio near 5 and perplexity near 9.54, read off their figure. The other costs remain. A dead expert wastes memory (Marin, an open training project: “Many experts immediately die out”), a hot one stalls its rank, and under dropless EP buffers must fit the peak: K3’s report gives as the worst case for a copy-free buffer in DeepEP, for sequence length , top- and ranks, where even load needs .
Kolibri’s report states the two goals together: “Load balancing ensures sufficient learning signal for every expert and efficient expert-parallel execution. The former requires balance across the global optimiser-step batch; the latter requires balance within each local microbatch.”
The auxiliary loss
The first fix adds a loss. Switch Transformers (Fedus et al., 2021) write it as
where is the fraction of tokens routed to expert and the mean router probability for it. The count passes no gradient, so carries the signal. GShard (Lepikhin et al., 2020) gives the reason for the shape: “we want to minimize mean square of ”, with one factor replaced by the differentiable mean gate. Since when the sum to one, as at Switch’s top-1, this is a surrogate for the squared coefficient of variation Shazeer penalized.
Switch says the loss “is minimized under a uniform distribution”. As a joint function of and that’s loose, and the exact statement is about the gradient training uses. With constant and softmax probabilities over tokens (derivation ours),
Each token moves probability off experts more loaded than its own probability-weighted average. When is uniform the bracket is zero whatever is, so the loss asks for even counts, not flat probabilities, and goes silent once counts are even. Neitemeier et al. (2026) state this as the straight-through gradient of the squared distance between load shares and uniform: the backward pass treats the count as if it were . It needs probabilities that sum to one. On a raw sigmoid (Wang’s Eq. 2) the gradient is , which pushes every score down even at even load; DeepSeek-V3’s Eq. 19 normalizes to remove this.
Constants don’t carry between papers, since the forms differ by factors of and . Switch “found balanced load quickly without interfering with training loss” on a top-1 softmax router. Wang’s 1B model, top-6 sigmoid, shows a cost: read off their figure, of 0, , and give MaxVio near 5.1, 2.2, 0.72 and 0.1, and perplexity near 9.54, 9.53, 9.56 and 9.69. Whether the raw-sigmoid term explains that last step is untested.
Later models often keep a loss as a safety net. DeepSeek-V3 sets “just to avoid extreme imbalance within any single sequence”, and MiMo-V2-Flash’s pretraining uses . No report removes it to measure what it does. A loss also promises no bound, so early systems capped each expert and let overflow tokens skip the layer, “typically < 1%” of them in Switch. DeepSeek-V3 “does not drop any tokens during training”. The inference code we can read has no cap, which says nothing about training stacks.
What you count over
The term needs a set of tokens, and the choice turned out to matter more than the coefficient. Qiu et al. (2025) trained a 3.4B model on 120B tokens with sequences of 4,096 tokens, counting over different numbers of sequences:
| Rule, sequences counted over | Avg. perplexity | MMLU (a benchmark, higher is better) |
|---|---|---|
| Loss, micro-batch (4) | 8.167 | 41.63 |
| Loss, partial (32) | 8.062 | 42.08 |
| Loss, global batch (512) | 8.038 | 43.48 |
| Bias, micro-batch (4) | 8.521 | 41.30 |
| Bias, global batch (512) | 8.080 | 42.74 |
The bias rows anticipate the next section, and use a sigmoid gate where the loss rows use softmax. Three coefficients from 0.001 to 0.008 at micro-batch scope stayed between 8.154 and 8.167. In a 400B-token run, going from 2 sequences to 8 gained more than raising from 4 to 6 did at 50% more active expert FLOPs.
DeepSeek-V3’s report built a batch-wise auxiliary loss to separate scope from channel: validation losses on 1B were 2.258 for a sequence-wise loss, 2.253 for the bias and 2.253 for the batch-wise loss, and on 3B 2.085, 2.080 and 2.080. “The key distinction … lies in their balancing scope.” So at batch scope the channel hardly matters in these two tests (Qiu’s 8.038 and 8.080 differ in gate too). At small scope it does, and the bias is the worst row.
An identity shows why tuning can’t catch up (ours). For equal micro-batches with load shares , mean and uniform ,
A micro-batch loss is the global loss plus a penalty on micro-batches differing from each other, with one coefficient on both. Qiu’s account of why that second term hurts is data: “one micro-batch is usually formed with packed and truncated sequences from one specific domain”, and tokens of a code micro-batch can’t favour the experts good at code. A control supports it: counting the same small number of tokens drawn at random from the whole batch scored 8.041, against global’s 8.038 and micro’s 8.167. Its authors measured specialization mainly by selection frequencies, “without conducting more rigorous validation”, and the size depends on the data pipeline: Kolibri deliberately mixes its micro-batches. Selection also sets early: on the 400B runs, switching from 2 sequences to 512 at step 10,000 of 100,000 reached 7.283, against 7.383 for never and 7.199 for always.
- Global MaxVio
- 0
- Local MaxVio
- 3
- ranks
- busiest rank at 1× its share
Tokens stay with their own domain's experts, so only 25% of the experts work on this micro-batch.
Global scope costs speed. On a 43B model, a global loss (128 sequences with an accumulation buffer) ran at 1.64 s per iteration against 1.55 for 8 sequences, 5.8% slower: about 1% from communicating counts and “the remainder mainly due to local expert load imbalance under the dropless strategy”. Adding a micro-batch loss at 1% of the global weight brought it to 1.59 s, with perplexity 5.795 between the two runs’ 5.779 and 5.862. This is the only timing of global against micro-batch scope. It compares two imbalanced runs, so it doesn’t price imbalance itself.
Other scopes target the machine. DeepSeekMoE argues that for computation bottlenecks “it becomes unnecessary to enforce strict balance constraints at the expert level”, and DeepSeek-V2 weighted a device-level loss above the expert-level one (0.05 against 0.003). Mistral Large 4 counts over each EP group, “the balancing metric responsible for slowdowns – expert inbound tokens”, reports that “Pretraining comparisons favored this objective” without numbers, and warns of coupling to the partitioning. Poolside’s Laguna XS.2 uses Qiu’s loss “across the non-padding tokens of the global batch”, for a reason: on Laguna M.1, identical padding representations meant “all padding tokens in a batch are routed to the same expert at the same time”.
The selection bias
Wang et al. (2024) removed the loss and added a number per expert that only affects choice. After each batch,
so a busy expert’s bias falls by and an idle one’s rises by . “It is not added to the that weights the output of the selected experts.” The update uses the previous batch: if a token’s route depended on counts over the current batch, it would depend on later tokens that generation can’t see. DeepSeek-V3 adopted this with and reports better results “on most of the evaluation benchmarks” at 15.7B and 228.7B. DeSieno’s “conscience” for competitive learning (1988, as described by Villmann and Claussen, 2006) is an ancestor no MoE paper we read cites: a per-unit bias inside the winner rule, set from the unit’s win rate.
Wang’s 1B model went from MaxVio 0.72 to 0.04 and perplexity 9.56 to 9.50, and a 3B model from 0.52 to 0.04 and 7.97 to 7.92. The loss they beat counts per sequence, so this moves scope and channel together.
Suppose we route tokens to maximize total score, with experts per token and tokens per expert. Put a Lagrange multiplier on each expert’s load constraint. Each token then takes its largest , and we minimize over the convex function
where adds a token’s largest values (Neitemeier’s App. B; a subgradient where tokens tie). The gradient is the load error, and adding a constant to every changes nothing. K3’s report names the update: “A SignSGD step on this objective recovers the fixed-step sign update”. The bias is minus a price in Bertsekas’s auction algorithm, which BASE layers (Lewis et al., 2021) used directly for routing, and Wang’s linear variant is Robbins and Monro’s (1951) stochastic approximation with constant gain (both mappings ours). Global scope means one price vector shared by all micro-batches, which is why “token-independent expert biases cannot ensure microbatch-level” balance (Neitemeier).
Because the gate is left alone, this adds no gradient term. It still changes what is trained: the task loss reaches only selected logits (C2), and the bias decides which those are and which tokens each expert sees. K3’s “without altering … the gradient-based optimization of the router” goes too far on that account.
What the step size costs
is a learning rate for : too low adapts slowly, too high oscillates. Wang found this tame on 1B, with perplexity 9.51, 9.50 and 9.51 at , and slow convergence at one end and “undesirable fluctuations” at the other. V3 sets to 0 for its last 500B of 14.8T tokens, MiMo-V2-Flash drops it to for its long-context stage, and Arcee’s Trinity Large adds momentum and a soft clamp (SMEBU). K3 argues the rule breaks near a thousand experts: “balancing the load of nearly experts exceeds the regime in which existing auxiliary-loss-free bias updates remain well behaved”. It prints no measurement, so the measured cost of the step size is small on 1B and untested at scale.
- first below 0.7
- step 221
- mean MaxVio, steps 500 to 749
- 0.371
- back below 0.7 after the change
- step 1,217 (467 steps later)
Quantile balancing
The bias can be computed instead of walked towards, which is quantile balancing (QB). For token , let the cutoff be the -th largest biased score. For a token that selects expert , is the bias at which it would just drop out. Taking the quantile of these values across the batch, and centering, gives
Centering changes no decision. The rule has no step size and still acts on the next step, for causality. K3 derives it as a minimizer of a two-variable dual along one coordinate, with the cutoffs held still: “QB jumps directly to the exact coordinate minimizer of the same dual objective”. For an overloaded expert it is also the exact minimizer of along its own coordinate, the other biases held. An underloaded expert has to beat the -th largest score, which is above , so its own update raises its bias and undershoots; it reaches its share as the overloaded release tokens (ours). One update isn’t a full solve, as Neitemeier warns. On the toy below, repeating it on one batch takes MaxVio from 1.70 through 0.30 and 0.13 to 0.05 (computed).
- MaxVio
- 1.70
- loads range
- 4 to 162
The method is Jianlin Su’s (2026, a blog post we could not fetch; K3, Neitemeier and Marin all credit it). Marin ran it on a 32B model with 64 experts over 326B tokens, with “zero loss spikes”, and K3 uses it at 896. A quantile from one batch passes that batch’s sampling error straight through, about times the floor if batches are independent (ours; the figure above shows it), and K3 mentions a moving average that “can improve load balance still further”, without saying it uses one. The larger cost is that the rule needs a distribution spread over ranks.
Three ways to get the quantile
Gathering all margins on Neitemeier’s 7.5B test model would mean 4.5 GiB per layer per step (9.4M tokens × 256 experts × 2 bytes for BF16, a 16-bit float, computed). A global sum is an all-reduce, whose cost follows what is summed, and quantiles don’t add.
Marin averages per-rank quantiles (its code takes jax.lax.pmean of each device’s value). This needs a small all-reduce and no bins, but if ranks hold different data the mean of quantiles isn’t the quantile of the mix.
K3 pools a histogram. Each rank bins each expert’s margins into buckets and one all-reduce sums the counts, so the result is the pooled batch up to one bin width, . The “+2” comes from sigmoid scores lying in , so the estimator depends on the bias space. With , K3 reports “at most a few , and we observe no measurable residual load imbalance”. At 1000 bins its layer sends 896,000 counts (computed).
Kolibri’s Exact Quantile Balancing (EQB, Neitemeier et al., 2026) selects by radix. A first all-reduce of 256 bins per expert finds the high byte of the BF16 quantile, and a second, among margins in the winning bin, finds the low byte. That is int32 per layer whatever is: 0.75 MiB for Kolibri’s , 37.5 MiB per step over 50 layers (computed). Under gradient accumulation the second pass must rescan margins from every micro-batch, so they have to be kept or recomputed (our inference). “Exact” means the exact order statistic of rounded values, and BF16 spacing grows with the margin: 0.0078 near 1, 0.125 from 16.
Neitemeier’s test model has , and two dense layers, and runs 100B tokens on one seed; BPB is bits per byte on held-out sets, lower is better, and MaxVio is the worst layer’s:
| Method | Global MaxVio | Local MaxVio | Mean BPB |
|---|---|---|---|
| Rank-averaged QB | 0.92 | 5.96 | 0.6782 |
| EQB | 0.74 | 5.38 | 0.6570 |
The baseline is rank averaging, so read this as pooled against averaged; nobody has compared a histogram with exact on one model. Marin’s team, on 64 GPUs, ran the histogram against averaging and saw “very slightly better balancing but no change in loss” with about 1% slowdown, and noted that 64 “may not be representative” (issue 7891). So one seed supports a quality gain, and one small run doesn’t show it.
EQB’s 0.74 is far from zero. Independent tokens would give Global MaxVio about 0.007 here (computed from ), but tokens in a sequence aren’t independent. If its 64 data-parallel ranks give about 64 interchangeable micro-batches, Local MaxVio over predicts about 0.7, close to the table (ours, rough).
Repairing the micro-batch
Global prices leave local imbalance by construction: in the table the busiest expert has more than 6 times its share in the worst layer.
One repair adds a small gradient on the router. Load-Error Injection (LEI, Neitemeier et al.) adds each expert’s local excess over a micro-batch of tokens directly to the gradient of its score, so descent lowers an overloaded expert’s score:
The factor caps the added term at , because “unbounded auxiliary score gradients caused attention logit instability for both the GShard loss and LEI”. Forward routing and weights don’t change. Any gradient added this way is the gradient of a loss with stop-gradient counts, so LEI is Wang’s raw-sigmoid form with its constant term removed and a clip added, which is why it doesn’t shrink scores (ours). Kolibri ships , .
| Added to EQB, same model and run length | Global MaxVio | Local MaxVio | Mean BPB |
|---|---|---|---|
| Nothing | 0.74 | 5.38 | 0.6570 |
| Normalized GShard loss | 0.74 | 4.30 | 0.6758 |
| GShard loss | 0.71 | 4.71 | 0.6853 |
| Normalized LEI | 0.60 | 3.52 | 0.6720 |
At 100B tokens, every local repair costs BPB. At 500B the picture changes metric and direction: average accuracy runs 48.25, 50.53, 48.56, 48.00 as rises from 0 to , the best accuracy going with the worst Global MaxVio. One seed can’t separate that from noise, and the paper adds that “Local MaxVio is a proxy for dispatch cost rather than measured EP throughput”.
The other repair leaves the router alone and moves experts. With copies of a hot expert on several GPUs, its tokens can be split among them while the router stays free to skew. No output changes, so this may use the current batch without breaking causality. DeepSeek-V3 serves this way, with “high-load experts” detected online “e.g., every 10 minutes”: one extra expert per GPU on top of 8 in prefill, and in decode 64 of 320 GPUs for redundant and shared experts. DeepSeek’s EPLB library packs such copies. K3’s MoonEP plans per micro-batch in training so that every rank receives “exactly tokens”, reserving up to redundant experts per rank, which can double a rank’s routed weights (computed from its bound), and prints no throughput. UltraEP (2026), a separate system, measures it: 1.49× throughput over no balancing, averaged across training and prefill, up to 256 GPUs. It adds that placement “cannot replace routing losses because they do not serve the same modeling objectives”. Copies address the stall and leave an expert’s training signal as it was, and replication is what lets a model keep the router global (our reading).
Serving changes the numbers. A decode step has one token per active sequence, so at K3’s shape uniform independent routing gives MaxVio 1.8 at 256 sequences and 0.8 at 1,024 from sampling alone (simulated), and a frozen bias can’t help. Prefill is compute-bound, and UltraEP finds it “the primary source of expert load imbalance”. For “memory-bound decode” it finds compute imbalance “largely diluted by memory access latency”, though DeepSeek’s decode has one expert per GPU and replicates anyway; no source resolves this.
When balanced routing is useless
Kolibri has no dense first layers, and its pretraining stayed at Global MaxVio 0.31 averaged over layers. Its first two layers reached 2.39, the worst in the model, with a moderate dead-expert fraction. Its report then checks the top- intervention rate, the share of a token’s router-preferred experts that the bias replaces: 98.5% in those layers towards the end of training. At one checkpoint, randomizing or silencing their routed experts changed the loss on 65,536 held-out tokens by an amount “indistinguishable from zero”. “Balance metrics can look healthy while the bias overrides the router’s own preferences, which leaves the routing effectively random.” No other report prints this metric, so its absence elsewhere isn’t evidence of health.
The 98.5% sits next to , but that level doesn’t indicate randomness. If the router’s own choice is one fixed set and load is exactly even, the overlap between and the selected set averages (ours). So balance plus fixed favourites gives that rate whatever routing is worth. Kolibri’s layer 48 has a high rate too, and “perturbing it raises the loss clearly”.
Load is even and the bias replaces 92% of the router's choices. Useful selections are at about chance: the routing is random as far as usefulness goes.
Selection is the sign of . A balancing bias lines up each expert’s quantile of margins, which removes most of its main effect, so selection runs on the token-by-expert interaction (our reading). If that residual is mostly noise, the expert set is too. A bias doesn’t cause low signal, and Kolibri has no control run with another balancer. It does remove pressure to fix it: the task loss reaches selected logits only, so nothing penalizes a preference the bias overrides. In a toy of ours such favourites persisted, and grew from a flat start under logit noise. If selection becomes independent of , the experts see the same data, stop differing, and the router’s task gradient vanishes (our reading).
Kolibri doesn’t renormalize, so an expert forced in against its score gets a small weight, which may be why silencing them costs nothing; a renormalized router would give them full weight (ours). A loss would tell the router only if it counts the router’s own choice, as V3’s Eq. 18 prints. One that counts dispatched load goes silent once the bias evens it, by the formula above, and Kolibri already had LEI on. No source tests the unbiased count here.
Early layers are where others give up on routing. DeepSeekMoE keeps its first layer dense since “the load balance status converges especially slower for the first layer”, Laguna keeps one, Neitemeier’s test model two (so its tables don’t test this failure), and DeepSeek-V4 uses a frozen token table (C2). Marin says quantile balancing meant it “didn’t need leading dense layers”. Kolibri concluded the same from a 327.6B-token ablation before its full run called that “into question”: the problem “only emerged after longer training durations”, and densifying the layers during fine-tuning gave “no clear downstream improvement”.
Per-modality balancing
Prices assume one token population. DeepSeek-V4.1’s report: “Image and text tokens exhibit distinct representation distributions … Balancing their aggregate load may therefore obscure modality-specific imbalance”. It keeps separate biases per modality, updated independently at the same 0.001, and each token uses its modality’s bias for selection only. The report gives no measurement. In vLLM’s code a row uses bias_vl in place of the text bias when its token id falls in a five-id image range. This is scope again, applied to one known split of the data, and the image bias updates on fewer tokens, so it is noisier (ours).
Freezing
Balancing is a training-stage decision. K3’s “final bias is frozen at inference”. V3 freezes for its last 500B tokens. Kolibri turns load balancing off for fine-tuning and its RL recipe lists no balancing loss. Nemotron 3.5 disables bias updates and the balance loss during quantization-aware distillation, as part of freezing everything but the quantized weights. MiMo-V2.6 froze the router itself after the run in the opening: restoring pre-RL router parameters at step 20 “recovers to near-initial levels while benchmark performance remains unchanged”, and the frozen run kept peak load near 5.5×.
A frozen bias then meets new data. Nemotron’s report finds MaxVio lower on training than validation tokens throughout pretraining (≈1.3 against ≈5 on Nano and ≈2 against ≈6 on Super, on its ratio scale), which suggests routing generalizes imperfectly; it doesn’t test a frozen bias. V3 lists “domain-shift-induced load imbalance during inference” among the costs of letting experts specialize, which is what replicated experts are for.
Comparison
| Method | Acts through | Counted over | You set | Local imbalance left | Sent per layer per step | Known cost | Examples |
|---|---|---|---|---|---|---|---|
| Capacity and drop | Forward path | Batch or group | Capacity factor | Capped | Counts | Dropped tokens | Switch, DeepSeek-V2 |
| Auxiliary loss | Gradient | Sequence, micro-batch | Reduced | None | Quality at small scope across three ; no bound | Switch, V3’s safety net | |
| Global-batch loss | Gradient | Global batch | Large; stalls | counts | 5.8% slower in the one timing | Qiu, Laguna XS.2 | |
| EP-group loss | Gradient | EP group | Reduced | Counts in group | Coupled to partitioning | Mistral Large 4 | |
| Fixed-step bias | Selection | Global batch (worst in Qiu at micro-batch) | , schedule | Large; stalls | counts | Slow or wobbling; can override | DeepSeek-V3, V4, MiMo-V2-Flash |
| QB, rank-averaged | Selection | Per rank, then mean | Nothing | Large; stalls | quantiles | Depends on partition; noise; can override | Marin |
| QB, histogram | Selection | Global batch | Bins | Large; stalls | counts | Bin width; noise; can override | K3 |
| EQB | Selection | Global batch | Nothing | Large; stalls | int32, two all-reduces | Rounding; noise; rescan; can override | Kolibri |
| LEI (on a bias) | Gradient | Micro-batch | , clip | Reduced | None | BPB at 100B, unclear at 500B | Kolibri |
| Modality biases | Selection | Global batch per modality | As the bias | Large; stalls | Twice | Assumes a known split | DeepSeek-V4.1 |
| Replicated experts | Placement | Rank | Copies, planner | None at rank level | Weights or plan | Memory; prediction or planning | V3 serving, EPLB, MoonEP |
Where the models in the map sit
Of the 35 models this series tracks, 33 have an MoE layer. By what their code does at selection, 28 from 11 labs add a selection-only bias (27 to scores, Kolibri to logits). The code check says present for 27 of those and unclear for Nemotron 3.5 Super-VL, where our run holds only a wrapper file and the checkpoint has the tensors. Four have a Switch-style loss function and no selection-only bias, and Mistral Large 4 has neither. No shipped code holds an update rule, so code shows a slot, the checkpoint a final value, and only a report what filled it.
Two checkpoint readings add to that (ours). The spread, max minus min, is what selection sees, and a bias that balances a sigmoid router stays below 1 unless two experts are always picked together: if then is chosen whenever is, so equal shares would mean they only ever appear as a pair. And any token that takes over needs , so a spread near 1 says some of the router’s scores have saturated (Step-5-preview reaches 0.948). Selection and the loss both ignore the mean, so it only records the update rule: the sign rule isn’t zero-sum, since usually more experts are under target than over, so its mean climbs, while linear and centered rules hold it at zero. Trinity’s report centers a fixed step, and V3’s describes one where V3.2’s mean is zero, so a zero mean doesn’t prove a quantile.
“Spread” is the median layer [widest], in score units unless marked. “Mean” is positive when every layer’s is, up to the maximum shown. The last column names the report behind each entry.
| Model | Bias added to | Spread | Mean | Loss function in code, coefficient in config | Report on training |
|---|---|---|---|---|---|
| Kimi K3 | Sigmoid scores | 0.341 [0.847] | ≈0 | None | QB, histogram (1000 bins in App. D, “a few hundred” in §2.3.3); MoonEP |
| Kolibri-1 | Logits | 3.449 [23.547] logits | ≈0 | None | EQB, LEI ; off in fine-tuning |
| DeepSeek-V3.2 | Sigmoid scores | 0.044 [0.221] | ≈0 | None | V3 report: step 0.001, sequence loss 0.0001 |
| DeepSeek-V4-Flash, -0731 | Root-softplus scores | 0.465 [1.688] | up to 27 | Present, default 0.001 | V4 report (Flash): step 0.001, sequence loss 0.0001 |
| DeepSeek-V4-Pro, -0813 | Root-softplus scores | 0.442 [1.634] | up to 48 | Present, default 0.001 | V4 report (Pro): same |
| DeepSeek-V4-Flash-vision-exp | Root-softplus scores; image bias in weights, unused by its code | 0.398 [1.517]; image 0.562 [2.734] | up to 38 | Present, default 0.001 | None found |
| DeepSeek-V4.1-Flash, -1017 | Root-softplus scores, plus image bias | 0.519 [2.221]; image 0.796 [2.939] | up to 58 | None in our run’s files | V4.1 report (Flash): per modality, step 0.001, sequence loss 0.0001 |
| MiMo-V2.5, V2.5-Pro | Sigmoid scores | 0.393 [0.855], 0.104 [0.586] | up to 13, 91 | None | MiMo-V2-Flash report: step 0.001, sequence loss |
| MiMo-V2.6-Flash, -Pro | Sigmoid scores | 0.356 [0.723], 0.283 [0.769] | up to 13, 0.5 | None | V2.6 report: router frozen in RL |
| Nemotron 3 Ultra | Sigmoid scores | 0.166 [0.659] | up to 57 | None | Not stated |
| Nemotron 3.5 Lightning, Super-VL | Sigmoid scores | 0.111 [0.199], 0.142 [0.539] | up to 47, 46 | None read | 3.5 report, distilling Lightning and Super Base: , ; QAD: bias updates off |
| GLM-5.2, 5.3 | Sigmoid scores | 0.319 [0.643] | up to 34 | None | Not stated |
| GLM-5.3-Flash | Sigmoid scores | 0.439 [0.584] | up to 14 | Present, 0.001 | Not stated |
| MiniMax-M3, M3.1-Flash | Sigmoid scores | 0.380 [0.590], 0.182 [0.413] | up to 20, 37 | Present, default 0.001 | Not stated |
| Step-3.7-Flash | Sigmoid scores | 0.547 [0.881] | up to 10 | None | Not stated |
| Step-5-preview | Sigmoid scores | 0.573 [0.948] | ≈0 | None in our run’s files | Not stated |
| Hy3, Hy4-preview | Sigmoid scores | 0.072 [0.230], 0.101 [0.237] | ≈0 | None | Not stated |
| Inkling | Sigmoid scores | 0.543 [0.937] | up to 5.4 | None in our run’s files | Blog: “auxiliary-loss-free” |
| Laguna-S-2.1 | Sigmoid scores | All zeros | 0 | Present, 0.0 | XS.2 report: Qiu’s loss, global batch |
| gpt-oss-120b | Logits, also in weights (not selection-only) | 0.473 [0.711] logits | ≈0 | Present, 0.9 | Not stated |
| Qwen3.5, 3.6, 3.8-Flash | None | Present, 0.001 | Not stated (Qiu et al. is from this team) | ||
| Mistral Large 4 | None | None | EP-group loss, coefficient not given |
Five models share one DeepSeek-V4 file, and seven models from four labs share a router body. Eight of the 14 labs with an MoE layer say how they balance, or part of it. MiniMax, OpenAI, Qwen (for its models), StepFun, Tencent and Z.ai don’t.
Easy to get wrong
MaxVio numbers don’t compare across reports. Wang and Kolibri average over layers, and Neitemeier takes the worst. Wang counts a validation set, Neitemeier and Kolibri a training step or micro-batch, and Nemotron a mix of training and validation tokens. Nemotron 3 Ultra also defines it without the minus one, so its 1.2 is 0.2 elsewhere and its ≈12 is 11. Three reports credit V3 for the metric, whose report never uses the word.
“Exact” hasn’t been shown to beat Kimi’s histogram, and “fixed step fails near a thousand experts” is K3’s claim without a measurement. Kolibri’s “without a tunable balancing hyperparameter” describes EQB’s update, while its recipe has and the clip.
A code check saying present means a slot exists, and Laguna’s holds zeros. Several flags are read by nothing: moe_router_enable_expert_bias, use_routing_bias and Step-3.7’s use_moe_router_bias. Nemotron, Hy4, Laguna and Kolibri have a bias and no config sign of it.
The stock Transformers loss function isn’t the loss a report describes: it pools all layers into one count per expert index and selects its own top- on an unbiased softmax, even for a sigmoid router with a bias.
Check yourself
- A layer has and . On a batch of 64,000 tokens, the busiest expert gets 4,000. What is MaxVio, and what is the ceiling?
Answer
Even share is , so MaxVio is . The ceiling is . If every token picks the same 8 experts, each receives all 64,000, which is 32 times the share. - Why can tuning at micro-batch scope fail to reach what a global-batch loss reaches?
Answer
The micro-batch objective equals the global one plus a penalty on micro-batches differing from each other, with one coefficient on both. Lowering eases the penalty and loses global balance at the same rate. Qiu et al. find 8.154 to 8.167 across three coefficients at 4 sequences, and 8.038 at 512. - With sign updates at , at least how many steps does it take to open a 0.2 gap between two experts? What does one quantile update do instead?
Answer
Each expert moves by at most in opposite directions, so the gap grows by at most per step: 100 steps. A quantile update moves an overloaded expert to where it would have its share in one step, given the others’ biases. An underloaded expert’s update undershoots, and it finishes filling as tokens are released, so a few updates are needed in all, each on the last step’s tokens. - A layer has MaxVio 0.1 and a top- intervention rate of . Is it useless?
Answer
Not necessarily. Balance plus a fixed favourite set gives that rate whatever routing is worth. The test is what happens to the loss when the layer’s routed experts are randomized or silenced: Kolibri’s layers 0 and 1 didn’t change the loss, and its layer 48 did.
In the map
Methods
- Auxiliary-loss load balancing 1 source
- Auxiliary-loss-free load balancing 3 sources
- MoE sequence auxiliary loss 1 source
- Expert-parallel-group load-balancing loss 1 source
- Expert bias update factor 1 source
- Quantile Balancing 3 sources
- Exact Quantile Balancing 2 sources
- Histogram-based global quantile estimation 1 source
- Pooled-global-batch quantile estimation 1 source
- Exponential moving average of estimated quantiles 1 source
- Load-error gradient injection 2 sources
- Modality-specific load balancing 1 source
- MaxVio 1 source
- Fully balanced expert-parallel training 1 source
- Freezing the MoE router during reinforcement learning 1 source
- Disabled router-bias updates 1 source
- Expert Parallelism Load Balancing 1 source
- Mixture-of-experts load balancing 1 source
Taxonomy nodes
- expert load balancing 25 methods
- auxiliary loss 26 methods
Further reading
- Shazeer et al., Outrageously Large Neural Networks (2017), arXiv:1701.06538. The self-reinforcing imbalance and the first balancing losses, with a smooth estimate of load.
- Lepikhin et al., GShard (2020), arXiv:2006.16668, and Fedus et al., Switch Transformers (2021), arXiv:2101.03961. The auxiliary loss and capacity.
- Wang et al., Auxiliary-Loss-Free Load Balancing (2024), arXiv:2408.15664. The fixed-step bias, MaxVio, and step and rule variants on 1B.
- DeepSeek-AI, DeepSeek-V3 (2024), arXiv:2412.19437, §2.1.2, §3.4 and §4.5. The bias at 15.7B and 228.7B, replication, and the scope test with a batch-wise loss.
- Qiu et al., Demons in the Detail (2025), arXiv:2501.11873. Scope against quality and speed, with the shuffle control.
- Kimi K3 report §2.3.3, §5.2.1 and Appendices C to E, https://raw.githubusercontent.com/MoonshotAI/Kimi-K3/main/k3_tech_report.pdf. QB’s dual derivation, the histogram estimator and MoonEP.
- Neitemeier et al., Exact Quantile Balancing and Load-Error Injection (2026), arXiv:2609.28053. Radix selection and LEI’s derivation, with the full tables and their limits.
- Kolibri report §2.1.5 and Appendix B.6, https://aleph-alpha.com/downloads/tech-report.pdf. The diagnostics, including intervention rate, dead-expert fraction and perturbation.
- Dial, Mixture of Experts Quantile Balancing (2026), https://www.openathena.ai/blog/quantile-balancing/, and Marin issue 7891. A public QB implementation, and the pooled-against-averaged ablation.
- UltraEP (2026), arXiv:2606.04101, and deepseek-ai/EPLB. Balancing by placement, and what it buys in throughput.
- DeepSeek-V4.1-Flash report §2.1.1, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/resolve/main/DeepSeek_V41_Tech_Report.pdf. Modality-specific bias.
- Bertsekas, The auction algorithm (1988), Annals of Operations Research 14; Robbins and Monro (1951), Annals of Mathematical Statistics 22; Caron et al., SwAV (2020), arXiv:2006.09882. The classical versions of prices, stepping and equipartition.
