Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

specific method · filed under model architecture

Quantile Balancing

Quantile Balancing is a technique used for load balancing in Mixture-of-Experts models.1 It sets each expert bias from the router-score quantile that matches its target load.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU (SiTU-GLU) to suppress activation explosion, and Quantile Balancing (QB) for load balancing — k3 tech report
  2. we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load [111 — k3 tech report

Also called QB load balancing for mixture-of-experts, QB for MoE load balancing, Quantile Balancing (QB).

sources
3
model
1
lab adopt it
1
strongest
core

Documented in

Moonshot AI

Evidence

5 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 4core 1

Kimi K3

Moonshot AIadopted3 sources

we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load [111].

coremodel architecturein Kimi K3Moonshot AI
4 more from this model

Quantile Balancing derives expert allocation directly from router-score quantiles, eliminating heuristic updates and a sensitive balancing hyperparameter

usedoptimizationin Kimi K3Moonshot AI

Both updates are thus the same quantile along the token and expert axes, respectively, which gives the method its name.

usedoptimizationin Kimi K3Moonshot AI

while adopting QB (§ 2.3.3) for MoE load balancing

usedunclearin Kimi K3Moonshot AI

Quantile Balancing for load management

usedmodel architecturein Kimi K3Moonshot AI
↑