specific method · filed under model architecture
Quantile Balancing
Sets each expert's bias from the router-score quantile corresponding to its target load.
Also called QB load balancing for mixture-of-experts, QB for MoE load balancing, Quantile Balancing (QB).
- sources
- 3
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
5 spans quoted from the sources, strongest treatment first.
we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load [111].
while adopting QB (§ 2.3.3) for MoE load balancing
Both updates are thus the same quantile along the token and expert axes, respectively, which gives the method its name.
Quantile Balancing for load management
Quantile Balancing derives expert allocation directly from router-score quantiles, eliminating heuristic updates and a sensitive balancing hyperparameter
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: expert load balancing.