Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

specific method · filed under model architecture

Exact Quantile Balancing

Exact Quantile Balancing is an expert load balancing method that adjusts token and expert biases through alternating quantile updates to balance expert loads.1 It routes with top-K of the biased logits and maintains global balance without a tunable balancing hyperparameter.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. computes token biases α∈ RT and expert biases β through alternating quantile updates — tech report
  2. we use the top-K of the biased logits for routing — tech report
  3. EQB maintains stable global load balance without a tunable balancing hyperparameter — tech report

Also called Exact Quantile Balancing (EQB).

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Aleph Alpha

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 1

Kolibri-1

Aleph Alphaadopted2 sources

Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)

coremodel architecturein KolibriAleph Alpha
2 more from this model

Kolibri combines Exact Quantile Balancing (EQB) for global balance with Load-Error Injection (LEI) for local balance

usedunclearin KolibriAleph Alpha

for routing between experts during training, we introduce exact quantile balancing.

usedoptimizationin KolibriAleph Alpha
↑