Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Pooled-global-batch quantile estimation

Estimates a bias quantile from all tokens of the optimizer step, for example by summing per-rank histograms, rather than averaging quantiles computed on each rank. The estimate then doesn't depend on how tokens are split across ranks.

Written for the explainer Expert load balancing.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing