single mention · model architecture
Pooled-global-batch quantile estimation
Estimates a bias quantile from all tokens of the optimizer step, for example by summing per-rank histograms, rather than averaging quantiles computed on each rank. The estimate then doesn't depend on how tokens are split across ranks.
Written for the explainer Expert load balancing.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing