Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single source · model architecture

Mixture-of-experts load balancing

The family of methods that keep token load even across the experts of a mixture-of-experts layer, so that each expert gets training signal and no rank stalls the others. They differ in which tokens they count over and in whether they act through a loss, a selection bias or placement.

Written for the explainer Expert load balancing.

Also called MoE load balancing methodology.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing