single source · model architecture
Mixture-of-experts load balancing
The family of methods that keep token load even across the experts of a mixture-of-experts layer, so that each expert gets training signal and no rank stalls the others. They differ in which tokens they count over and in whether they act through a loss, a selection bias or placement.
Written for the explainer Expert load balancing.
Also called MoE load balancing methodology.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing