Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Auxiliary-loss load balancing

Adds a term to the training loss that pushes router probability away from experts over their share, computed from token counts that pass no gradient. Which tokens are counted (sequence, micro-batch, global batch) decides more than the coefficient in the tests that vary both.

Written for the explainer Expert load balancing.

Also called auxiliary loss.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing