single mention · model architecture
Auxiliary-loss load balancing
Adds a term to the training loss that pushes router probability away from experts over their share, computed from token counts that pass no gradient. Which tokens are counted (sequence, micro-batch, global batch) decides more than the coefficient in the tests that vary both.
Written for the explainer Expert load balancing.
Also called auxiliary loss.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing