single mention · model architecture
Expert-parallel-group load-balancing loss
An auxiliary balancing loss whose token counts are taken over one expert-parallel group, rather than a micro-batch or the global batch. Mistral Large 4 reports it favoured over other balancing choices without printing numbers, and notes it couples the loss to the partitioning.
Written for the explainer Expert load balancing.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing