Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Load-balancing regularization

Adds regularization to MoE routing to mitigate load imbalance, with the cited use reporting destabilizing expert switching.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See expert load balancing for the methods filed beside it.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing