single mention · model architecture
Load-balancing regularization
Adds regularization to MoE routing to mitigate load imbalance, with the cited use reporting destabilizing expert switching.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See expert load balancing for the methods filed beside it.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing