Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Expert Parallelism Load Balancing

Replicating hot experts across expert-parallel ranks, so that their tokens can be split among copies. It leaves routing unchanged, costs memory for the copies, and addresses stalls rather than each expert's training signal.

Written for the explainer Expert load balancing.

Also called EPLB.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing