single mention · model architecture
Expert Parallelism Load Balancing
Replicating hot experts across expert-parallel ranks, so that their tokens can be split among copies. It leaves routing unchanged, costs memory for the copies, and addresses stalls rather than each expert's training signal.
Written for the explainer Expert load balancing.
Also called EPLB.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing
Explainer · C3 · 30 minExpert load balancingBalance over the whole batch lets experts specialize and leaves GPUs waiting; balance inside each micro-batch is faster and costs quality; a selection bias drops the loss and can quietly override the routerExplainer · I3 · 36 minParallelism for serving large MoE models: what each layout copies, and what it sendsEvery layout copies something onto every GPU: tensor parallelism the single-head KV cache, data-parallel attention the non-expert weights. The batch and the context decide which copy costs more, and memory often decides before speed does