general family · filed under optimization
Expert parallelism
Expert parallelism (EP) distributes full routed experts across ranks while Attention and Mamba remain data-parallel.1 It confines inter-rank traffic to the all-to-all dispatch and combine that bracket each routed-expert block, with no AllReduce on the critical path of any GEMM.2
Trade-off EP wins at large batch sizes for high-throughput serving, but its synchronous all-to-all requires careful load balancing, since an under-loaded rank stalls the entire group.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
expert parallelism (EP, which distributes full routed experts across ranks while Attention and Mamba remain data-parallel
— Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoninginter-rank traffic is confined to the all-to-all dispatch and combine that bracket each routed-expert block, with no AllReduce on the critical path of any GEMM
— Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningbecause the routed-expert all-to-all is synchronous, an under-loaded rank stalls the entire group
— Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Also called MoE Expert Parallelism (EP), expert parallel, expert-parallel traffic.