Open Models Almanac
Methodsoptimizationtraining parallelism

general family · filed under optimization

Expert parallelism

Expert parallelism (EP) distributes full routed experts across ranks while Attention and Mamba remain data-parallel.1 It confines inter-rank traffic to the all-to-all dispatch and combine that bracket each routed-expert block, with no AllReduce on the critical path of any GEMM.2

Trade-off EP wins at large batch sizes for high-throughput serving, but its synchronous all-to-all requires careful load balancing, since an under-loaded rank stalls the entire group.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. expert parallelism (EP, which distributes full routed experts across ranks while Attention and Mamba remain data-parallel — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. inter-rank traffic is confined to the all-to-all dispatch and combine that bracket each routed-expert block, with no AllReduce on the critical path of any GEMM — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  3. because the routed-expert all-to-all is synchronous, an under-loaded rank stalls the entire group — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

Also called MoE Expert Parallelism (EP), expert parallel, expert-parallel traffic.

sources
12
models
8
labs adopt it
6
strongest
used

Documented in

Evidence

15 quotes from 12 sources, one card per model, each opening with its strongest.

How sources treat it
optional 3used 12

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted4 sources

expert parallelism (EP, which distributes full routed experts across ranks)

usedinference servingin Nemotron 3 UltraNVIDIA
5 more from this model

Megatron’s Expert Parallelism (EP) groups can span multiple racks

usedsoftware implementationin Nemotron 3 UltraNVIDIA

wide EP the practical choice for high-throughput serving

usedinference servingin Nemotron 3 UltraNVIDIA

128-way expert parallelism

usedsoftware implementationin Nemotron 3 UltraNVIDIA

enable-expert-parallel

usedinference servingin Nemotron 3 UltraNVIDIA

--enable-expert-parallel

optionalinference servingin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑