general family · filed under optimization
Expert Parallelism
A model-parallelism family that distributes experts across ranks; the evidence does not specify a more particular expert-placement method.
Also called MoE Expert Parallelism (EP), expert parallel, expert-parallel traffic.
- sources
- 10
- models
- 6
- labs adopt it
- 5
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Lepikhin et al., 2020) paper arxiv.orgestablishes expert-parallel sharding across devices
Evidence
13 spans quoted from the sources, strongest treatment first.
--enable-expert-parallel
keeping expert-parallel traffic inside one high-bandwidth domain
128-way expert parallelism
wide EP the practical choice for high-throughput serving
use --language-model-only to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.
enable-expert-parallel
We utilized a hybrid approach combining Attention Data Parallelism (DP) and MoE Expert Parallelism (EP)
Megatron’s Expert Parallelism (EP) groups can span multiple racks
expert parallelism (EP, which distributes full routed experts across ranks)
vllm serve ... --enable-expert-parallel
sglang serve ... --ep 4
--enable-expert-parallel
Filed alongside
Other methods under optimization :: training parallelism.