single mention · optimization
Data and expert parallelism
DEP runs attention data-parallel, each GPU serving its own sequences and keeping only their KV cache, and spreads the routed experts over all GPUs, sending each token to its experts and back with two all-to-alls per MoE layer. It stores a single-head cache once instead of once per GPU, at the price of copying every non-expert weight onto every GPU.
Written for the explainer Parallelism for serving large MoE models: what each layout copies, and what it sends.
Also called Combined data and expert parallelism.
Filed under optimization › training parallelism