Open Models Almanac
Methodsoptimizationtraining parallelism

specific method · filed under optimization

Tensor and expert parallelism

TEP keeps attention tensor-parallel and places the routed experts whole on the GPUs. Memory per GPU, the copied single-head cache and the traffic stay those of tensor parallelism; the expert matrix multiplications run on whole experts instead of slices, which keeps quantization blocks intact.

Written for the explainer Parallelism for serving large MoE models: what each layout copies, and what it sends.

Also called Combined tensor and expert parallelism, TEP8.

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 1
↑