taxonomy node · level 2

training parallelism

29 methods filed at this node or below it, from the sources of 15 models.

optimization :: training parallelism

Matching aids for the classifier: tensor parallelism; pipeline parallelism; context parallelism; data-weighted data parallelism; training-time expert parallelism.

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Model	techniques
DeepSeek-V4.1-Flash core	Communication-Computation Overlap coreLoad-Balanced Image Sharding usedPipeline Payload Extensions used—
NVIDIA-Nemotron-3-Ultra-550B-A55B used	Context Parallelism usedExpert Parallelism usedPipeline Parallelism usedTensor Parallelism usedData-Weighted Data Parallelism mentioned—
MiMo-V2.6-Flash used	Window-Bounded KV Exchange under Context Parallelism used—
DeepSeek-V4-Flash used	All-to-all Gradient Exchange with Local FP32 Summation usedHybrid ZeRO Bucket Assignment for Muon usedKnapsack-Based Balanced Assignment of Dense Parameter Matrices usedModified DualPipe 1F1B Pipeline Overlap for mHC used—
GLM-5.3 used	Expert Parallelism used—
GLM-5.2 used	Expert Parallelism used—
DeepSeek-V4-Pro used	All-to-all Gradient Exchange with Local FP32 Summation usedHybrid ZeRO Bucket Assignment for Muon usedKnapsack-Based Balanced Assignment of Dense Parameter Matrices usedModified DualPipe 1F1B Pipeline Overlap for mHC used—
Gemma 4 31B used	Data Replica Reduction over the Data Center Network usedZeRO-3 Optimizer State Sharding used—
Inkling used	SConv-Aware Tensor-Parallel Sharding used—
Kimi K3 core	MoonEP coreRedundant-Expert Capacity Reservation coreSequence Parallelism for Activations coreCache-Based Pipeline Communication usedDynamic Context Parallelism for Large Multimodal Samples usedExpert Parallelism usedFully Balanced Expert-Parallel Training usedGPU Planning Kernel for Redundant Expert Migration usedKDA Context Parallelism usedPipeline ZeRO-2 Gradient Sharding with CPU Offloading usedpipeline-bubble scheduling of ViT computation usedSM-Level Context Parallelism used—
MiMo-V2.6-Pro used	Window-Bounded KV Exchange under Context Parallelism used—
Qwen3.5-397B-A17B used	Expert Parallelism used—
Qwen3.6-35B-A3B used	Tensor Parallelism used—
Qwen3.8-Flash-Next used	NS-FLOP-Balanced Static Parameter Partitioning usedTensor and Expert Parallelism with Degree Eight used—
Step-3.7-Flash used	Expert Parallelism usedTensor Parallelism used—

training parallelism

In this branch 29

filed here 29

By model