Open Models Almanac
Methodsoptimizationtraining parallelism

methods · optimization

training parallelism

47 methods filed at this node or below it, from the sources of 25 models.

In this branch 47

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 47

Expert parallelism used · 12 sources · 15 quotes
Tensor parallelism used · 6 sources · 7 quotes
Fully Sharded Data Parallel used · 5 sources · 6 quotes
Context parallelism used · 2 sources · 2 quotes
Pipeline parallelism used · 2 sources · 2 quotes
Tensor and expert parallelism used · 2 sources · 2 quotes
41 single mentions
FSDP2 with WebDataset streaming core · 1 source · 1 quote
MoonEP core · 1 source · 4 quotes
Redundant-expert capacity reservation core · 1 source · 1 quote
Sequence parallelism for activations core · 1 source · 1 quote
Action-expert-specific sharding groups used · 1 source · 1 quote
Batch-size scaling with GPU count used · 1 source · 1 quote
Cache-based pipeline communication used · 1 source · 1 quote
Communication-computation overlap used · 1 source · 1 quote
Dedicated reduce-scatter process group used · 1 source · 1 quote
Fully balanced expert-parallel training used · 1 source · 1 quote
Hybrid ZeRO bucket assignment for Muon used · 1 source · 1 quote
Intra-device context parallelism used · 1 source · 1 quote
KDA Context Parallelism used · 1 source · 1 quote
Pathways data-replica reduction used · 1 source · 1 quote
Sconv-aware tensor-parallel sharding used · 1 source · 1 quote
Three-buffer reduce-scatter used · 1 source · 1 quote
ZeRO-2 used · 1 source · 1 quote
ZeRO-3 optimizer-state sharding used · 1 source · 1 quote
Data and expert parallelism optional · 1 source · 1 quote
Data parallelism optional · 1 source · 1 quote
Model parallelism optional · 1 source · 1 quote
Prefill context parallelism optional · 1 source · 1 quote
Data-Weighted Data Parallelism mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
Step-5-Preview optionalData and expert parallelism optionalData parallelism optionalExpert parallelism optionalTensor and expert parallelism optionalTensor parallelism optional—
GLM-5.3 usedExpert parallelism used—
MiMo-V2.6-Flash usedWindow-bounded KV exchange for sliding-window layers used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedContext parallelism usedExpert parallelism usedPipeline parallelism usedTensor parallelism usedData-Weighted Data Parallelism mentioned—
DeepSeek-V4-Flash usedAll-to-all gradient exchange with local FP32 summation usedHybrid ZeRO bucket assignment for Muon usedKnapsack-based balanced assignment of dense parameter matrices usedModified DualPipe 1F1B pipeline overlap for mHC used—
DeepSeek-V4-Pro usedAll-to-all gradient exchange with local FP32 summation usedHybrid ZeRO bucket assignment for Muon usedKnapsack-based balanced assignment of dense parameter matrices usedModified DualPipe 1F1B pipeline overlap for mHC used—
GLM-5.2 usedExpert parallelism used—
Gemma 4 31B usedPathways data-replica reduction usedZeRO-3 optimizer-state sharding used—
Inkling usedSconv-aware tensor-parallel sharding used—
Kimi K3 coreMoonEP coreRedundant-expert capacity reservation coreSequence parallelism for activations coreCache-based pipeline communication usedDynamic context parallelism for multimodal samples usedExpert parallelism usedFully balanced expert-parallel training usedGPU planning kernel for redundant expert migration usedIntra-device context parallelism usedKDA Context Parallelism usedPipeline ZeRO-2 gradient sharding with CPU offloading usedpipeline-bubble scheduling of ViT computation used—
Kolibri-1 usedBatch-size scaling with GPU count usedDedicated reduce-scatter process group usedKeep weights unsharded after backward across gradient-accumulation steps usedThree-buffer reduce-scatter used—
LFM3-1.7B usedZeRO-2 with strided activation checkpointing used—
LingBot-VLA 2.0 usedAction-expert-specific sharding groups usedFully Sharded Data Parallel used—
MiMo-V2.6-Pro usedWindow-bounded KV exchange for sliding-window layers used—
Mistral-Large-4 usedCommunication-computation overlap usedContext parallelism usedExpert parallelism usedLatency-optimized context-parallel collectives usedZeRO-2 usedPrefill context parallelism optional—
MolmoAct 2 usedCross-device average token-count loss normalization usedFully Sharded Data Parallel usedGradient-hook synchronization for replicated LoRA parameters usedReplicated, unsharded LoRA parameters under FSDP used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedCombined context, tensor, and expert parallel training usedCombined context, tensor, expert, and pipeline parallel training usedData and expert parallelism for calibration used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedCombined context, tensor, and expert parallel training usedCombined context, tensor, expert, and pipeline parallel training usedData and expert parallelism for calibration used—
OpenVLA / OpenVLA-OFT optionalFully Sharded Data Parallel optional—
Qwen3.5-397B-A17B usedExpert parallelism used—
Qwen3.6-35B-A3B usedTensor parallelism used—
Qwen3.8-Flash-Next usedNS-FLOP-balanced static parameter partitioning usedTensor and expert parallelism used—
Step-3.7-Flash usedExpert parallelism usedTensor parallelism used—
VLA Foundry (Foundry-VLA-1.7B) coreFSDP2 with WebDataset streaming coreFully Sharded Data Parallel optional—
π0.5 optionalFully Sharded Data Parallel optionalModel parallelism optional—
↑