specific method · filed under optimization
Communication-Computation Overlap
An overlap method that hides all-gather communication behind forward or backward computation rather than stalling the pipeline.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1core 1
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
each all-gather can be overlapped with the forward or backward pass instead of stalling the pipeline
coreunclearin DeepSeek-V4.1-FlashDeepSeek
each all-gather can be overlapped with the forward or backward pass instead of stalling the pipeline
usedunclearin DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under optimization :: training parallelism.
Expert ParallelismTensor ParallelismMoonEPPipeline ParallelismAll-to-all Gradient Exchange with Local FP32 SummationCache-Based Pipeline CommunicationContext ParallelismData Replica Reduction over the Data Center NetworkData-Weighted Data ParallelismDynamic Context Parallelism for Large Multimodal SamplesFully Balanced Expert-Parallel TrainingGPU Planning Kernel for Redundant Expert MigrationHybrid ZeRO Bucket Assignment for MuonKDA Context ParallelismKnapsack-Based Balanced Assignment of Dense Parameter MatricesLoad-Balanced Image ShardingModified DualPipe 1F1B Pipeline Overlap for mHCNS-FLOP-Balanced Static Parameter PartitioningPipeline Payload ExtensionsPipeline ZeRO-2 Gradient Sharding with CPU Offloadingpipeline-bubble scheduling of ViT computationRedundant-Expert Capacity ReservationSConv-Aware Tensor-Parallel ShardingSequence Parallelism for Activations