general family · filed under optimization
Tensor Parallelism
A model-parallelism family that partitions tensor computation across ranks; the evidence does not specify a more particular sharding method.
- sources
- 5
- models
- 3
- labs adopt it
- 3
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
optional 1used 5
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
Evidence
6 spans quoted from the sources, strongest treatment first.
8-way tensor parallelism
usedsoftware implementationin Nemotron 3 UltraNVIDIA
wide TP the practical choice for low-latency serving
usedinference servingin Nemotron 3 UltraNVIDIA
--tensor-parallel-size 2
usedunclearin Qwen3.6-35B-A3BQwen
tensor parallelism (TP, which shards the weight matrices of each linear layer across ranks)
usedinference servingin Nemotron 3 UltraNVIDIA
--tensor-parallel-size 4
optionalinference servingin Nemotron 3 UltraNVIDIA
Filed alongside
Other methods under optimization :: training parallelism.
Expert ParallelismMoonEPCommunication-Computation OverlapPipeline ParallelismAll-to-all Gradient Exchange with Local FP32 SummationCache-Based Pipeline CommunicationContext ParallelismData Replica Reduction over the Data Center NetworkData-Weighted Data ParallelismDynamic Context Parallelism for Large Multimodal SamplesFully Balanced Expert-Parallel TrainingGPU Planning Kernel for Redundant Expert MigrationHybrid ZeRO Bucket Assignment for MuonKDA Context ParallelismKnapsack-Based Balanced Assignment of Dense Parameter MatricesLoad-Balanced Image ShardingModified DualPipe 1F1B Pipeline Overlap for mHCNS-FLOP-Balanced Static Parameter PartitioningPipeline Payload ExtensionsPipeline ZeRO-2 Gradient Sharding with CPU Offloadingpipeline-bubble scheduling of ViT computationRedundant-Expert Capacity ReservationSConv-Aware Tensor-Parallel ShardingSequence Parallelism for Activations