Open Models Almanac
Methodsoptimizationtraining parallelism

general family · filed under optimization

Tensor parallelism

Tensor parallelism partitions tensor computations across ranks; the cited degrees are configuration details.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training parallelism for the methods filed beside it.

sources
6
models
4
labs adopt it
3
strongest
used

Documented in

Evidence

7 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
optional 2used 5

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted3 sources

tensor parallelism (TP, which shards the weight matrices of each linear layer across ranks)

usedinference servingin Nemotron 3 UltraNVIDIA
3 more from this model

wide TP the practical choice for low-latency serving

usedinference servingin Nemotron 3 UltraNVIDIA

8-way tensor parallelism

usedsoftware implementationin Nemotron 3 UltraNVIDIA

--tensor-parallel-size 4

optionalinference servingin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑