single mention · optimization
Sconv-aware tensor-parallel sharding
Each rank stores only its channel shard of the sconv cache under tensor parallelism.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training parallelism for the methods filed beside it.
Also called sconv-aware TP sharding.
Filed under optimization › training parallelism