Open Models Almanac
Methodsoptimizationtraining parallelism

single mention · optimization

Three-buffer reduce-scatter

Three reduce-scatter input buffers are used to hide irregular communication latency from the critical path.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training parallelism for the methods filed beside it.

Also called set_reduce_scatter_max_input_buffers().

Filed under optimization › training parallelism