single mention · optimization
Cache-based pipeline communication
Cache-based pipeline communication transfers tensors between pipeline-parallel stages by sending only newly produced blocks incrementally instead of full activations.1 The transferred blocks are freed as soon as their micro-batch completes, minimizing resident memory.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
only newly generated blocks are incrementally transferred between stages
— k3 tech reportreleased as soon as the micro-batch finishes, reaching the theoretical lower bound on memory footprint
— k3 tech report
Filed under optimization › training parallelism