specific method · filed under optimization
GPU-to-GPU weight synchronization over GPUDirect RDMA
Transferring weights with NCCL point-to-point operations over GPUDirect RDMA without intermediate host-memory or object-storage offload.
Also called GPU-to-GPU weight transfer.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
The weight sync uses NCCL point-to-point primitives over GPUDirect RDMA with no intermediate offload to host memory or remote object storage.
usedinference servingin LAGUNA M.1/XS.2Poolside
Filed alongside
Other methods under optimization :: training runtime.
Offline checkpoint mergingAsynchronous checkpointingAsymmetric local-read and single-writer cache pathsAutotune configuration generationBatch-level embedding prefetchCaching the distributed checkpoint save planComposable activation storage policiesCPU-resident optimizer statesCross-rank remote activation offloadingDouble-buffered chunked streaming of reference-model weightsElement-wise activation recomputationHeartbeat-driven fault toleranceIn-flight recovery systemIn-flight weight updatesJob-level eviction and reclaimMinimal recomputation-graph extraction by backward traversalOverlapping NCCL transfers with device-to-host copiesPersistent checkpoint worker processesPersistent shared-storage cache for compiled artifactsPolicy-model gradient-buffer reuse for reference-model weightsPost-iteration NVMe offloading of training statesPre-admission hardware stress testingSame-node sticky pod respawnSeeding node-local storage from a warm shared cache