Laguna-S-2.1
Poolsidean in-flight recovery system detects hangs, NCCL failures, and node losses, replaces affected workers, and resumes from the latest checkpoint without human involvement.
specific method · filed under optimization
Restarts interrupted training on other nodes and resumes from a checkpoint; the evidence does not prescribe a particular recovery algorithm.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training runtime for the methods filed beside it.
Also called automatic recovery, in-flight recovery system.
3 quotes from 3 sources, one card per model, each opening with its strongest.
an in-flight recovery system detects hangs, NCCL failures, and node losses, replaces affected workers, and resumes from the latest checkpoint without human involvement.
Those were automatically handled by the pipeline without manual intervention. The cluster automatically restarted the training job(s) on a different set of nodes, and training picked up from a checkpoint at most 250 steps back.
Our fault-tolerance setup combines frequent checkpointing with automatic recovery.