Open Models Almanac
Methodsoptimizationtraining runtime

specific method · filed under optimization

Automatic training recovery after failures

Restarts interrupted training on other nodes and resumes from a checkpoint; the evidence does not prescribe a particular recovery algorithm.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training runtime for the methods filed beside it.

Also called automatic recovery, in-flight recovery system.

sources
3
models
2
labs adopt it
2
strongest
core

Documented in

Poolside
Aleph Alpha

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 1

Laguna-S-2.1

Poolsideadopted1 source

an in-flight recovery system detects hangs, NCCL failures, and node losses, replaces affected workers, and resumes from the latest checkpoint without human involvement.

coresoftware implementationin Model FactoryPoolside

Kolibri-1

Aleph Alphaadopted2 sources

Those were automatically handled by the pipeline without manual intervention. The cluster automatically restarted the training job(s) on a different set of nodes, and training picked up from a checkpoint at most 250 steps back.

usedsoftware implementationin KolibriAleph Alpha
1 more from this model

Our fault-tolerance setup combines frequent checkpointing with automatic recovery.

usedsoftware implementationin KolibriAleph Alpha
↑