Open Models Almanac
Methodsoptimizationtraining runtime

specific method · filed under optimization

Asynchronous checkpointing

Copies or saves checkpoint state in the background while training continues, rather than blocking the training loop.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training runtime for the methods filed beside it.

Also called Asynchronous background checkpointing, Asynchronous checkpoint saving, asynchronously saves a checkpoint.

sources
3
models
2
labs adopt it
2
strongest
used

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

Enabling asynchronous checkpointing from the Nvidia Resiliency Extension (NVRx), where model parameters are copied to CPU and persisted in the background while training continues

usedsoftware implementationin Nemotron 3 UltraNVIDIA
1 more from this model

Enabling asynchronous checkpointing from the Nvidia Resiliency Extension (NVRx)

usedsoftware implementationin Nemotron 3 UltraNVIDIA

Kolibri-1

Aleph Alphaadopted1 source

The trainer asynchronously saves a checkpoint consisting of model weights, optimiser states and learning-rate schedules at regular intervals.

usedsoftware implementationin Kolibri RL training runAleph Alpha
↑