Open Models Almanac
Methodsoptimizationtraining runtime

methods · optimization

training runtime

64 methods filed at this node or below it, from the sources of 16 models.

In this branch 64

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 64

Automatic training recovery after failures core · 3 sources · 3 quotes
Checkpointing complete data-loader state core · 2 sources · 2 quotes
Checkpoint merging used · 4 sources · 5 quotes
Asynchronous checkpointing used · 3 sources · 3 quotes
59 single mentions
Model-agnostic training loop core · 1 source · 1 quote
Checkpointing every 250 training steps default · 1 source · 1 quote
Activation CPU offloading used · 1 source · 1 quote
Asynchronous fine-tuning used · 1 source · 1 quote
Autotune configuration generation used · 1 source · 1 quote
Checkpoint averaging used · 1 source · 1 quote
Cross-rank remote activation offloading used · 1 source · 1 quote
CUDA-event timing of transformer blocks used · 1 source · 1 quote
Double-buffered CUDA-event evaluation used · 1 source · 2 quotes
Double-buffered RDMA batch transfer used · 1 source · 1 quote
Element-wise activation recomputation used · 1 source · 1 quote
Fine-grained activation recomputation used · 1 source · 1 quote
Frequent checkpointing for fault tolerance used · 1 source · 1 quote
Full activation checkpointing used · 1 source · 1 quote
Fused full-logit KL-loss backward kernel used · 1 source · 1 quote
Heartbeat-driven fault tolerance used · 1 source · 1 quote
In-flight weight updates used · 1 source · 1 quote
Job-level eviction and reclaim used · 1 source · 1 quote
Keep optimizer states in CPU memory used · 1 source · 1 quote
Linear checkpoint merging in weight space used · 1 source · 1 quote
Node-local weight broadcast over NVLink used · 1 source · 1 quote
Periodic profiling for straggler diagnosis used · 1 source · 1 quote
Persistent checkpoint worker processes used · 1 source · 1 quote
Pre-admission hardware stress testing used · 1 source · 1 quote
PyTorch-profile-based straggler detection used · 1 source · 1 quote
Reuse one ElasticBuffer across MoE layers used · 1 source · 1 quote
Same-node sticky pod respawn used · 1 source · 1 quote
Slice-Granularity Elasticity used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash usedCheckpoint merging used—
MiMo-V2.6-Flash usedKeep optimizer states in CPU memory used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedAsymmetric local-read and single-writer cache paths usedAsynchronous checkpointing usedCaching the distributed checkpoint save plan usedCheckpoint merging usedIn-flight weight updates usedOverlapping NCCL transfers with device-to-host copies usedPersistent checkpoint worker processes usedPersistent shared-storage cache for compiled artifacts usedSeeding node-local storage from a warm shared cache used—
DeepSeek-V4-Flash coreTensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward graph traversal usedSelective tensor recomputation for activation-memory reduction used—
DeepSeek-V4-Pro coreTensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward graph traversal usedSelective tensor recomputation for activation-memory reduction used—
GLM-5.2 usedHeartbeat-driven fault tolerance used—
Gemma 4 31B usedSlice-Granularity Elasticity used—
Kimi K3 coreComposable tensor-granularity activation storage policies coreUnified pluggable activation storage manager coreCross-rank remote activation offloading usedDouble-buffered chunked streaming of reference-model weights usedElement-wise activation recomputation usedPolicy-model gradient-buffer reuse for reference-model weights usedPost-iteration NVMe offloading of training states used—
Kolibri-1 coreVersion control as shared training-recipe consensus coreVersion-controlled model-development workflows coreVersion-controlled training pipeline and recipe coreAutomated end-to-end training and evaluation on every code change defaultCheckpointing every 250 training steps defaultImmutable version pinning for hermetic training runs defaultAsynchronous checkpointing usedAutomatic faulty-node detection and replacement usedAutomatic training recovery after failures usedCentralized block-runtime measurement collection usedCheckpoint averaging usedCheckpoint merging usedCheckpointing complete data-loader state usedCross-rank collective-kernel profile matching usedCUDA-event timing of transformer blocks usedDefer inter-replica all-reduce and FSDP2 finalization until the last microbatch usedDouble-buffered CUDA-event evaluation usedDouble-buffered RDMA batch transfer usedFrequent checkpointing for fault tolerance usedFull activation checkpointing usedGPU-to-inference weight sync over GPUDirect RDMA usedIndependent proxy-task scheduling to fill cluster gaps usedNode-local weight broadcast over NVLink usedPeriodic evaluation of merged recent checkpoints usedPeriodic profiling for straggler diagnosis usedPyTorch-profile-based straggler detection usedRecompute expert dispatch during the backward pass usedRecompute MoE feed-forward activations instead of checkpointing them usedReproducible training runs tied to version-control commits usedReuse one ElasticBuffer across MoE layers usedWeight averaging of runs trained on different data mixes used—
LFM3-1.7B usedLinear checkpoint merging in weight space used—
Laguna-S-2.1 coreAutomatic training recovery after failures coreGPU-to-inference weight sync over GPUDirect RDMA usedJob-level eviction and reclaim usedPre-admission hardware stress testing usedSame-node sticky pod respawn used—
MiMo-V2.6-Pro usedKeep optimizer states in CPU memory used—
MiniMax-M3 usedAutotune configuration generation used—
Mistral-Large-4 usedActivation CPU offloading usedAsynchronous execution of supporting training workloads usedAutomated health checks and crash analysis for training operations usedFine-grained activation recomputation usedFused full-logit KL-loss backward kernel used—
VLA Foundry (Foundry-VLA-1.7B) coreCheckpointing complete data-loader state coreDeterministic per-rank random-number-generator seeding coreModel-agnostic training loop core—
Xiaomi-Robotics-1 usedAsynchronous fine-tuning used—
↑