Open Models Almanac
Methodsoptimizationtraining stability

methods · optimization

training stability

23 methods filed at this node or below it, from the sources of 21 models.

In this branch 23

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 23

Global-norm gradient clipping used · 6 sources · 7 quotes
Exponential moving average (EMA) used · 2 sources · 2 quotes
Router replay used · 2 sources · 2 quotes
Weight padding used · 2 sources · 2 quotes
19 from one source, or only mentioned as not used
Per-token logit-gradient norm bound core · 1 source · 1 quote
Two-step prompt-and-head warm-up adaptation default · 1 source · 1 quote
Anti-hallucination training used · 1 source · 1 quote
Behavioral regularization used · 1 source · 1 quote
Loss masking for excessively stale tokens used · 1 source · 1 quote
Off-policy sample filtering used · 1 source · 1 quote
Per-token gradient-norm bounding used · 1 source · 1 quote
Soft dropping used · 1 source · 1 quote
Training stability stress testing used · 1 source · 2 quotes
Training-only embedding input dropout used · 1 source · 1 quote
Weight clipping used · 1 source · 1 quote
State-input dropout optional · 1 source · 1 quote
Activation or logit clipping not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash usedLoss masking for excessively stale tokens used—
Hy3 usedAnti-hallucination training used—
MiMo-V2.6-Flash usedBehavioral regularization usedSafeguards against training drift and reward hacking used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedWeight padding used—
GLM-5.2 usedOff-policy sample filtering used—
GR00T N1.7 optionalState-input dropout optional—
Inkling usedWeight decay coupled to learning rate squared used—
Kimi K3 usedPer-token regularization for off-policy stability usedSoft dropping usedWeight clipping used—
Kolibri-1 corePer-token logit-gradient norm bound coreGlobal-norm gradient clipping usedPer-token gradient-norm bounding used—
LFM3-1.7B usedTraining-only embedding input dropout used—
Laguna-S-2.1 usedCross-replica model-weight hash consistency checks usedExponential moving average (EMA) used—
LingBot-VLA 2.0 usedGlobal-norm gradient clipping used—
MiMo-V2.5 usedGlobal-norm gradient clipping used—
MiMo-V2.6-Pro usedBehavioral regularization usedSafeguards against training drift and reward hacking used—
Mistral-Large-4 usedRouter replay used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedGlobal-norm gradient clipping usedRouter replay used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedGlobal-norm gradient clipping usedRouter replay used—
Qwen3.8-Flash-Next usedSimulate large-scale training instability by raising the learning rate usedTraining stability stress testing usedActivation or logit clipping not used—
VLA Foundry (Foundry-VLA-1.7B) coreAction and proprioceptive-state normalization core—
X-VLA defaultTwo-step prompt-and-head warm-up adaptation default—
π0.5 usedGlobal-norm gradient clipping usedExponential moving average (EMA) optional—
↑