Open Models Almanac
Methodsoptimizationoptimizer

methods · optimization

optimizer

29 methods filed at this node or below it, from the sources of 16 models.

In this branch 29

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 29

Muon core · 13 sources · 15 quotes
AdamW core · 7 sources · 10 quotes
Category-specific optimizer assignment core · 5 sources · 6 quotes
Per-Head Muon used · 3 sources · 4 quotes
Adam used · 2 sources · 2 quotes
Decoupled weight decay used · 2 sources · 2 quotes
Nesterov momentum used · 2 sources · 2 quotes
21 single mentions
Newton–Schulz orthogonalization default · 1 source · 2 quotes
Asynchronous Micro-Group pipeline used · 1 source · 1 quote
Canzona used · 1 source · 1 quote
CUDA graph capture of the optimizer step used · 1 source · 1 quote
Differential learning rates used · 1 source · 1 quote
Distributed AdamW used · 1 source · 2 quotes
Hybrid Muon and Adam optimization used · 1 source · 1 quote
Hybrid Newton–Schulz iterations used · 1 source · 1 quote
Learning-rate ratio tuning used · 1 source · 1 quote
Moonlight-style learning-rate scaling used · 1 source · 1 quote
Muon orthogonalization accuracy refinement used · 1 source · 1 quote
Muon Split used · 1 source · 1 quote
Muown used · 1 source · 1 quote
Nesterov Muon used · 1 source · 1 quote
Polar Express coefficient schedule used · 1 source · 1 quote
Spectral scaling convention for Muon used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
MiMo-V2.6-Flash usedMuown used—
DeepSeek-V4-Flash coreMuon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—
DeepSeek-V4-Pro coreMuon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—
GLM-5.2 usedMuon usedMuon Split used—
Inkling usedHybrid Muon and Adam optimization used—
Kimi K3 usedMuon usedPeer-to-peer shard retrieval for Muon orthogonalization usedPer-Head Muon used—
Kolibri-1 usedAdam usedAdamW usedCategory-specific optimizer assignment usedDecoupled weight decay usedLearning-rate ratio tuning usedMuon usedNesterov Muon usedSpectral scaling convention for Muon used—
Laguna-S-2.1 coreMuon coreMoonlight-style learning-rate scaling used—
LingBot-VLA 2.0 usedDifferential learning rates used—
MiMo-V2.5 usedAdamW used—
MiMo-V2.6-Pro usedMuown used—
Mistral-Large-4 usedAdamW usedMuon usedMuon with Nesterov momentum and spectral scaling used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedDistributed AdamW used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedDistributed AdamW used—
Qwen3.8-Flash-Next coreAdamW coreCategory-specific optimizer assignment coreMuon coreRestricting Muon to two-dimensional linear-map weights coreSplitting fused parameters before orthogonalization coreNewton–Schulz orthogonalization defaultAdam usedAsynchronous Micro-Group pipeline usedCanzona usedCUDA graph capture of the optimizer step usedMuon orthogonalization accuracy refinement usedNesterov momentum usedPer-head splitting of attention and GDN input projections usedPolar Express coefficient schedule usedSplitting SwiGLU fc1 into gate and up halves used—
π0.5 usedAdamW used—
↑