Open Models Almanac
Methodsoptimization

methods · pipeline stage 4 of 6

optimization

242 methods filed at this node or below it, from the sources of 37 models.

In this branch 242

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 16

Hyperparameter scaling-law fitting core · 3 sources · 4 quotes
Exponential moving average of model weights used · 2 sources · 2 quotes
Gradient checkpointing used · 2 sources · 3 quotes
Model souping used · 2 sources · 2 quotes
12 single mentions
Sample-based training budgets core · 1 source · 1 quote
Two-stage pre-training core · 1 source · 1 quote
Factorized hyperparameter transfer used · 1 source · 1 quote
Fused loss computation used · 1 source · 1 quote
Gradient accumulation used · 1 source · 2 quotes
Maximal Update Parametrization used · 1 source · 1 quote
Multi-epoch dataset training used · 1 source · 1 quote
Phase-by-phase training cost accounting used · 1 source · 1 quote
Magnitude-aware parameter dropout sampling evaluated · 1 source · 1 quote
Magnitude-based task-vector sparsification evaluated · 1 source · 1 quote
Majority-sign election evaluated · 1 source · 1 quote

optimizer 29

Muon core · 13 sources · 15 quotes
AdamW core · 7 sources · 10 quotes
Category-specific optimizer assignment core · 5 sources · 6 quotes
Per-Head Muon used · 3 sources · 4 quotes
Adam used · 2 sources · 2 quotes
Decoupled weight decay used · 2 sources · 2 quotes
Nesterov momentum used · 2 sources · 2 quotes
21 single mentions
Newton–Schulz orthogonalization default · 1 source · 2 quotes
Asynchronous Micro-Group pipeline used · 1 source · 1 quote
Canzona used · 1 source · 1 quote
CUDA graph capture of the optimizer step used · 1 source · 1 quote
Differential learning rates used · 1 source · 1 quote
Distributed AdamW used · 1 source · 2 quotes
Hybrid Muon and Adam optimization used · 1 source · 1 quote
Hybrid Newton–Schulz iterations used · 1 source · 1 quote
Learning-rate ratio tuning used · 1 source · 1 quote
Moonlight-style learning-rate scaling used · 1 source · 1 quote
Muon orthogonalization accuracy refinement used · 1 source · 1 quote
Muon Split used · 1 source · 1 quote
Muown used · 1 source · 1 quote
Nesterov Muon used · 1 source · 1 quote
Polar Express coefficient schedule used · 1 source · 1 quote
Spectral scaling convention for Muon used · 1 source · 1 quote

learning-rate schedule 16

Cosine decay default · 4 sources · 5 quotes
Batch-size warmup used · 4 sources · 8 quotes
Warmup-Stable-Decay used · 4 sources · 5 quotes
No batch-size warmup used · 2 sources · 2 quotes
11 from one source, or only mentioned as not used
Fixed elevated learning rate used · 1 source · 1 quote
Fixed learning rate used · 1 source · 1 quote
Learning-rate annealing used · 1 source · 1 quote
Learning-rate decay used · 1 source · 1 quote
Learning-rate tuning used · 1 source · 1 quote
Linear warmup followed by step decay used · 1 source · 1 quote
Warmup-Stable-Merge used · 1 source · 1 quote
WSD-specific scaling law used · 1 source · 1 quote
Learning-rate warmup not used · 1 source · 1 quote

training precision 30

NVFP4 core · 13 sources · 16 quotes
NVFP4 pretraining core · 5 sources · 5 quotes
BF16 mixed-precision training default · 5 sources · 5 quotes
BF16 precision used · 4 sources · 4 quotes
FP8 mixed-precision training used · 4 sources · 4 quotes
FP4 and FP8 mixed precision used · 2 sources · 2 quotes
FP8-precision reinforcement learning used · 2 sources · 2 quotes
Mixed-precision training used · 2 sources · 3 quotes
22 from one source, or only mentioned as not used
BF16 attention projections used · 1 source · 1 quote
Blockwise scales in generator and trainer used · 1 source · 1 quote
E2M1 floating-point format used · 1 source · 1 quote
FP32 attention-output retention used · 1 source · 1 quote
FP32 gradient reduction used · 1 source · 1 quote
FP32 language-model head used · 1 source · 1 quote
FP8 quantization of selected layers used · 1 source · 1 quote
FP8 residual-state storage used · 1 source · 1 quote
FP8 tensor-core training used · 1 source · 1 quote
High-precision final network layers used · 1 source · 1 quote
MXFP8 used · 1 source · 1 quote
MXFP8 Mamba output projections used · 1 source · 1 quote
NVFP4 fine-grained micro-block scaling used · 1 source · 1 quote
Two-dimensional block quantization used · 1 source · 1 quote
Automatic mixed precision optional · 1 source · 1 quote
BF16 expert activations evaluated · 1 source · 1 quote
BF16 language-model head evaluated · 1 source · 1 quote
BF16 gradient reduction not used · 1 source · 1 quote

quantization-aware training 17

Quantization-Aware Training core · 10 sources · 10 quotes
MXFP4 Quantization default · 3 sources · 3 quotes
Stochastic Rounding used · 5 sources · 6 quotes
MXFP4 Quantization-Aware Training used · 4 sources · 4 quotes
MXFP4 Weights with MXFP8 Activations used · 2 sources · 3 quotes
12 single mentions
FP8 Quantization-Aware Training core · 1 source · 1 quote
Quantize-Dequantize Training core · 1 source · 1 quote
Calibration Data used · 1 source · 1 quote
FP4 Quantization used · 1 source · 1 quote
Freezing Non-Target Parameters used · 1 source · 1 quote
INT4 Quantization-Aware Training used · 1 source · 1 quote
NVFP4 Training used · 1 source · 1 quote
Per-Block Scalar Scaling used · 1 source · 1 quote
Q4_0 Quantization Format used · 1 source · 1 quote
Random Hadamard Transform used · 1 source · 1 quote
SFT-Based Quantization-Aware Distillation used · 1 source · 1 quote
Straight-Through Estimator used · 1 source · 1 quote

training stability 23

Global-norm gradient clipping used · 6 sources · 7 quotes
Exponential moving average (EMA) used · 2 sources · 2 quotes
Router replay used · 2 sources · 2 quotes
Weight padding used · 2 sources · 2 quotes
19 from one source, or only mentioned as not used
Per-token logit-gradient norm bound core · 1 source · 1 quote
Two-step prompt-and-head warm-up adaptation default · 1 source · 1 quote
Anti-hallucination training used · 1 source · 1 quote
Behavioral regularization used · 1 source · 1 quote
Loss masking for excessively stale tokens used · 1 source · 1 quote
Off-policy sample filtering used · 1 source · 1 quote
Per-token gradient-norm bounding used · 1 source · 1 quote
Soft dropping used · 1 source · 1 quote
Training stability stress testing used · 1 source · 2 quotes
Training-only embedding input dropout used · 1 source · 1 quote
Weight clipping used · 1 source · 1 quote
State-input dropout optional · 1 source · 1 quote
Activation or logit clipping not used · 1 source · 1 quote

training parallelism 47

Expert parallelism used · 12 sources · 15 quotes
Tensor parallelism used · 6 sources · 7 quotes
Fully Sharded Data Parallel used · 5 sources · 6 quotes
Context parallelism used · 2 sources · 2 quotes
Pipeline parallelism used · 2 sources · 2 quotes
Tensor and expert parallelism used · 2 sources · 2 quotes
41 single mentions
FSDP2 with WebDataset streaming core · 1 source · 1 quote
MoonEP core · 1 source · 4 quotes
Redundant-expert capacity reservation core · 1 source · 1 quote
Sequence parallelism for activations core · 1 source · 1 quote
Action-expert-specific sharding groups used · 1 source · 1 quote
Batch-size scaling with GPU count used · 1 source · 1 quote
Cache-based pipeline communication used · 1 source · 1 quote
Communication-computation overlap used · 1 source · 1 quote
Dedicated reduce-scatter process group used · 1 source · 1 quote
Fully balanced expert-parallel training used · 1 source · 1 quote
Hybrid ZeRO bucket assignment for Muon used · 1 source · 1 quote
Intra-device context parallelism used · 1 source · 1 quote
KDA Context Parallelism used · 1 source · 1 quote
Pathways data-replica reduction used · 1 source · 1 quote
Sconv-aware tensor-parallel sharding used · 1 source · 1 quote
Three-buffer reduce-scatter used · 1 source · 1 quote
ZeRO-2 used · 1 source · 1 quote
ZeRO-3 optimizer-state sharding used · 1 source · 1 quote
Data and expert parallelism optional · 1 source · 1 quote
Data parallelism optional · 1 source · 1 quote
Model parallelism optional · 1 source · 1 quote
Prefill context parallelism optional · 1 source · 1 quote
Data-Weighted Data Parallelism mentioned · 1 source · 1 quote

training runtime 64

Automatic training recovery after failures core · 3 sources · 3 quotes
Checkpointing complete data-loader state core · 2 sources · 2 quotes
Checkpoint merging used · 4 sources · 5 quotes
Asynchronous checkpointing used · 3 sources · 3 quotes
59 single mentions
Model-agnostic training loop core · 1 source · 1 quote
Checkpointing every 250 training steps default · 1 source · 1 quote
Activation CPU offloading used · 1 source · 1 quote
Asynchronous fine-tuning used · 1 source · 1 quote
Autotune configuration generation used · 1 source · 1 quote
Checkpoint averaging used · 1 source · 1 quote
Cross-rank remote activation offloading used · 1 source · 1 quote
CUDA-event timing of transformer blocks used · 1 source · 1 quote
Double-buffered CUDA-event evaluation used · 1 source · 2 quotes
Double-buffered RDMA batch transfer used · 1 source · 1 quote
Element-wise activation recomputation used · 1 source · 1 quote
Fine-grained activation recomputation used · 1 source · 1 quote
Frequent checkpointing for fault tolerance used · 1 source · 1 quote
Full activation checkpointing used · 1 source · 1 quote
Fused full-logit KL-loss backward kernel used · 1 source · 1 quote
Heartbeat-driven fault tolerance used · 1 source · 1 quote
In-flight weight updates used · 1 source · 1 quote
Job-level eviction and reclaim used · 1 source · 1 quote
Keep optimizer states in CPU memory used · 1 source · 1 quote
Linear checkpoint merging in weight space used · 1 source · 1 quote
Node-local weight broadcast over NVLink used · 1 source · 1 quote
Periodic profiling for straggler diagnosis used · 1 source · 1 quote
Persistent checkpoint worker processes used · 1 source · 1 quote
Pre-admission hardware stress testing used · 1 source · 1 quote
PyTorch-profile-based straggler detection used · 1 source · 1 quote
Reuse one ElasticBuffer across MoE layers used · 1 source · 1 quote
Same-node sticky pod respawn used · 1 source · 1 quote
Slice-Granularity Elasticity used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled hereoptimizerlearning-rate scheduletraining precisionquantization-aware trainingtraining stabilitytraining parallelismtraining runtime
DeepSeek-V4.1-Flash used—————Loss masking for excessively stale tokens used——Checkpoint merging used—
Step-5-Preview optional———BF16 precision optional———Data and expert parallelism optionalData parallelism optionalExpert parallelism optionalTensor and expert parallelism optionalTensor parallelism optional——
MiniMax-M3.1-Flash used————Quantization-Aware Training used————
GLM-5.3 used——————Expert parallelism used——
Hy3 used—————Anti-hallucination training used———
MiMo-V2.6-Flash coreFused loss computation used—Muown used———Quantize-Dequantize Training coreMXFP4 Quantization-Aware Training used—Behavioral regularization usedSafeguards against training drift and reward hacking used—Window-bounded KV exchange for sliding-window layers used—Keep optimizer states in CPU memory used—
NVIDIA-Nemotron-3-Ultra-550B-A55B core——Cosine decay usedLearning-rate annealing usedWarmup-Stable-Decay used—NVFP4 coreNVFP4 pretraining coreBF16 precision usedE2M1 floating-point format usedFP32 gradient reduction usedFP8 quantization of selected layers usedMXFP8 usedTwo-dimensional block quantization usedBF16 gradient reduction not used—Stochastic Rounding used—Weight padding used—Context parallelism usedExpert parallelism usedPipeline parallelism usedTensor parallelism usedData-Weighted Data Parallelism mentioned—Asymmetric local-read and single-writer cache paths usedAsynchronous checkpointing usedCaching the distributed checkpoint save plan usedCheckpoint merging usedIn-flight weight updates usedOverlapping NCCL transfers with device-to-host copies usedPersistent checkpoint worker processes usedPersistent shared-storage cache for compiled artifacts usedSeeding node-local storage from a warm shared cache used—
DeepSeek-V4-Flash core—Muon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—Batch-size warmup used—FP4 and FP8 mixed precision used—FP4 Quantization usedMXFP4 Quantization-Aware Training usedQuantization-Aware Training usedStraight-Through Estimator used——All-to-all gradient exchange with local FP32 summation usedHybrid ZeRO bucket assignment for Muon usedKnapsack-based balanced assignment of dense parameter matrices usedModified DualPipe 1F1B pipeline overlap for mHC used—Tensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward graph traversal usedSelective tensor recomputation for activation-memory reduction used—
DeepSeek-V4-Pro core—Muon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—Batch-size warmup used—FP4 and FP8 mixed precision used—FP4 Quantization usedMXFP4 Quantization-Aware Training usedQuantization-Aware Training usedStraight-Through Estimator used——All-to-all gradient exchange with local FP32 summation usedHybrid ZeRO bucket assignment for Muon usedKnapsack-based balanced assignment of dense parameter matrices usedModified DualPipe 1F1B pipeline overlap for mHC used—Tensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward graph traversal usedSelective tensor recomputation for activation-memory reduction used—
GLM-5.2 used—Muon usedMuon Split used—Batch-size warmup usedCosine decay used—Mixed-precision training used—INT4 Quantization-Aware Training used—Off-policy sample filtering used—Expert parallelism used—Heartbeat-driven fault tolerance used—
GR00T N1.7 optional—————State-input dropout optional———
Gemma 4 31B core————Quantization-Aware Training corePer-Block Scalar Scaling usedQ4_0 Quantization Format used——Pathways data-replica reduction usedZeRO-3 optimizer-state sharding used—Slice-Granularity Elasticity used—
Inkling used—Hybrid Muon and Adam optimization used——NVFP4 optional——Weight decay coupled to learning rate squared used—Sconv-aware tensor-parallel sharding used——
Kimi K3 coreHyperparameter scaling-law fitting used—Muon usedPeer-to-peer shard retrieval for Muon orthogonalization usedPer-Head Muon used—Cosine decay defaultWarmup-Stable-Decay not used—Block-wise FP8 activation quantization with offload usedFP32 attention-output retention used—MXFP4 Quantization-Aware Training usedMXFP4 Weights with MXFP8 Activations usedQuantization-Aware Training used—Per-token regularization for off-policy stability usedSoft dropping usedWeight clipping used—MoonEP coreRedundant-expert capacity reservation coreSequence parallelism for activations coreCache-based pipeline communication usedDynamic context parallelism for multimodal samples usedExpert parallelism usedFully balanced expert-parallel training usedGPU planning kernel for redundant expert migration usedIntra-device context parallelism usedKDA Context Parallelism usedPipeline ZeRO-2 gradient sharding with CPU offloading usedpipeline-bubble scheduling of ViT computation used—Composable tensor-granularity activation storage policies coreUnified pluggable activation storage manager coreCross-rank remote activation offloading usedDouble-buffered chunked streaming of reference-model weights usedElement-wise activation recomputation usedPolicy-model gradient-buffer reuse for reference-model weights usedPost-iteration NVMe offloading of training states used—
Kolibri-1 coreFactorized hyperparameter transfer usedMaximal Update Parametrization usedModel souping used—Adam usedAdamW usedCategory-specific optimizer assignment usedDecoupled weight decay usedLearning-rate ratio tuning usedMuon usedNesterov Muon usedSpectral scaling convention for Muon used—Learning-rate tuning usedWarmup-Stable-Merge used—Mixed-precision distributed training with BF16 parameters and FP32 reductions used—FP8 Quantization-Aware Training coreQuantization-Aware Training used—Per-token logit-gradient norm bound coreGlobal-norm gradient clipping usedPer-token gradient-norm bounding used—Batch-size scaling with GPU count usedDedicated reduce-scatter process group usedKeep weights unsharded after backward across gradient-accumulation steps usedThree-buffer reduce-scatter used—Version control as shared training-recipe consensus coreVersion-controlled model-development workflows coreVersion-controlled training pipeline and recipe coreAutomated end-to-end training and evaluation on every code change defaultCheckpointing every 250 training steps defaultImmutable version pinning for hermetic training runs defaultAsynchronous checkpointing usedAutomatic faulty-node detection and replacement usedAutomatic training recovery after failures usedCentralized block-runtime measurement collection usedCheckpoint averaging usedCheckpoint merging usedCheckpointing complete data-loader state usedCross-rank collective-kernel profile matching usedCUDA-event timing of transformer blocks usedDefer inter-replica all-reduce and FSDP2 finalization until the last microbatch usedDouble-buffered CUDA-event evaluation usedDouble-buffered RDMA batch transfer usedFrequent checkpointing for fault tolerance usedFull activation checkpointing usedGPU-to-inference weight sync over GPUDirect RDMA usedIndependent proxy-task scheduling to fill cluster gaps usedNode-local weight broadcast over NVLink usedPeriodic evaluation of merged recent checkpoints usedPeriodic profiling for straggler diagnosis usedPyTorch-profile-based straggler detection usedRecompute expert dispatch during the backward pass usedRecompute MoE feed-forward activations instead of checkpointing them usedReproducible training runs tied to version-control commits usedReuse one ElasticBuffer across MoE layers usedWeight averaging of runs trained on different data mixes used—
LFM3-1.7B coreTwo-stage pre-training coreMagnitude-aware parameter dropout sampling evaluatedMagnitude-based task-vector sparsification evaluatedMajority-sign election evaluated—————Training-only embedding input dropout used—ZeRO-2 with strided activation checkpointing used—Linear checkpoint merging in weight space used—
Laguna-S-2.1 core—Muon coreMoonlight-style learning-rate scaling used—Warmup-Stable-Decay usedWSD-specific scaling law used—BF16 mixed-precision training defaultFP8-precision reinforcement learning used——Cross-replica model-weight hash consistency checks usedExponential moving average (EMA) used——Automatic training recovery after failures coreGPU-to-inference weight sync over GPUDirect RDMA usedJob-level eviction and reclaim usedPre-admission hardware stress testing usedSame-node sticky pod respawn used—
LingBot-VLA 2.0 used—Differential learning rates used—Linear warmup followed by step decay used—BF16 mixed-precision training usedMixed-precision policy with BF16 storage and communication and FP32 reductions used——Global-norm gradient clipping used—Action-expert-specific sharding groups usedFully Sharded Data Parallel used——
MiMo-V2.5 used—AdamW used—Cosine decay used—FP8 mixed-precision training used——Global-norm gradient clipping used———
MiMo-V2.5-Pro used———FP8 mixed-precision training used—————
MiMo-V2.6-Pro coreFused loss computation used—Muown used———Quantize-Dequantize Training coreMXFP4 Quantization-Aware Training used—Behavioral regularization usedSafeguards against training drift and reward hacking used—Window-bounded KV exchange for sliding-window layers used—Keep optimizer states in CPU memory used—
MiniMax-M3 used———————Autotune configuration generation used—
Mistral-Large-4 used—AdamW usedMuon usedMuon with Nesterov momentum and spectral scaling used—Batch-size warmup used—Blockwise scales in generator and trainer usedFP8 tensor-core training usedNVFP4 optional——Router replay used—Communication-computation overlap usedContext parallelism usedExpert parallelism usedLatency-optimized context-parallel collectives usedZeRO-2 usedPrefill context parallelism optional—Activation CPU offloading usedAsynchronous execution of supporting training workloads usedAutomated health checks and crash analysis for training operations usedFine-grained activation recomputation usedFused full-logit KL-loss backward kernel used—
MolmoAct 2 usedUnfreezing tokenizer embeddings and language-model head used———BF16 mixed-precision training usedFP32 layer normalization and rotary position embeddings used———Cross-device average token-count loss normalization usedFully Sharded Data Parallel usedGradient-hook synchronization for replicated LoRA parameters usedReplicated, unsharded LoRA parameters under FSDP used——
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core—Distributed AdamW used—Linear learning-rate warmup followed by a constant rate used—NVFP4 coreBF16 attention projections usedFP32 language-model head usedHigh-precision final network layers usedMXFP8 Mamba output projections usedNVFP4 fine-grained micro-block scaling usedNVFP4 pretraining usedBF16 expert activations evaluatedBF16 language-model head evaluated—Calibration Data usedFreezing Non-Target Parameters usedNVFP4 Training usedRandom Hadamard Transform usedSFT-Based Quantization-Aware Distillation usedStochastic Rounding used—Global-norm gradient clipping usedRouter replay used—Combined context, tensor, and expert parallel training usedCombined context, tensor, expert, and pipeline parallel training usedData and expert parallelism for calibration used——
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B core—Distributed AdamW used—Linear learning-rate warmup followed by a constant rate used—NVFP4 coreFP32 language-model head usedBF16 expert activations evaluatedBF16 language-model head evaluated—Calibration Data usedFreezing Non-Target Parameters usedSFT-Based Quantization-Aware Distillation usedStochastic Rounding used—Global-norm gradient clipping usedRouter replay used—Combined context, tensor, and expert parallel training usedCombined context, tensor, expert, and pipeline parallel training usedData and expert parallelism for calibration used——
OpenVLA / OpenVLA-OFT usedMulti-epoch dataset training used——Fixed learning rate usedLearning-rate decay usedLearning-rate warmup not used—Automatic mixed precision optional———Fully Sharded Data Parallel optional——
Qwen3.5-397B-A17B used——————Expert parallelism used——
Qwen3.6-35B-A3B used——————Tensor parallelism used——
Qwen3.8-Flash-Next coreHyperparameter scaling-law fitting corePhase-by-phase training cost accounting used—AdamW coreCategory-specific optimizer assignment coreMuon coreRestricting Muon to two-dimensional linear-map weights coreSplitting fused parameters before orthogonalization coreNewton–Schulz orthogonalization defaultAdam usedAsynchronous Micro-Group pipeline usedCanzona usedCUDA graph capture of the optimizer step usedMuon orthogonalization accuracy refinement usedNesterov momentum usedPer-head splitting of attention and GDN input projections usedPolar Express coefficient schedule usedSplitting SwiGLU fc1 into gate and up halves used—Fixed elevated learning rate usedNo batch-size warmup usedScaling-law fit for learning-rate and batch-size selection usedBatch-size warmup evaluated—FP8 residual-state storage used——Simulate large-scale training instability by raising the learning rate usedTraining stability stress testing usedActivation or logit clipping not used—NS-FLOP-balanced static parameter partitioning usedTensor and expert parallelism used——
SmolVLA usedGradient checkpointing optional———BF16 mixed-precision training used—————
Step-3.7-Flash used———NVFP4 optional———Expert parallelism usedTensor parallelism used——
VLA Foundry (Foundry-VLA-1.7B) coreSample-based training budgets coreGradient accumulation usedGradient checkpointing usedExponential moving average of model weights unclear———Mixed-precision training used——Action and proprioceptive-state normalization core—FSDP2 with WebDataset streaming coreFully Sharded Data Parallel optional—Checkpointing complete data-loader state coreDeterministic per-rank random-number-generator seeding coreModel-agnostic training loop core—
X-VLA default——Reduced learning rates for soft prompts and vision-language modules used———Two-step prompt-and-head warm-up adaptation default———
Xiaomi-Robotics-1 used———————Asynchronous fine-tuning used—
gpt-oss-120b default————MXFP4 Quantization default————
π0.5 usedExponential moving average of model weights used—AdamW used—Linear learning-rate warmup followed by a constant rate used———Global-norm gradient clipping usedExponential moving average (EMA) optional—Fully Sharded Data Parallel optionalModel parallelism optional——

Proposed children

Paths the corpus wanted and the taxonomy does not have. A human promotes them into the outline; the extraction cannot.

automated hyperparameter and experiment searchcoordinate descent, loss extrapolation, learning-rate landscape, experiment opti
↑