Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Model	filed here	optimizer	learning-rate schedule	training precision	quantization-aware training	training stability	training parallelism	training runtime
DeepSeek-V4.1-Flash core	Momentum update with Sinkhorn balancing core—	Per-Head Muon defaultSinkhorn-balanced update defaultAdamW usedMuon used—	Engram learning-rate scaling usedLinear warmup followed by cosine decay used—	—	—	Loss masking for excessively stale tokens used—	Communication-Computation Overlap coreLoad-Balanced Image Sharding usedPipeline Payload Extensions used—	Batch-level embedding prefetch usedOffline checkpoint merging used—
NVIDIA-Nemotron-3-Ultra-550B-A55B core	—	—	Cosine decay usedLearning-rate annealing usedWarmup-Stable-Decay used—	NVFP4 coreNVFP4 pre-training coreBF16 usedE2M1 usedFP32 gradient reduction usedMixed-FP8 quantization usedMXFP8 usedTwo-dimensional block quantization usedBF16 gradient reduction not used—	Stochastic Rounding usedStochastic Rounding for Mamba Cache used—	Weight padding used—	Context Parallelism usedExpert Parallelism usedPipeline Parallelism usedTensor Parallelism usedData-Weighted Data Parallelism mentioned—	Asymmetric local-read and single-writer cache paths usedAsynchronous checkpointing usedCaching the distributed checkpoint save plan usedIn-flight weight updates usedOffline checkpoint merging usedOverlapping NCCL transfers with device-to-host copies usedPersistent checkpoint worker processes usedPersistent shared-storage cache for compiled artifacts usedSeeding node-local storage from a warm shared cache used—
MiMo-V2.6-Flash core	Fused loss computation used—	Muown used—	—	—	Quantize-Dequantize Training coreFP4 Quantization-Aware Training used—	Behavioral regularization usedSafeguards against training drift and reward hacking used—	Window-Bounded KV Exchange under Context Parallelism used—	CPU-resident optimizer states used—
DeepSeek-V4-Flash core	—	Muon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—	Scheduled batch-size growth used—	FP4+FP8 mixed precision used—	FP4 Quantization usedFP4 Quantization-Aware Training usedQuantization-Aware Training usedStraight-Through Estimator used—	—	All-to-all Gradient Exchange with Local FP32 Summation usedHybrid ZeRO Bucket Assignment for Muon usedKnapsack-Based Balanced Assignment of Dense Parameter Matrices usedModified DualPipe 1F1B Pipeline Overlap for mHC used—	Tensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward traversal usedSelective tensor recomputation for activation-memory reduction used—
MiMo-V2.5 used	—	AdamW used—	Cosine decay used—	FP8 mixed-precision training used—	—	Gradient clipping used—	—	—
GLM-5.3 used	—	—	—	—	—	—	Expert Parallelism used—	—
Hy3 used	—	—	—	—	—	Anti-hallucination training used—	—	—
GLM-5.2 used	—	Muon usedMuon Split used—	Batch-size warmup usedCosine decay used—	Mixed-precision training used—	INT4 Quantization-Aware Training used—	Off-policy sample filtering used—	Expert Parallelism used—	Heartbeat-driven fault tolerance used—
MiniMax-M3 used	—	—	—	—	—	—	—	Autotune configuration generation used—
DeepSeek-V4-Pro core	—	Muon coreAdamW usedHybrid Newton–Schulz iterations usedNesterov momentum used—	Scheduled batch-size growth used—	FP4+FP8 mixed precision used—	FP4 Quantization usedFP4 Quantization-Aware Training usedQuantization-Aware Training usedStraight-Through Estimator used—	—	All-to-all Gradient Exchange with Local FP32 Summation usedHybrid ZeRO Bucket Assignment for Muon usedKnapsack-Based Balanced Assignment of Dense Parameter Matrices usedModified DualPipe 1F1B Pipeline Overlap for mHC used—	Tensor-level automatic-differentiation activation checkpointing coreMinimal recomputation-graph extraction by backward traversal usedSelective tensor recomputation for activation-memory reduction used—
Gemma 4 31B core	—	—	—	—	Quantization-Aware Training corePer-Block Scalar Scaling usedQ4_0 Quantization Format used—	—	Data Replica Reduction over the Data Center Network usedZeRO-3 Optimizer State Sharding used—	Slice-Granularity Elasticity used—
Inkling used	—	Hybrid optimization with Muon and Adam used—	—	NVFP4 optional—	—	Weight decay coupled to learning-rate squared used—	SConv-Aware Tensor-Parallel Sharding used—	—
Kimi K3 core	Updated hyperparameter scaling law used—	Muon usedPeer-to-peer shard retrieval for Muon orthogonalization usedPer-Head Muon used—	Cosine decay defaultWarmup-Stable-Decay not used—	Block-wise FP8 activation quantization with offload usedFP32 attention-output retention used—	MXFP4 Quantization-Aware Post-Training usedMXFP4 Weights with MXFP8 Activations usedQuantization-Aware Training used—	Per-token regularization for off-policy RL usedSoft dropping usedWeight clipping used—	MoonEP coreRedundant-Expert Capacity Reservation coreSequence Parallelism for Activations coreCache-Based Pipeline Communication usedDynamic Context Parallelism for Large Multimodal Samples usedExpert Parallelism usedFully Balanced Expert-Parallel Training usedGPU Planning Kernel for Redundant Expert Migration usedKDA Context Parallelism usedPipeline ZeRO-2 Gradient Sharding with CPU Offloading usedpipeline-bubble scheduling of ViT computation usedSM-Level Context Parallelism used—	Composable activation storage policies coreUnified pluggable activation storage manager coreCross-rank remote activation offloading usedDouble-buffered chunked streaming of reference-model weights usedElement-wise activation recomputation usedPolicy-model gradient-buffer reuse for reference-model weights usedPost-iteration NVMe offloading of training states used—
Laguna-S-2.1 core	—	Muon coreMoonlight-style learning-rate scaling used—	Warmup-Stable-Decay usedWSD-specific scaling law used—	BF16 mixed-precision training defaultFP8-precision reinforcement learning used—	—	Cross-replica model-weight hash consistency checks usedExponential moving average of checkpoints used—	—	In-flight recovery system coreGPU-to-GPU weight synchronization over GPUDirect RDMA usedJob-level eviction and reclaim usedPre-admission hardware stress testing usedSame-node sticky pod respawn used—
MiMo-V2.5-Pro used	—	—	—	FP8 mixed-precision training used—	—	—	—	—
MiMo-V2.6-Pro core	Fused loss computation used—	Muown used—	—	—	Quantize-Dequantize Training coreFP4 Quantization-Aware Training used—	Behavioral regularization usedSafeguards against training drift and reward hacking used—	Window-Bounded KV Exchange under Context Parallelism used—	CPU-resident optimizer states used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core	—	—	—	NVFP4 coreBF16 usedHigh-precision final network layers usedMXFP8 usedNVFP4 fine-grained micro-block scaling usedNVFP4 pre-training used—	NVFP4 Training usedRandom Hadamard Transforms usedStochastic Rounding of Gradients used—	—	—	—
Qwen3.5-397B-A17B used	—	—	—	—	—	—	Expert Parallelism used—	—
Qwen3.6-35B-A3B used	—	—	—	—	—	—	Tensor Parallelism used—	—
Qwen3.8-Flash-Next core	Updated hyperparameter scaling law corePhase-by-phase training cost accounting used—	AdamW for the MoE router coreCategory-specific assignment of Muon and AdamW coreMuon coreMuon restricted to two-dimensional linear-map weights coreSplit fused gradients before orthogonalization coreEight-step Newton–Schulz iteration defaultHybrid Muon and AdamW parameter-group optimizer assignment defaultAdam without weight decay for the N-gram embedding table usedAdamW for attention and GDN output gates usedAdamW for gated-residual low-rank projections usedAdamW for input embeddings and output head usedAsynchronous Micro-Group pipeline usedCanzona usedCUDA graph capture of the optimizer step usedMuon orthogonalization accuracy refinement usedNesterov momentum usedPer-head splitting of attention and GDN input projections usedPolar Express coefficient schedule usedSelected number of Newton–Schulz iterations usedSplit SwiGLU fc1 into gate and up sub-matrices used—	Fixed elevated learning rate usedScaling-law fit usedTraining without batch-size warmup usedBatch-size warmup with adjusted peak learning rate evaluatedBatch-size warmup with constant-batch optimum peak learning rate evaluatedBatch-size warmup not used—	FP8 storage for the residual state used—	—	Elevated constant-learning-rate training stability stress test usedLearning-rate elevation for training stability stress testing usedTraining stability stress testing usedActivation or logit clipping not used—	NS-FLOP-Balanced Static Parameter Partitioning usedTensor and Expert Parallelism with Degree Eight used—	—
Step-3.7-Flash used	—	—	—	BF16 usedNVFP4 optional—	—	—	Expert Parallelism usedTensor Parallelism used—	—
gpt-oss-120b default	—	—	—	—	FP4 Quantization default—	—	—	—