Mistral pre-trained Large 4 on 3,840 GB200 GPUs. It took 45 days, “or about 4.15 million GPU-hours”. NVIDIA’s Nemotron 3 Ultra diverged twice: its training loss rose instead of falling, and kept rising. After the second time, around 16T tokens, NVIDIA went back to the checkpoint (a saved copy of the weights and optimiser state) from 15T, started lowering the learning rate at once, and “eventually made the practical decision to cut the total pretraining token horizon down to 20T tokens”. A lab can repair a run of this size from a checkpoint, but it can’t start it again from scratch.
The loss is next-token cross-entropy. What a lab has to settle before the run starts is the number of tokens, the batch size, and how and when the learning rate rises and falls. It can’t try these at full size, so it tries them on proxies, smaller models trained on smaller budgets, and on laws fitted to the proxies’ results. This article goes through those choices as the labs report them, and what each costs when it transfers badly. The learning-rate schedule has the clearest trade-off of them.
Training recipes aren’t in the configs or modeling code we check, so everything here comes from documents: the labs’ reports first, then cards and blogs, then method papers. A few reports outside the site’s collection of lab documents are labelled “outside our corpus” where used: NVIDIA’s Nemotron 3 Super report, whose recipe Ultra inherits, its Nemotron 3 Nano report, and the reports for Moonshot’s K2 and Z.ai’s GLM-4.5. A blank in a table means we searched and didn’t find it. Tokens are pre-training tokens seen, repeats included, unless a sentence says otherwise. The primer introduces the stages.
What a run fixes in advance
The objective is next-token prediction. For a training sequence the model minimises
averaged over the sequences in a batch. Six of the eleven labs train a multi-token prediction head, an extra output that predicts tokens further ahead, as a second loss; NVIDIA, DeepSeek and Xiaomi state its weight, between 0.1 and 0.3 (P1). Liquid’s LFM2 adds distillation, training the model to also match a larger teacher model’s output distribution. NVIDIA’s Nemotron 3.5 Lightning goes furthest: it starts from the existing Nano 3 base and trains another 1T tokens on a loss that is 0.9 KL divergence to a larger teacher and 0.1 cross-entropy (distillation from a larger Nemotron 3, S6), so its “pre-training” never started from random weights.
Around the objective sit the settings this article is about. A step is one update of the weights. The batch is how many tokens go into one step’s gradient. The horizon is how many tokens the whole run sees. The learning rate scales each step. It rises from near zero over a warmup, because large steps on a random network are unstable, and its path after that is the schedule. When the rate is lowered at the end, the labs call that stretch the decay or cooldown; the two words mean the same thing. A run saves a checkpoint every few hundred to few thousand steps. Later stages, mid-training on a smaller, higher-quality mix and context extension on longer sequences, continue from a checkpoint (S2).
The worked example in this article is Aleph Alpha’s Kolibri, a mixture-of-experts model with 78.1B parameters, of which 3.46B are active per token. Its report is the most detailed public account of a pre-training run we have found. Each step takes 4,608 sequences of 16,384 tokens, 75.5M tokens, and 264,750 steps make 20T. A common estimate of training compute is floating-point operations, with the parameters used per token and the tokens: two per parameter for the forward pass and four for the backward pass. It leaves out attention over the context, which isn’t small at 16K tokens. For Kolibri it gives (computed), and the run took 768 B200 GPUs for 21 days.
How many tokens
The best-known rule for a budget comes from DeepMind’s Chinchilla study. For a fixed amount of training compute, the loss is lowest when parameters and tokens grow together, at about 20 tokens per parameter; Chinchilla itself was a 70B dense model trained on 1.4T tokens. (One of the study’s three methods gave a higher ratio, but Besiroglu et al.’s replication found that fit flawed and their refit gives about 20.) Among the models with a stated count, DeepSeek’s V4-Pro comes closest, at about 21 tokens per total parameter, and every other run is above it, from about 29 (MiMo-V2.6-Pro) up.
pre-training text onlyincludes a later stageincludes image, audio or video tokensstages not statedboth (Inkling)dashed lines: tokens per parameter
V4 and V4.1 are DeepSeek's, MiMo is MiMo-V2.6, Ultra, Super and Nano are Nemotron 3, M.1 and XS.2 are Laguna. Lighter: earlier or sibling models, not in the map (GLM-5, Nemotron 3 Super, Nemotron 3 Nano, Laguna M.1, Laguna XS.2). Current models with no stated count, not drawn: Mistral Large 4, Kimi K3, Qwen3.8.
Hover over or focus a dot for the model, its count and the ratio.
Ultra sits at 36 tokens per total parameter, Kolibri at 256, Poolside’s Laguna XS.2 at about 900, and Liquid’s LFM3-1.7B at about 13,000 (computed). Chinchilla’s rule minimises training compute only. A model that serves many requests also costs compute for every token it generates, so a smaller model trained longer can be cheaper overall, as Sardana et al. work out; none of the reports in the map gives that reason. Mistral (for Large 4) and Moonshot (for K3) treat the ratio as a setting to tune, but give no value.
A token count also has to say what it includes. GLM-5’s 28.5T includes mid-training, and LFM3’s 22T includes 3.5T of it. Xiaomi’s 1.02T-parameter MiMo-V2.6-Pro saw 30T tokens against 48T for the 310B Flash, but in the text stage Pro saw 27T and Flash 26T; the gap is a stage of image, audio and video tokens, which don’t carry what a text token carries. For mixture-of-experts models, “per parameter” also needs a denominator, since Chinchilla was fitted on dense models. Sequences start at 4K to 64K tokens, and long contexts come later (S2). NVIDIA and Mistral split pre-training into two data phases (two-stage pre-training, D2).
Choosing settings on proxies
A proxy is cheap only if what’s learned on it carries over. The usual tool is a power law. Chinchilla fitted the loss of dense models as
with fitted constants . is the loss no model can beat, and the other two terms shrink as the model grows and as it trains longer. The same idea applies to a hyperparameter: train small models at several learning rates, find the best rate for each size and budget, fit a law to those optima, and read off the value for the real run.
Poolside’s report shows every step of one such fit (a WSD-specific scaling law). It swept four model sizes, from 0.3B to 2.2B active parameters, six learning rates and five budgets from 30B to 480B tokens, all at a batch of 8M tokens. The fitted law for the best peak learning rate is
with the active parameters and the tokens, both as raw counts. For Laguna XS.2, with 3.0B active parameters and 30T tokens, the law gives . XS.2 trained at a batch of 24M tokens, three times the fit’s, and Poolside scaled the rate by the square root of that ratio, the usual rule for Adam-type optimisers: (computed; it matches the report’s “∼5.5×10⁻⁴” only with the square root). Poolside used , “to leave a small safety margin”. (The map’s Laguna S 2.1 is a later scale-up of the same family, and its own settings aren’t published.)
A margin on the low side is the right direction: near the best rate the loss changes slowly, while a rate too high risks divergence. A companion fit in Poolside’s appendix misses the mean best rate of each proxy size by 0.91× to 1.14×, and a cross-check on Moonshot’s K2 predicts 1.75 times the rate K2 used, which Poolside calls “suggestive rather than a validation”. A least-squares line is best determined at the centre of its data, here 120B tokens if the five budgets double from 30B to 480B as the main text says (the appendix lists 60B to 960B), so an error of 0.02 in the exponent of moves the prediction at 30T by 12% (computed): more than the margin. Poolside gives no standard errors; from its reported of 0.89 we estimate one at about 0.04, which would move the prediction by about 25%.
N = 3.0B active parameters, batch 24M tokens (√3 times the law at its fitted 8M). The fan pivots at 120B, the geometric mean of the fitted budgets. Poolside reports no error for the exponent; the marks on the slider are 0.04, ≈ one standard error (our estimate from the reported R²), and 0.054, Bjorck et al.'s exponent.
At 30T the law gives 5.54×10⁻⁴; Poolside used 5.0×10⁻⁴, 0.90 of it. With the D exponent off by 0.020, the prediction would be 4.96×10⁻⁴ (+Δ) or 6.19×10⁻⁴ (−Δ), a factor of 1.117 either way.
The horizon has its own transfer problem. Bjorck et al. found that the best learning rate falls as a run gets longer, roughly as , which matches the direction of stochastic gradient descent theory (for a convex loss with noisy gradients, the best constant step over steps falls as ). Their rule was fitted below 800B tokens, so using it at 20T is an extrapolation too.
Mistral predicts its peak rate with a log-linear function of width, depth, horizon, expert ratio and batch, checked on a scaling ladder of proxies that “differs from the reference model in scale only”, and gives no sizes or errors. Qwen refitted its law and checked it on a 156B-parameter proxy: the old recipe ended 0.0078 higher in loss than the new law’s settings. Only Poolside publishes its law with values.
Aleph Alpha dropped laws for the best hyperparameters for Kolibri, because laws fitted for its earlier Kolibri Origin gave “unrealistic hyperparameter predictions” when extended. It split the transfer in two (factorised transfer). For width it used µP, a parametrisation meant to keep the best rate fixed as a model gets wider; on proxies 1,024 and 2,048 wide the best rate moved by , “which we deem acceptable”. For the horizon it trained a half-width proxy to 500B tokens at several settings, extrapolated each setting’s loss curve 40 times further to 20T, and kept the setting with the lowest forecast. That is still a power law, but fitted on the curve each setting actually traced. The report gives “an average prediction error at the target token horizon below 1 %”, on curves “not reported here”, while the settings it ranks differ by about as much (the larger batch was forecast “0.9 % behind”). The best learning rate, the batch and the weight decay all sat at the edge of the grids tried.
Some problems never show up in a proxy’s loss. Mistral computes its load-balancing loss per group of GPUs that share experts, so “downscaling experiments must be careful when changing partitioning”. A cache-saving attention design also looked fine on the proxies’ perplexity, but when Mistral fine-tuned intermediate checkpoints of the real run on example answers (SFT), these “early SFT cooldowns on intermediate checkpoints revealed poor performance” on agentic, multi-step tasks. It changed the architecture mid-run and trained “approximately two-thirds of the original phase 2”.
The learning-rate schedule
A high learning rate makes fast progress but leaves the weights moving around the bottom of a valley in the loss. For plain stochastic gradient descent near a minimum, the extra loss this costs is about , with the batch and the covariance of one example’s gradient (a standard result for a locally quadratic loss). Lowering the rate lowers that floor, and the loss drops quickly when it does. But the total area under the learning-rate curve drives progress (Tissue et al. fit the loss on it, and in the classical error bound for stochastic gradient descent the sum of the rates is the progress term), so decaying early gives up progress for the rest of the run. Choosing a schedule means choosing how long to stay high and how to come down.
A cosine schedule lowers the rate along half a cosine, from the end of warmup () over the steps that remain:
It has no knob beyond the peak and the floor, but it needs before the run starts. The Chinchilla authors found that a cosine set more than 25% longer than the run, so that the run stops before the rate has come down, “noticeably degraded” the result.
Warmup-stable-decay (WSD), named in the MiniCPM paper, holds the peak rate and decays only over a final stretch. The decay can start from any checkpoint in the stable phase, so one long stable run can be finished several ways: to compare data mixes, or to produce a model at a shorter horizon. The cost is three more settings: the decay’s length, its shape and its floor. A common shape is over the decay fraction , which NVIDIA calls minus-sqrt. It falls to half the peak a quarter of the way through the decay, where a cosine-shaped decay over the same stretch is still at 85% (computed).
Ultra (Nemotron 3 Ultra): WSD, minus-sqrt; decay start set by a rollbackXS.2 (Laguna XS.2): WSD, 1−√ (drawn: warmup 1%, assumed)V4.1 (DeepSeek V4.1): no name: hold, cosine, holdK3 (Kimi K3): cosine; floor not stated; a cooldown stage follows (drawn: floor 10%, assumed)GLM-5: cosine to 20% (drawn: warmup 1%, assumed)Kolibri: constant, then merge
Drawn but not stated: Laguna XS.2 warmup 1%; Kimi K3 floor 10%; GLM-5 warmup 1%. Dashed: the same schedule finished at the slider's point. Highlighted lines are in pre-training's colour, the rest grey.
Laguna XS.2: decay from 42% to 60% of the full run, the same shape and share. Kimi K3: at 60% the cosine is still at 42% of peak; a cosine has to be restarted or decayed off-schedule.
The labels don’t predict the shapes. Ultra’s decay started at 15T because of the rollback, not by plan; Poolside’s larger Laguna M.1 used cosine where XS.2 used WSD; Moonshot’s K3 uses cosine where its K2 (outside our corpus) used WSD.
The one per-schedule comparison a lab has published is Moonshot’s. Its scaling-law study, on proxies whose sizes it doesn’t give, ran “an independent scaling-law search for each schedule” and found: “Under their respective optimal hyperparameter settings, cosine decay consistently achieves a lower final loss than WSD.” Both decayed to the same fixed floor, and the report gives no budgets or size of the gap. Z.ai’s GLM-4.5 report (outside our corpus) also chose cosine, after WSD did worse on two benchmarks, without saying whether each schedule was tuned.
Poolside chose WSD for checkpoint reuse, and states the cost: “WSD is attractive because a single stable-phase checkpoint can be reused across data-mix iterations via shorter cooldown runs. But, according to our ablations, final-loss calibration is less direct than for one-shot cosine, and naively tuned WSD runs sometimes underperform tuned cosine at matched compute.”
Method papers at published scales disagree with Moonshot. Hägele et al., on models from 124M to 1B parameters (and one short 8B run), found that WSD with a cooldown matches a well-tuned cosine, and Tissue et al. found WSD lower. Schaipp et al. show one reason such comparisons are fragile: in a convex-optimisation bound and in their experiments, the best peak rate for cosine is about twice that for WSD, so a comparison at one shared peak rate favours whichever schedule that rate suits. Moonshot’s per-schedule search answers that objection. No study at full scale tunes both schedules and reports the gap.
At most labs the decay is also where the data changes. Ultra’s decay and its higher-quality second data phase both cover the last 5T; GLM-5, MiMo-V2-Flash and Liquid continue the decay into mid-training, and K3 extends its context during its “cooldown phase”. Kolibri’s quality-heavy mid-training runs at the constant peak instead. None of the comparisons above says it held the data fixed, so a gain credited to a decay may partly belong to the data (D2).
Merging in place of decay
There is a third option: never decay, and average checkpoints instead. Averaging the iterates of stochastic gradient descent is an old idea (Polyak–Ruppert averaging, and later stochastic weight averaging). The WSM paper by Tian et al. ties it to decay.
Take a base checkpoint and the next saved checkpoints, and write for everything the optimiser changed between two of them, often hundreds of steps. Each checkpoint is the base plus the changes so far, . A merge with weights summing to 1 is then, after swapping the two sums,
This identity holds for any run. It says the merge keeps a fraction of the -th block of changes. Since each change is a learning rate times a direction, keeping a fraction of it is like having taken that block at times the rate: a decay schedule, applied after the fact, constant within each block. The weights start at and fall when every . Every non-increasing decay between 0 and 1 has merge weights that produce it (, , ). A plain average of the checkpoints after the base, with none on the base, gives : a linear decay (computed).
Exact for the updates the run made. A run trained with this decay would have made different updates. The EMA weighs the ten checkpoints after the base by β = 0.8 per checkpoint (for illustration), normalised so the base gets none.
linear: the base checkpoint gets c₀ = 0.000; the 10 after it get 0.100 each, a plain average; the weights sum to 1.000.
What isn’t exact is the step to “the same as training with that decay”: a run at the lower rates would have computed different changes, from different weights. In Maskey et al.’s words, a merge “is not equivalent to cosine cooldown because the optimizer never follows the decayed trajectory.” Classical results on averaging say what it can do. Near a minimum, a constant-rate run keeps moving around it, and an average over a long enough window removes most of that movement. But on a loss that isn’t quadratic, the points the run visits are centred off the minimum by an amount proportional to the learning rate (Dieuleveut et al.), and only a lower rate removes that offset. Hägele et al. found, at 210M parameters, that averaging narrows the gap to a cooldown without closing it.
Labs use merges for different jobs. NVIDIA scored merges over a sliding 500B-token window throughout Ultra’s pre-training, “weighted to emulate our learning rate decay schedule”, as a cheap preview of a decay; its Super report (outside our corpus) puts the best merge 2 to 4 points above the raw stable-phase checkpoint on a 12-benchmark average, and estimates that the dedicated decay runs it avoids could cost “roughly 16% of the total pretraining FLOP budget”. After Ultra’s decay, NVIDIA also started long-context training from a 500B-window merge. Laguna XS.2 releases an exponential moving average of its 10 most recent checkpoints, also after a decay. WSM found that merging after a decay adds little, so only Kolibri’s merge replaces one.
Kolibri never decays. It holds the rate constant through pre-training, mid-training and context extension, and its base model is the plain average of the last 20 checkpoints, 10B tokens apart (Warmup-Stable-Merge). The reason given: “In our experiments, a constant learning rate with checkpoint merging improves post-trained performance over a learning-rate cooldown (Maskey et al. 2026).”
That citation is Aleph Alpha’s own study, on one 30B-parameter model with one seed per setup, comparing a constant rate, a cooldown to 10% of peak over the last 800B tokens, and a merge of the constant run. Before fine-tuning, the cooldown beat the constant run on every signal (validation loss 1.648 against 1.734), but the merge scored highest on the benchmark aggregate (0.460, against 0.440 and 0.415). Mid-training then warmed all three back up to the full peak rate, so the cooldown run left its low rate after one stage. After mid-training, context extension and fine-tuning, the aggregate was 0.363 for the merge and 0.360 for the constant run against 0.247 for the cooldown, a gap that shrinks to 0.016 without HumanEval+, where the cooldown model failed to stop generating. The cooldown’s mid-training and long-context schedule was the best of nine, and the other two runs reused it once, which tilts the test toward the cooldown; even the best of 27 cooldown runs, counting an SFT-rate sweep, reached only 0.294. Yano et al., at 1B and 8B, independently report that skipping the decay helps after fine-tuning.
NVIDIA’s Super report tested merging against decay at the scale of Nemotron 3 Nano and kept its decay: it reproduced WSM’s gains “when emulating short (∼500B) decay windows. However, direct comparisons at 1T and 1.5T merge horizons showed no improvement over decay-trained checkpoints.” NVIDIA scored base models and Aleph Alpha scored after fine-tuning; no study has done both. Merging’s own costs are checkpoints to store, a window and weights to set, and a released base that was never decayed.
Batch size
A larger batch averages more examples (here, tokens) into each gradient, so each step points closer to the true gradient . Past some size, the extra examples stop helping. McCandlish et al. measure this with a gradient noise scale, in its simple form
the total variance of the per-example gradients over the squared norm of the mean gradient; their full version weights both by the curvature, and they find the two differ by a roughly constant factor. Well below it, doubling the batch nearly halves the steps; well above it, the extra examples are wasted. Their model gives a hyperbola between the steps and the tokens needed to reach a given loss:
Since , and the critical batch size is , this gives and . At a run takes twice the minimum steps and twice the minimum tokens.
"Later in the run" multiplies B_crit by 4, a factor chosen for illustration; McCandlish et al. report growth of "at least an order of magnitude" over training on their tasks.
B = 1 × B_crit: K/K_min = 2.00, D/D_min = 2.00.
The noise scale grows as training goes on, because the mean gradient shrinks while the per-example noise doesn’t: by “at least an order of magnitude” in McCandlish et al.’s runs, which were on image, game and small language tasks. That argues for a batch-size ramp, starting small and growing, though McCandlish et al. call the expected gain “relatively modest unless the B_crit changes greatly during training”. A ramp is also part of the schedule: in the floor formula above is what matters, so a fourfold larger batch lowers the floor about as much as a fourfold decay in plain stochastic gradient descent (Smith et al.), or, by the square-root rule’s heuristic for Adam, a twofold one.
Mistral gives the clearest case for a ramp: “Early in training, a larger batch size wastes compute (and tokens) because a gradient estimate from a fraction of the batch’s tokens suffices, whereas later in training, we benefit from more precise gradients.” Large 4 ramps from 40M to 80M to 160M (tokens, by our reading), spends the last half of the run at the largest, and retunes the learning rate at each stage. A ramp from 0.25× to 2× of its predicted batch worked best for optimisation, but Mistral chose 1× to 4× because “larger batch sizes achieve higher MFU in our setting”, MFU being the share of the hardware’s peak arithmetic a run uses.
DeepSeek V4, MiMo-V2-Flash and GLM-5 ramp too. Qwen tested a ramp for Qwen3.8 on a 10.8B-parameter proxy over 4T tokens: “Neither variant improves performance.” The ramps ended 0.00025 and 0.00035 higher in loss, “within the run-to-run variance”, which Qwen doesn’t size, and took 18.8% more optimiser steps (training without batch-size warmup). DeepSeek V4.1 keeps its batch “fixed at 100.6 million tokens throughout training” without saying why, and NVIDIA, Poolside and Aleph Alpha don’t discuss ramps.
The choice doesn’t follow the optimizer (O1). Mistral and MiMo ramp with AdamW and NVIDIA doesn’t; DeepSeek V4 and GLM-5 ramp with Muon, and DeepSeek V4.1 and Qwen don’t.
When a run goes wrong
Four of the eleven labs describe a production pre-training run going wrong: NVIDIA, Poolside, DeepSeek and Aleph Alpha. A few terms first. Most runs keep weights and activations in BF16, a 16-bit format with FP32’s range but far fewer digits, and Ultra trained most layers in the 4-bit NVFP4, checking it against BF16 branches started from its 5T, 10T and 16T checkpoints (O2). An all-reduce sums a tensor across GPUs; tensor parallelism splits one layer’s matrices over several GPUs, so their partial results have to be summed.
| Run | When | What happened | Stated cause | Fix |
|---|---|---|---|---|
| Nemotron 3 Ultra | ~8T | loss and weight-gradient norm rise; a multi-token head’s loss spikes first | the gradient from the multi-token heads into the shared output layer was lost when its accumulation moved “from FP32 to BF16” | roll back, reduce gradients in FP32 |
| Nemotron 3 Ultra | ~16T | loss and weight-gradient norm rise | “we could not find a smoking gun” | roll back to 15T, decay at once, end at 20T |
| Laguna M.1 | not stated | logits drift upward | no z-loss to hold logits down, so the BF16 all-reduce of the output layer’s input gradients became “the dominant source of numerical error” | do that reduction in FP32 |
| DeepSeek V4 | not stated | recurring loss spikes | “consistently tied to outliers in the MoE layers”, with routing that “appears to exacerbate” them | Anticipatory Routing after a short rollback; SwiGLU activations clamped |
| Kolibri Origin | after “a few trillion tokens” | not stated | “a data-shuffling bug that escaped our tests” | restart from scratch |
Two of the four diagnosed incidents trace to the precision of a sum at the output layer, by different paths (our grouping). In Ultra’s case the lost signal came from small auxiliary losses (weight 0.05 per head), and that head’s loss spiked first. DeepSeek’s fix, Anticipatory Routing, computes expert assignments with slightly older weights, switches on after a spike, and costs about 20% in wall-clock time while it runs; its mechanism “remains an open question”. Mistral’s mid-run architecture change was a capability problem rather than a failing loss.
The commonest failure is mundane. Kolibri had 38 unplanned interruptions in 21 days, each restarted from a checkpoint at most 250 steps back, up to about 19B tokens recomputed (computed), and Mistral reports 94.1% of its time spent on useful training. Checkpoint spacing sets both the cost of a rollback and which merges are possible. Kolibri Origin’s restart came from the data pipeline, which no loss curve flagged.
Other labs report clean runs: DeepSeek V4.1 “with no instability”, Qwen3.8 “without a single loss spike”, Laguna XS.2 “without encountering further stability issues”. A clean run is weak evidence by itself, since there is no matched run without the recipe and reports describe the runs that shipped. The routine guards are rarely stated. Gradient clipping, rescaling any gradient whose norm exceeds a threshold, is given at 1.0 by Kolibri and MiMo-V2-Flash. Weight decay is 0.1 at 4 of the 5 labs that state a pre-training value; Aleph Alpha’s looks 400 times smaller, but it applies per step regardless of the learning rate, where AdamW’s 0.1 is usually multiplied by the rate, which makes Kolibri’s the stronger at Ultra’s peak rate (computed; NVIDIA doesn’t state its convention).
Qwen is the one lab that reports a stability stress test (stability stress testing). Following Wortsman et al.’s observation that instabilities of large runs reappear in small ones at high learning rates, it holds a 25B-parameter proxy’s rate at two and four times its optimum, with no decay, and counts loss spikes. The criterion is “that the new recipe must be at least as stable as the generation it replaces under equal stress”.
AdamW, Qwen3.5 structureMuonMuon + gated residual
Each proxy is held at a constant multiple of its own best learning rate, with no decay. A spike is a step whose loss is more than 0.1 above the rolling median of 201 steps. At 4× Qwen gives no spike count for Muon alone, only that it never crossed the clipping threshold. Panel B toggles only the gate (GatedNorm) of the gated residual, under AdamW with the Qwen3.5 structure; its 3× is relative to that setup's own optimum.
At twice the optimum, the old structure with AdamW spiked 4.3 times per 10,000 steps and both Muon runs 0.2. At four times, AdamW spiked 183 times and Muon with the gated residual not at all. Qwen credits “the new architecture and optimizer combination” without ranking the two. Because both recipes run on the same proxy, whatever the proxy gets wrong for both cancels. Nothing in the test shows that the margin survives at full size, where Wortsman et al. found sensitivity to the learning rate is larger, and none of the incidents in the table had a learning rate as its stated cause. If the Muon run was as long as the AdamW run’s 19,932 steps, zero spikes bounds its rate below about 1.5 per 10,000 at 95% confidence, not at zero (computed).
The choices side by side
| Choice | What it decides | What it buys | What it costs | Evidence |
|---|---|---|---|---|
| Fitted law for hyperparameters | peak rate, batch | values for a run no one can try | extrapolation far past the fit; a safety margin chosen by hand | only Poolside publishes its law with values; its reported errors exceed its margin |
| Factorised transfer (µP + loss forecast) | rate and batch at width, then at horizon | no law for optimal hyperparameters across sizes | still a 40× loss forecast; settings at grid edges | Kolibri: under 1% forecast error on unreported curves; ranked gaps about 1% |
| Cosine | when the rate falls | lower final loss in Moonshot’s tuned proxy comparison | length fixed in advance | Moonshot, Z.ai (benchmarks); Hägele, Tissue find parity or the reverse |
| WSD | when the rate falls | stable checkpoints that can be decayed several ways | decay length, shape and floor to set | Poolside, NVIDIA use it; decay usually coincides with a data change |
| Constant rate + merge | whether the rate falls at all | no decay; Aleph Alpha reports better scores after fine-tuning | window and weights to set; a base never decayed | Aleph Alpha (one seed) for; NVIDIA finds no gain at 1T to 1.5T windows |
| Batch ramp | tokens per step over time | fewer wasted tokens early; acts partly as a decay | more steps; lower MFU while the batch is small | Mistral, DeepSeek V4, MiMo, GLM-5 ramp; Qwen finds no gain on a proxy |
| Stress test | whether a recipe is stable enough | a failure seen on a proxy, before the run | proxy compute; tests learning-rate instability only | Qwen only |
Where the models in the map sit
| Lab | Model | Tokens (kind) | Starting length | Schedule | Batch (tokens) | Merging | Incident reported |
|---|---|---|---|---|---|---|---|
| Aleph Alpha | Kolibri | 20T pre | 16K | constant after 100B warmup | fixed 75.5M | released base: mean of last 20 | Origin restarted |
| Mistral | Large 4 | 4K | ramp 40M → 160M | regression found mid-run | |||
| NVIDIA | Nemotron 3 Ultra | 20T text | 8K | WSD, minus-sqrt over last 25% | fixed 25.2M | evaluation, selection | two divergences |
| Poolside | Laguna S 2.1 (recipe from the XS.2 and M.1 report) | XS.2: >30T | XS.2: 4K | XS.2: WSD, 1−√ over last 30% (M.1 cosine) | XS.2: fixed 24M | XS.2: EMA of 10 | M.1: logit drift |
| Moonshot | Kimi K3 | 8K | cosine | ||||
| DeepSeek | V4.1-Flash (V4) | 45T, multimodal, incl. 1M extension | 64K (V4 4K) | hold, cosine, hold | fixed 100.6M (V4 ramp) | V4: loss spikes | |
| Qwen | Qwen3.8-Flash-Next | 1/3 of Qwen3.5-397B’s | fixed | ||||
| Z.ai | GLM-5 | 28.5T incl. mid | 4K | cosine | ramp | ||
| Xiaomi | MiMo-V2.6-Flash (V2-Flash) | 48T incl. omni, excl. mid | 32K | V2-Flash: hold, cosine | V2-Flash: ramp | ||
| Liquid | LFM3-1.7B | 22T incl. mid | |||||
| Thinking Machines | Inkling | 45T, multimodal |
Mistral never states its token count, and Moonshot and Qwen give none in absolute terms. Kolibri’s own summary table counts 24T as “Pre-training tokens”, folding in its mid-training and context extension. Learning rates in these reports don’t compare directly across optimizers and conventions (O1), and GPU-hours compare only on the same hardware and scope.
Easy to get wrong
- “20T tokens on a 78B model is overtrained.” Chinchilla’s 20 tokens per parameter minimises training compute for one loss, on dense models. It leaves out the cost of serving, and for a mixture-of-experts model it doesn’t say which parameters to count.
- “Cosine beats WSD.” One lab found it on proxies, after tuning each schedule separately. Published studies find parity or the reverse, and at most labs the decay also brings new data.
- “Merging beats decay.” Aleph Alpha finds it after fine-tuning, in a one-seed study whose later stages re-warm the cooldown run to its peak, with most of the gap from one benchmark failure. NVIDIA finds no base-model gain from merging at 1T to 1.5T decay windows.
- “Qwen showed that batch ramps hurt.” Its ramps were worse by an amount within run-to-run variance, on a proxy. They didn’t help, and they cost 18.8% more steps.
- “The run was clean, so the recipe is stable.” Clean runs are what gets published, and none of them has a matched run to compare with.
Check yourself
Poolside’s law gives at its fitted batch of 8M tokens. What does it give at 24M tokens if the rate scales with the square root of the batch, and if it scales linearly? The report says about . Which rule did Poolside use?
, and . The report’s number matches the square root.
You average the five checkpoints that follow a base checkpoint, with equal weights, and give the base itself no weight. What fraction of each block of changes does the merge keep, and why isn’t the result the same as having trained with that decay?
: a linear decay over the five blocks. The changes were computed at the full rate, from weights that a decayed run would never have reached, so the merged weights are built from updates a decayed run wouldn’t have made.
At four times the critical batch size, how many tokens and how many steps does a run need, relative to the minimum of each? Which of the two does a larger batch save?
and : five times the minimum tokens and 25% more than the minimum steps. A larger batch saves steps, so wall-clock time, and spends tokens. Mistral’s choice of 1× to 4× of its predicted batch over 0.25× to 2× is that trade, made because larger batches ran at higher MFU.
Qwen’s stress test compares the new recipe with the old one on the same proxy, at the same multiples of the optimum. Why is that design stronger than reporting the new recipe’s spike count alone, and what can it still not tell you?
Anything about the proxy that makes it more or less stable than the full model affects both recipes, so the comparison between them is fairer than either count alone. It can’t tell you that the margin holds at full size, where sensitivity to the learning rate is larger, or anything about failures that don’t come from the learning rate, such as precision or data bugs.
In the map
Methods
- Causal next-token prediction 2 sources
- Two-stage pre-training 1 source
- Warmup-Stable-Decay 4 sources
- Cosine decay 4 sources
- Warmup-Stable-Merge 1 source
- Batch-size warmup 4 sources
- No batch-size warmup 2 sources
- Hyperparameter scaling-law fitting 3 sources
- WSD-specific scaling law 1 source
- Factorized hyperparameter transfer 1 source
- Checkpoint averaging 1 source
- Exponential moving average (EMA) 2 sources
- Global-norm gradient clipping 6 sources
- Training stability stress testing 1 source
- Distillation from a Larger Nemotron 3 Model 2 sources
Taxonomy nodes
- language modelling objective 15 methods
- learning-rate schedule 16 methods
- training stability 23 methods
- training runtime 64 methods
Further reading
- Hu et al., MiniCPM: the paper that named WSD, with the loss drop in decay and reuse of stable checkpoints on small models.
- Hägele et al., Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations: WSD against cosine up to 1B, the cooldown, and averaging against a cooldown.
- Schaipp et al., The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training: why the best peak rate differs by schedule.
- Tian et al., WSM: the merge-as-decay identity and merge weights for each decay shape.
- Maskey et al., Good Pretraining, Bad SFT: Aleph Alpha’s constant-rate, cooldown and merge comparison through fine-tuning, with its limits stated.
- Hoffmann et al., Training Compute-Optimal Large Language Models, and Besiroglu et al.’s replication: the loss law, the 20 tokens per parameter, and why the fitted law’s constants first gave a different ratio.
- Sardana et al., Beyond Chinchilla-Optimal: the budget when serving costs count too.
- Bjorck et al., Scaling Optimal LR Across Token Horizons: how the best learning rate falls with run length.
- McCandlish et al., An Empirical Model of Large-Batch Training: the gradient noise scale and the steps-against-tokens trade. Smith et al., Don’t Decay the Learning Rate, Increase the Batch Size: batch growth as a substitute for decay.
- Poolside’s Laguna report §3.1 and Appendix A.1: a full hyperparameter fit, published with its sweeps and checks.
- NVIDIA’s Nemotron 3 Ultra report §2.7: the two divergences in detail.
