Open Models Almanac

Training stages · S1

Pre-training: one long run, chosen on small ones

A run that can't be rerun is set up on proxies, and its learning-rate schedule decides whether the end is fixed in advance or left open

Illustration of an unfinished ink-blue rope bridge reaching out from a cliff across a wide misty gap, with a teal plumb line hanging from its free end. On a wooden workbench in the foreground stand small model bridges of different lengths, one of them coral, beside a coiled gold measuring tape.

Mistral pre-trained Large 4 on 3,840 GB200 GPUs. It took 45 days, “or about 4.15 million GPU-hours”. NVIDIA’s Nemotron 3 Ultra diverged twice: its training loss rose instead of falling, and kept rising. After the second time, around 16T tokens, NVIDIA went back to the checkpoint (a saved copy of the weights and optimiser state) from 15T, started lowering the learning rate at once, and “eventually made the practical decision to cut the total pretraining token horizon down to 20T tokens”. A lab can repair a run of this size from a checkpoint, but it can’t start it again from scratch.

The loss is next-token cross-entropy. What a lab has to settle before the run starts is the number of tokens, the batch size, and how and when the learning rate rises and falls. It can’t try these at full size, so it tries them on proxies, smaller models trained on smaller budgets, and on laws fitted to the proxies’ results. This article goes through those choices as the labs report them, and what each costs when it transfers badly. The learning-rate schedule has the clearest trade-off of them.

Training recipes aren’t in the configs or modeling code we check, so everything here comes from documents: the labs’ reports first, then cards and blogs, then method papers. A few reports outside the site’s collection of lab documents are labelled “outside our corpus” where used: NVIDIA’s Nemotron 3 Super report, whose recipe Ultra inherits, its Nemotron 3 Nano report, and the reports for Moonshot’s K2 and Z.ai’s GLM-4.5. A blank in a table means we searched and didn’t find it. Tokens are pre-training tokens seen, repeats included, unless a sentence says otherwise. The primer introduces the stages.

What a run fixes in advance

The objective is next-token prediction. For a training sequence x1,…,xTx_1, \dots, x_T the model minimises

L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t),\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}),

averaged over the sequences in a batch. Six of the eleven labs train a multi-token prediction head, an extra output that predicts tokens further ahead, as a second loss; NVIDIA, DeepSeek and Xiaomi state its weight, between 0.1 and 0.3 (P1). Liquid’s LFM2 adds distillation, training the model to also match a larger teacher model’s output distribution. NVIDIA’s Nemotron 3.5 Lightning goes furthest: it starts from the existing Nano 3 base and trains another 1T tokens on a loss that is 0.9 KL divergence to a larger teacher and 0.1 cross-entropy (distillation from a larger Nemotron 3, S6), so its “pre-training” never started from random weights.

Around the objective sit the settings this article is about. A step is one update of the weights. The batch is how many tokens go into one step’s gradient. The horizon is how many tokens the whole run sees. The learning rate η\eta scales each step. It rises from near zero over a warmup, because large steps on a random network are unstable, and its path after that is the schedule. When the rate is lowered at the end, the labs call that stretch the decay or cooldown; the two words mean the same thing. A run saves a checkpoint every few hundred to few thousand steps. Later stages, mid-training on a smaller, higher-quality mix and context extension on longer sequences, continue from a checkpoint (S2).

Figure 1.Nemotron 3 Ultra's pre-training on one axis. The decay runs on the second, higher-quality data phase, and both started at 15T, the checkpoint NVIDIA rolled back to after the second divergence.

The worked example in this article is Aleph Alpha’s Kolibri, a mixture-of-experts model with 78.1B parameters, of which 3.46B are active per token. Its report is the most detailed public account of a pre-training run we have found. Each step takes 4,608 sequences of 16,384 tokens, 75.5M tokens, and 264,750 steps make 20T. A common estimate of training compute is C≈6NDC \approx 6ND floating-point operations, with NN the parameters used per token and DD the tokens: two per parameter for the forward pass and four for the backward pass. It leaves out attention over the context, which isn’t small at 16K tokens. For Kolibri it gives 6×3.46B×20T≈4.2×10236 \times 3.46\text{B} \times 20\text{T} \approx 4.2 \times 10^{23} (computed), and the run took 768 B200 GPUs for 21 days.

How many tokens

The best-known rule for a budget comes from DeepMind’s Chinchilla study. For a fixed amount of training compute, the loss is lowest when parameters and tokens grow together, at about 20 tokens per parameter; Chinchilla itself was a 70B dense model trained on 1.4T tokens. (One of the study’s three methods gave a higher ratio, but Besiroglu et al.’s replication found that fit flawed and their refit gives about 20.) Among the models with a stated count, DeepSeek’s V4-Pro comes closest, at about 21 tokens per total parameter, and every other run is above it, from about 29 (MiMo-V2.6-Pro) up.

pre-training tokens, trillions (log scale)1015203040601101001,000total parameters, billions (log scale)20 per param200 per param2,000 per paramV4-ProV4-FlashV4.1GLM-5MiMo ProMiMo FlashInklingUltraSuperNanoM.1XS.2KolibriLFM3DeepSeek V4-Pro: 33T tokens (includes the extension to 1M context), 1,600B total parameters, 20.6 tokens per total parameter. In the map: yes.DeepSeek V4-Flash: 32T tokens (includes the extension to 1M context), 284B total parameters, 113 tokens per total parameter. In the map: yes.DeepSeek V4.1-Flash: 45T tokens (multimodal, includes the extension to 1M context; backbone only, 196B of Engram tables excluded; 8B active at prefill), 552B total parameters, 81.5 tokens per total parameter. In the map: yes.GLM-5: 28.5T tokens (27T pre-training + 1.55T mid-training), 744B total parameters, 38.3 tokens per total parameter. In the map: no.MiMo-V2.6-Pro: 30T tokens (27T text + 3T omni stage; an unsized mid-training stage excluded), 1,020B total parameters, 29.4 tokens per total parameter. In the map: yes.MiMo-V2.6-Flash: 48T tokens (26T text + 22T omni stage; an unsized mid-training stage excluded), 310B total parameters, 155 tokens per total parameter. In the map: yes.Inkling: 45T tokens (text, images, audio and video; stages not stated), 975B total parameters, 46.2 tokens per total parameter. In the map: yes.Nemotron 3 Ultra: 20T tokens (pre-training text; long-context phase counted separately), 550B total parameters, 36.4 tokens per total parameter. In the map: yes.Nemotron 3 Super: 25T tokens (pre-training text; long-context phase counted separately (report outside our corpus)), 121B total parameters, 207 tokens per total parameter. In the map: no.Nemotron 3 Nano: 25T tokens (pre-training text; long-context phase counted separately (report outside our corpus)), 30B total parameters, 833 tokens per total parameter. In the map: no.Laguna M.1: 30T tokens ("more than 30T" seen; whether context extension is included isn't stated), 226B total parameters, 133 tokens per total parameter. In the map: no.Laguna XS.2: 30T tokens ("more than 30T" seen; whether context extension is included isn't stated), 33B total parameters, 898 tokens per total parameter. In the map: no.Kolibri: 20T tokens (pre-training text; mid-training and context extension counted separately), 78B total parameters, 256 tokens per total parameter. In the map: yes.LFM3-1.7B: 22T tokens (18.5T pre-training + 3.5T mid-training), 1.7B total parameters, 12,941 tokens per total parameter. In the map: yes.

pre-training text onlyincludes a later stageincludes image, audio or video tokensstages not statedboth (Inkling)dashed lines: tokens per parameter

V4 and V4.1 are DeepSeek's, MiMo is MiMo-V2.6, Ultra, Super and Nano are Nemotron 3, M.1 and XS.2 are Laguna. Lighter: earlier or sibling models, not in the map (GLM-5, Nemotron 3 Super, Nemotron 3 Nano, Laguna M.1, Laguna XS.2). Current models with no stated count, not drawn: Mistral Large 4, Kimi K3, Qwen3.8.

Hover over or focus a dot for the model, its count and the ratio.

Figure 2.Pre-training tokens against model size, for models with a stated count from the labs in the map (lighter dots are earlier or sibling models). Only DeepSeek V4-Pro sits near Chinchilla's 20 tokens per parameter. The mark shows what each count includes, so not every dot measures the same kind of token.

Ultra sits at 36 tokens per total parameter, Kolibri at 256, Poolside’s Laguna XS.2 at about 900, and Liquid’s LFM3-1.7B at about 13,000 (computed). Chinchilla’s rule minimises training compute only. A model that serves many requests also costs compute for every token it generates, so a smaller model trained longer can be cheaper overall, as Sardana et al. work out; none of the reports in the map gives that reason. Mistral (for Large 4) and Moonshot (for K3) treat the ratio as a setting to tune, but give no value.

A token count also has to say what it includes. GLM-5’s 28.5T includes mid-training, and LFM3’s 22T includes 3.5T of it. Xiaomi’s 1.02T-parameter MiMo-V2.6-Pro saw 30T tokens against 48T for the 310B Flash, but in the text stage Pro saw 27T and Flash 26T; the gap is a stage of image, audio and video tokens, which don’t carry what a text token carries. For mixture-of-experts models, “per parameter” also needs a denominator, since Chinchilla was fitted on dense models. Sequences start at 4K to 64K tokens, and long contexts come later (S2). NVIDIA and Mistral split pre-training into two data phases (two-stage pre-training, D2).

Choosing settings on proxies

A proxy is cheap only if what’s learned on it carries over. The usual tool is a power law. Chinchilla fitted the loss of dense models as

L(N,D)=E+ANNα+ADDβ,L(N, D) = E + \frac{A_N}{N^{\alpha}} + \frac{A_D}{D^{\beta}},

with fitted constants E,AN,AD,α,βE, A_N, A_D, \alpha, \beta. EE is the loss no model can beat, and the other two terms shrink as the model grows and as it trains longer. The same idea applies to a hyperparameter: train small models at several learning rates, find the best rate for each size and budget, fit a law to those optima, and read off the value for the real run.

Poolside’s report shows every step of one such fit (a WSD-specific scaling law). It swept four model sizes, from 0.3B to 2.2B active parameters, six learning rates and five budgets from 30B to 480B tokens, all at a batch of 8M tokens. The fitted law for the best peak learning rate is

η∗(N,D)=104.488 N−0.4639 D−0.2661,\eta^*(N, D) = 10^{4.488}\, N^{-0.4639}\, D^{-0.2661},

with NN the active parameters and DD the tokens, both as raw counts. For Laguna XS.2, with 3.0B active parameters and 30T tokens, the law gives 3.2×10−43.2 \times 10^{-4}. XS.2 trained at a batch of 24M tokens, three times the fit’s, and Poolside scaled the rate by the square root of that ratio, the usual rule for Adam-type optimisers: 3.2×10−4×3≈5.5×10−43.2 \times 10^{-4} \times \sqrt{3} \approx 5.5 \times 10^{-4} (computed; it matches the report’s “∼5.5×10⁻⁴” only with the square root). Poolside used 5×10−45 \times 10^{-4}, “to leave a small safety margin”. (The map’s Laguna S 2.1 is a later scale-up of the same family, and its own settings aren’t published.)

A margin on the low side is the right direction: near the best rate the loss changes slowly, while a rate too high risks divergence. A companion fit in Poolside’s appendix misses the mean best rate of each proxy size by 0.91× to 1.14×, and a cross-check on Moonshot’s K2 predicts 1.75 times the rate K2 used, which Poolside calls “suggestive rather than a validation”. A least-squares line is best determined at the centre of its data, here 120B tokens if the five budgets double from 30B to 480B as the main text says (the appendix lists 60B to 960B), so an error of 0.02 in the exponent of DD moves the prediction at 30T by 12% (computed): more than the margin. Poolside gives no standard errors; from its reported R2R^2 of 0.89 we estimate one at about 0.04, which would move the prediction by about 25%.

N = 3.0B active parameters, batch 24M tokens (√3 times the law at its fitted 8M). The fan pivots at 120B, the geometric mean of the fitted budgets. Poolside reports no error for the exponent; the marks on the slider are 0.04, ≈ one standard error (our estimate from the reported R²), and 0.054, Bjorck et al.'s exponent.

At 30T the law gives 5.54×10⁻⁴; Poolside used 5.0×10⁻⁴, 0.90 of it. With the D exponent off by 0.020, the prediction would be 4.96×10⁻⁴ (+Δ) or 6.19×10⁻⁴ (−Δ), a factor of 1.117 either way.

Figure 3.Poolside's fitted learning-rate law for Laguna XS.2. The value used at 30T lies 62 times beyond the largest budget in the fit. An exponent error of 0.02, about half a standard error by our estimate, moves the prediction by 12%; using Bjorck et al.'s exponent instead would move it by 35%.

The horizon has its own transfer problem. Bjorck et al. found that the best learning rate falls as a run gets longer, roughly as D−0.32D^{-0.32}, which matches the direction of stochastic gradient descent theory (for a convex loss with noisy gradients, the best constant step over KK steps falls as 1/K1/\sqrt K). Their rule was fitted below 800B tokens, so using it at 20T is an extrapolation too.

Mistral predicts its peak rate with a log-linear function of width, depth, horizon, expert ratio and batch, checked on a scaling ladder of proxies that “differs from the reference model in scale only”, and gives no sizes or errors. Qwen refitted its law and checked it on a 156B-parameter proxy: the old recipe ended 0.0078 higher in loss than the new law’s settings. Only Poolside publishes its law with values.

Aleph Alpha dropped laws for the best hyperparameters for Kolibri, because laws fitted for its earlier Kolibri Origin gave “unrealistic hyperparameter predictions” when extended. It split the transfer in two (factorised transfer). For width it used µP, a parametrisation meant to keep the best rate fixed as a model gets wider; on proxies 1,024 and 2,048 wide the best rate moved by 2\sqrt 2, “which we deem acceptable”. For the horizon it trained a half-width proxy to 500B tokens at several settings, extrapolated each setting’s loss curve 40 times further to 20T, and kept the setting with the lowest forecast. That is still a power law, but fitted on the curve each setting actually traced. The report gives “an average prediction error at the target token horizon below 1 %”, on curves “not reported here”, while the settings it ranks differ by about as much (the larger batch was forecast “0.9 % behind”). The best learning rate, the batch and the weight decay all sat at the edge of the grids tried.

Some problems never show up in a proxy’s loss. Mistral computes its load-balancing loss per group of GPUs that share experts, so “downscaling experiments must be careful when changing partitioning”. A cache-saving attention design also looked fine on the proxies’ perplexity, but when Mistral fine-tuned intermediate checkpoints of the real run on example answers (SFT), these “early SFT cooldowns on intermediate checkpoints revealed poor performance” on agentic, multi-step tasks. It changed the architecture mid-run and trained “approximately two-thirds of the original phase 2”.

The learning-rate schedule

A high learning rate makes fast progress but leaves the weights moving around the bottom of a valley in the loss. For plain stochastic gradient descent near a minimum, the extra loss this costs is about η4Btr⁡(Σ)\frac{\eta}{4B}\operatorname{tr}(\Sigma), with BB the batch and Σ\Sigma the covariance of one example’s gradient (a standard result for a locally quadratic loss). Lowering the rate lowers that floor, and the loss drops quickly when it does. But the total area under the learning-rate curve drives progress (Tissue et al. fit the loss on it, and in the classical error bound for stochastic gradient descent the sum of the rates is the progress term), so decaying early gives up progress for the rest of the run. Choosing a schedule means choosing how long to stay high and how to come down.

A cosine schedule lowers the rate along half a cosine, from the end of warmup (s=0s = 0) over the KdecK_{\text{dec}} steps that remain:

η(s)=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡πsKdec).\eta(s) = \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{\pi s}{K_{\text{dec}}}\right).

It has no knob beyond the peak and the floor, but it needs KdecK_{\text{dec}} before the run starts. The Chinchilla authors found that a cosine set more than 25% longer than the run, so that the run stops before the rate has come down, “noticeably degraded” the result.

Warmup-stable-decay (WSD), named in the MiniCPM paper, holds the peak rate and decays only over a final stretch. The decay can start from any checkpoint in the stable phase, so one long stable run can be finished several ways: to compare data mixes, or to produce a model at a shorter horizon. The cost is three more settings: the decay’s length, its shape and its floor. A common shape is 1−x1 - \sqrt{x} over the decay fraction xx, which NVIDIA calls minus-sqrt. It falls to half the peak a quarter of the way through the decay, where a cosine-shaped decay over the same stretch is still at 85% (computed).

Ultra (Nemotron 3 Ultra): WSD, minus-sqrt; decay start set by a rollbackXS.2 (Laguna XS.2): WSD, 1−√ (drawn: warmup 1%, assumed)V4.1 (DeepSeek V4.1): no name: hold, cosine, holdK3 (Kimi K3): cosine; floor not stated; a cooldown stage follows (drawn: floor 10%, assumed)GLM-5: cosine to 20% (drawn: warmup 1%, assumed)Kolibri: constant, then merge

Drawn but not stated: Laguna XS.2 warmup 1%; Kimi K3 floor 10%; GLM-5 warmup 1%. Dashed: the same schedule finished at the slider's point. Highlighted lines are in pre-training's colour, the rest grey.

Laguna XS.2: decay from 42% to 60% of the full run, the same shape and share. Kimi K3: at 60% the cosine is still at 42% of peak; a cosine has to be restarted or decayed off-schedule.

Figure 4.Learning-rate schedules as the labs describe them, on a common time axis. The name a lab uses doesn't predict the shape: DeepSeek gives no name and holds the rate, then decays and holds again. Moving the slider finishes a run early, which a stable phase allows and a cosine doesn't.

The labels don’t predict the shapes. Ultra’s decay started at 15T because of the rollback, not by plan; Poolside’s larger Laguna M.1 used cosine where XS.2 used WSD; Moonshot’s K3 uses cosine where its K2 (outside our corpus) used WSD.

The one per-schedule comparison a lab has published is Moonshot’s. Its scaling-law study, on proxies whose sizes it doesn’t give, ran “an independent scaling-law search for each schedule” and found: “Under their respective optimal hyperparameter settings, cosine decay consistently achieves a lower final loss than WSD.” Both decayed to the same fixed floor, and the report gives no budgets or size of the gap. Z.ai’s GLM-4.5 report (outside our corpus) also chose cosine, after WSD did worse on two benchmarks, without saying whether each schedule was tuned.

Poolside chose WSD for checkpoint reuse, and states the cost: “WSD is attractive because a single stable-phase checkpoint can be reused across data-mix iterations via shorter cooldown runs. But, according to our ablations, final-loss calibration is less direct than for one-shot cosine, and naively tuned WSD runs sometimes underperform tuned cosine at matched compute.”

Method papers at published scales disagree with Moonshot. Hägele et al., on models from 124M to 1B parameters (and one short 8B run), found that WSD with a 1−x1-\sqrt{x} cooldown matches a well-tuned cosine, and Tissue et al. found WSD lower. Schaipp et al. show one reason such comparisons are fragile: in a convex-optimisation bound and in their experiments, the best peak rate for cosine is about twice that for WSD, so a comparison at one shared peak rate favours whichever schedule that rate suits. Moonshot’s per-schedule search answers that objection. No study at full scale tunes both schedules and reports the gap.

At most labs the decay is also where the data changes. Ultra’s decay and its higher-quality second data phase both cover the last 5T; GLM-5, MiMo-V2-Flash and Liquid continue the decay into mid-training, and K3 extends its context during its “cooldown phase”. Kolibri’s quality-heavy mid-training runs at the constant peak instead. None of the comparisons above says it held the data fixed, so a gain credited to a decay may partly belong to the data (D2).

Merging in place of decay

There is a third option: never decay, and average checkpoints instead. Averaging the iterates of stochastic gradient descent is an old idea (Polyak–Ruppert averaging, and later stochastic weight averaging). The WSM paper by Tian et al. ties it to decay.

Take a base checkpoint θn\theta_n and the next kk saved checkpoints, and write Δi=θn+i−θn+i−1\Delta_i = \theta_{n+i} - \theta_{n+i-1} for everything the optimiser changed between two of them, often hundreds of steps. Each checkpoint is the base plus the changes so far, θn+j=θn+∑i≤jΔi\theta_{n+j} = \theta_n + \sum_{i \le j} \Delta_i. A merge with weights c0,…,ckc_0, \dots, c_k summing to 1 is then, after swapping the two sums,

θ^=∑j=0kcj θn+j=θn+∑i=1kwi Δi,wi=∑j=ikcj.\hat\theta = \sum_{j=0}^{k} c_j\, \theta_{n+j} = \theta_n + \sum_{i=1}^{k} w_i\, \Delta_i, \qquad w_i = \sum_{j=i}^{k} c_j .

This identity holds for any run. It says the merge keeps a fraction wiw_i of the ii-th block of changes. Since each change is a learning rate times a direction, keeping a fraction wiw_i of it is like having taken that block at wiw_i times the rate: a decay schedule, applied after the fact, constant within each block. The weights start at w1=1−c0w_1 = 1 - c_0 and fall when every cj≥0c_j \ge 0. Every non-increasing decay between 0 and 1 has merge weights that produce it (ck=wkc_k = w_k, cj=wj−wj+1c_j = w_j - w_{j+1}, c0=1−w1c_0 = 1 - w_1). A plain average of the kk checkpoints after the base, with none on the base, gives w=1,k−1k,…,1kw = 1, \frac{k-1}{k}, \dots, \frac1k: a linear decay (computed).

Exact for the updates the run made. A run trained with this decay would have made different updates. The EMA weighs the ten checkpoints after the base by β = 0.8 per checkpoint (for illustration), normalised so the base gets none.

linear: the base checkpoint gets c₀ = 0.000; the 10 after it get 0.100 each, a plain average; the weights sum to 1.000.

Figure 5.Merge weights and the decay they amount to, over ten checkpoints. A plain average is a linear decay. The identity is exact for the updates the run made at its constant rate; a run that had actually decayed would have made different ones.

What isn’t exact is the step to “the same as training with that decay”: a run at the lower rates would have computed different changes, from different weights. In Maskey et al.’s words, a merge “is not equivalent to cosine cooldown because the optimizer never follows the decayed trajectory.” Classical results on averaging say what it can do. Near a minimum, a constant-rate run keeps moving around it, and an average over a long enough window removes most of that movement. But on a loss that isn’t quadratic, the points the run visits are centred off the minimum by an amount proportional to the learning rate (Dieuleveut et al.), and only a lower rate removes that offset. Hägele et al. found, at 210M parameters, that averaging narrows the gap to a cooldown without closing it.

Labs use merges for different jobs. NVIDIA scored merges over a sliding 500B-token window throughout Ultra’s pre-training, “weighted to emulate our learning rate decay schedule”, as a cheap preview of a decay; its Super report (outside our corpus) puts the best merge 2 to 4 points above the raw stable-phase checkpoint on a 12-benchmark average, and estimates that the dedicated decay runs it avoids could cost “roughly 16% of the total pretraining FLOP budget”. After Ultra’s decay, NVIDIA also started long-context training from a 500B-window merge. Laguna XS.2 releases an exponential moving average of its 10 most recent checkpoints, also after a decay. WSM found that merging after a decay adds little, so only Kolibri’s merge replaces one.

Kolibri never decays. It holds the rate constant through pre-training, mid-training and context extension, and its base model is the plain average of the last 20 checkpoints, 10B tokens apart (Warmup-Stable-Merge). The reason given: “In our experiments, a constant learning rate with checkpoint merging improves post-trained performance over a learning-rate cooldown (Maskey et al. 2026).”

That citation is Aleph Alpha’s own study, on one 30B-parameter model with one seed per setup, comparing a constant rate, a cooldown to 10% of peak over the last 800B tokens, and a merge of the constant run. Before fine-tuning, the cooldown beat the constant run on every signal (validation loss 1.648 against 1.734), but the merge scored highest on the benchmark aggregate (0.460, against 0.440 and 0.415). Mid-training then warmed all three back up to the full peak rate, so the cooldown run left its low rate after one stage. After mid-training, context extension and fine-tuning, the aggregate was 0.363 for the merge and 0.360 for the constant run against 0.247 for the cooldown, a gap that shrinks to 0.016 without HumanEval+, where the cooldown model failed to stop generating. The cooldown’s mid-training and long-context schedule was the best of nine, and the other two runs reused it once, which tilts the test toward the cooldown; even the best of 27 cooldown runs, counting an SFT-rate sweep, reached only 0.294. Yano et al., at 1B and 8B, independently report that skipping the decay helps after fine-tuning.

NVIDIA’s Super report tested merging against decay at the scale of Nemotron 3 Nano and kept its decay: it reproduced WSM’s gains “when emulating short (∼500B) decay windows. However, direct comparisons at 1T and 1.5T merge horizons showed no improvement over decay-trained checkpoints.” NVIDIA scored base models and Aleph Alpha scored after fine-tuning; no study has done both. Merging’s own costs are checkpoints to store, a window and weights to set, and a released base that was never decayed.

Batch size

A larger batch averages more examples (here, tokens) into each gradient, so each step points closer to the true gradient ∇L\nabla L. Past some size, the extra examples stop helping. McCandlish et al. measure this with a gradient noise scale, in its simple form

Bsimple=tr⁡(Σ)∥∇L∥2,B_{\text{simple}} = \frac{\operatorname{tr}(\Sigma)}{\lVert \nabla L \rVert^2},

the total variance of the per-example gradients over the squared norm of the mean gradient; their full version weights both by the curvature, and they find the two differ by a roughly constant factor. Well below it, doubling the batch nearly halves the steps; well above it, the extra examples are wasted. Their model gives a hyperbola between the steps KK and the tokens DD needed to reach a given loss:

(KKmin⁡−1)(DDmin⁡−1)=1.\left(\frac{K}{K_{\min}} - 1\right)\left(\frac{D}{D_{\min}} - 1\right) = 1 .

Since D=BKD = BK, and the critical batch size is Bcrit=Dmin⁡/Kmin⁡B_{\text{crit}} = D_{\min}/K_{\min}, this gives K/Kmin⁡=1+Bcrit/BK/K_{\min} = 1 + B_{\text{crit}}/B and D/Dmin⁡=1+B/BcritD/D_{\min} = 1 + B/B_{\text{crit}}. At B=BcritB = B_{\text{crit}} a run takes twice the minimum steps and twice the minimum tokens.

"Later in the run" multiplies B_crit by 4, a factor chosen for illustration; McCandlish et al. report growth of "at least an order of magnitude" over training on their tasks.

B = 1 × B_crit: K/K_min = 2.00, D/D_min = 2.00.

Figure 6.The trade between steps and tokens at a fixed loss, in McCandlish et al.'s model. At the critical batch size a run needs twice the minimum of each. The same batch moves toward fewer wasted tokens as the critical size grows, which is the argument for a ramp.

The noise scale grows as training goes on, because the mean gradient shrinks while the per-example noise doesn’t: by “at least an order of magnitude” in McCandlish et al.’s runs, which were on image, game and small language tasks. That argues for a batch-size ramp, starting small and growing, though McCandlish et al. call the expected gain “relatively modest unless the B_crit changes greatly during training”. A ramp is also part of the schedule: in the floor formula above η/B\eta/B is what matters, so a fourfold larger batch lowers the floor about as much as a fourfold decay in plain stochastic gradient descent (Smith et al.), or, by the square-root rule’s heuristic for Adam, a twofold one.

Mistral gives the clearest case for a ramp: “Early in training, a larger batch size wastes compute (and tokens) because a gradient estimate from a fraction of the batch’s tokens suffices, whereas later in training, we benefit from more precise gradients.” Large 4 ramps from 40M to 80M to 160M (tokens, by our reading), spends the last half of the run at the largest, and retunes the learning rate at each stage. A ramp from 0.25× to 2× of its predicted batch worked best for optimisation, but Mistral chose 1× to 4× because “larger batch sizes achieve higher MFU in our setting”, MFU being the share of the hardware’s peak arithmetic a run uses.

DeepSeek V4, MiMo-V2-Flash and GLM-5 ramp too. Qwen tested a ramp for Qwen3.8 on a 10.8B-parameter proxy over 4T tokens: “Neither variant improves performance.” The ramps ended 0.00025 and 0.00035 higher in loss, “within the run-to-run variance”, which Qwen doesn’t size, and took 18.8% more optimiser steps (training without batch-size warmup). DeepSeek V4.1 keeps its batch “fixed at 100.6 million tokens throughout training” without saying why, and NVIDIA, Poolside and Aleph Alpha don’t discuss ramps.

The choice doesn’t follow the optimizer (O1). Mistral and MiMo ramp with AdamW and NVIDIA doesn’t; DeepSeek V4 and GLM-5 ramp with Muon, and DeepSeek V4.1 and Qwen don’t.

When a run goes wrong

Four of the eleven labs describe a production pre-training run going wrong: NVIDIA, Poolside, DeepSeek and Aleph Alpha. A few terms first. Most runs keep weights and activations in BF16, a 16-bit format with FP32’s range but far fewer digits, and Ultra trained most layers in the 4-bit NVFP4, checking it against BF16 branches started from its 5T, 10T and 16T checkpoints (O2). An all-reduce sums a tensor across GPUs; tensor parallelism splits one layer’s matrices over several GPUs, so their partial results have to be summed.

RunWhenWhat happenedStated causeFix
Nemotron 3 Ultra~8Tloss and weight-gradient norm rise; a multi-token head’s loss spikes firstthe gradient from the multi-token heads into the shared output layer was lost when its accumulation moved “from FP32 to BF16”roll back, reduce gradients in FP32
Nemotron 3 Ultra~16Tloss and weight-gradient norm rise“we could not find a smoking gun”roll back to 15T, decay at once, end at 20T
Laguna M.1not statedlogits drift upwardno z-loss to hold logits down, so the BF16 all-reduce of the output layer’s input gradients became “the dominant source of numerical error”do that reduction in FP32
DeepSeek V4not statedrecurring loss spikes“consistently tied to outliers in the MoE layers”, with routing that “appears to exacerbate” themAnticipatory Routing after a short rollback; SwiGLU activations clamped
Kolibri Originafter “a few trillion tokens”not stated“a data-shuffling bug that escaped our tests”restart from scratch

Two of the four diagnosed incidents trace to the precision of a sum at the output layer, by different paths (our grouping). In Ultra’s case the lost signal came from small auxiliary losses (weight 0.05 per head), and that head’s loss spiked first. DeepSeek’s fix, Anticipatory Routing, computes expert assignments with slightly older weights, switches on after a spike, and costs about 20% in wall-clock time while it runs; its mechanism “remains an open question”. Mistral’s mid-run architecture change was a capability problem rather than a failing loss.

The commonest failure is mundane. Kolibri had 38 unplanned interruptions in 21 days, each restarted from a checkpoint at most 250 steps back, up to about 19B tokens recomputed (computed), and Mistral reports 94.1% of its time spent on useful training. Checkpoint spacing sets both the cost of a rollback and which merges are possible. Kolibri Origin’s restart came from the data pipeline, which no loss curve flagged.

Other labs report clean runs: DeepSeek V4.1 “with no instability”, Qwen3.8 “without a single loss spike”, Laguna XS.2 “without encountering further stability issues”. A clean run is weak evidence by itself, since there is no matched run without the recipe and reports describe the runs that shipped. The routine guards are rarely stated. Gradient clipping, rescaling any gradient whose norm exceeds a threshold, is given at 1.0 by Kolibri and MiMo-V2-Flash. Weight decay is 0.1 at 4 of the 5 labs that state a pre-training value; Aleph Alpha’s 2−122^{-12} looks 400 times smaller, but it applies per step regardless of the learning rate, where AdamW’s 0.1 is usually multiplied by the rate, which makes Kolibri’s the stronger at Ultra’s peak rate (computed; NVIDIA doesn’t state its convention).

Qwen is the one lab that reports a stability stress test (stability stress testing). Following Wortsman et al.’s observation that instabilities of large runs reappear in small ones at high learning rates, it holds a 25B-parameter proxy’s rate at two and four times its optimum, with no decay, and counts loss spikes. The criterion is “that the new recipe must be at least as stable as the generation it replaces under equal stress”.

Figure 7.Qwen's stress test: loss spikes per 10,000 steps when a 25B-parameter proxy is held at two or four times its best learning rate, and a separate test of the gate alone at three times. The new recipe stays near zero where the old one spikes. Qwen credits the combination of optimizer and structure.

At twice the optimum, the old structure with AdamW spiked 4.3 times per 10,000 steps and both Muon runs 0.2. At four times, AdamW spiked 183 times and Muon with the gated residual not at all. Qwen credits “the new architecture and optimizer combination” without ranking the two. Because both recipes run on the same proxy, whatever the proxy gets wrong for both cancels. Nothing in the test shows that the margin survives at full size, where Wortsman et al. found sensitivity to the learning rate is larger, and none of the incidents in the table had a learning rate as its stated cause. If the Muon run was as long as the AdamW run’s 19,932 steps, zero spikes bounds its rate below about 1.5 per 10,000 at 95% confidence, not at zero (computed).

The choices side by side

ChoiceWhat it decidesWhat it buysWhat it costsEvidence
Fitted law for hyperparameterspeak rate, batchvalues for a run no one can tryextrapolation far past the fit; a safety margin chosen by handonly Poolside publishes its law with values; its reported errors exceed its margin
Factorised transfer (µP + loss forecast)rate and batch at width, then at horizonno law for optimal hyperparameters across sizesstill a 40× loss forecast; settings at grid edgesKolibri: under 1% forecast error on unreported curves; ranked gaps about 1%
Cosinewhen the rate fallslower final loss in Moonshot’s tuned proxy comparisonlength fixed in advanceMoonshot, Z.ai (benchmarks); Hägele, Tissue find parity or the reverse
WSDwhen the rate fallsstable checkpoints that can be decayed several waysdecay length, shape and floor to setPoolside, NVIDIA use it; decay usually coincides with a data change
Constant rate + mergewhether the rate falls at allno decay; Aleph Alpha reports better scores after fine-tuningwindow and weights to set; a base never decayedAleph Alpha (one seed) for; NVIDIA finds no gain at 1T to 1.5T windows
Batch ramptokens per step over timefewer wasted tokens early; acts partly as a decaymore steps; lower MFU while the batch is smallMistral, DeepSeek V4, MiMo, GLM-5 ramp; Qwen finds no gain on a proxy
Stress testwhether a recipe is stable enougha failure seen on a proxy, before the runproxy compute; tests learning-rate instability onlyQwen only

Where the models in the map sit

LabModelTokens (kind)Starting lengthScheduleBatch (tokens)MergingIncident reported
Aleph AlphaKolibri20T pre16Kconstant after 100B warmupfixed 75.5Mreleased base: mean of last 20Origin restarted
MistralLarge 44Kramp 40M → 160Mregression found mid-run
NVIDIANemotron 3 Ultra20T text8KWSD, minus-sqrt over last 25%fixed 25.2Mevaluation, selectiontwo divergences
PoolsideLaguna S 2.1 (recipe from the XS.2 and M.1 report)XS.2: >30TXS.2: 4KXS.2: WSD, 1−√ over last 30% (M.1 cosine)XS.2: fixed 24MXS.2: EMA of 10M.1: logit drift
MoonshotKimi K38Kcosine
DeepSeekV4.1-Flash (V4)45T, multimodal, incl. 1M extension64K (V4 4K)hold, cosine, holdfixed 100.6M (V4 ramp)V4: loss spikes
QwenQwen3.8-Flash-Next1/3 of Qwen3.5-397B’sfixed
Z.aiGLM-528.5T incl. mid4Kcosineramp
XiaomiMiMo-V2.6-Flash (V2-Flash)48T incl. omni, excl. mid32KV2-Flash: hold, cosineV2-Flash: ramp
LiquidLFM3-1.7B22T incl. mid
Thinking MachinesInkling45T, multimodal

Mistral never states its token count, and Moonshot and Qwen give none in absolute terms. Kolibri’s own summary table counts 24T as “Pre-training tokens”, folding in its mid-training and context extension. Learning rates in these reports don’t compare directly across optimizers and conventions (O1), and GPU-hours compare only on the same hardware and scope.

Easy to get wrong

  • “20T tokens on a 78B model is overtrained.” Chinchilla’s 20 tokens per parameter minimises training compute for one loss, on dense models. It leaves out the cost of serving, and for a mixture-of-experts model it doesn’t say which parameters to count.
  • “Cosine beats WSD.” One lab found it on proxies, after tuning each schedule separately. Published studies find parity or the reverse, and at most labs the decay also brings new data.
  • “Merging beats decay.” Aleph Alpha finds it after fine-tuning, in a one-seed study whose later stages re-warm the cooldown run to its peak, with most of the gap from one benchmark failure. NVIDIA finds no base-model gain from merging at 1T to 1.5T decay windows.
  • “Qwen showed that batch ramps hurt.” Its ramps were worse by an amount within run-to-run variance, on a proxy. They didn’t help, and they cost 18.8% more steps.
  • “The run was clean, so the recipe is stable.” Clean runs are what gets published, and none of them has a matched run to compare with.

Check yourself

Poolside’s law gives 3.2×10−43.2 \times 10^{-4} at its fitted batch of 8M tokens. What does it give at 24M tokens if the rate scales with the square root of the batch, and if it scales linearly? The report says about 5.5×10−45.5 \times 10^{-4}. Which rule did Poolside use?

3.2×10−4×3≈5.5×10−43.2 \times 10^{-4} \times \sqrt 3 \approx 5.5 \times 10^{-4}, and 3.2×10−4×3≈9.6×10−43.2 \times 10^{-4} \times 3 \approx 9.6 \times 10^{-4}. The report’s number matches the square root.

You average the five checkpoints that follow a base checkpoint, with equal weights, and give the base itself no weight. What fraction of each block of changes does the merge keep, and why isn’t the result the same as having trained with that decay?

w=1,0.8,0.6,0.4,0.2w = 1, 0.8, 0.6, 0.4, 0.2: a linear decay over the five blocks. The changes were computed at the full rate, from weights that a decayed run would never have reached, so the merged weights are built from updates a decayed run wouldn’t have made.

At four times the critical batch size, how many tokens and how many steps does a run need, relative to the minimum of each? Which of the two does a larger batch save?

D/Dmin⁡=1+4=5D/D_{\min} = 1 + 4 = 5 and K/Kmin⁡=1+1/4=1.25K/K_{\min} = 1 + 1/4 = 1.25: five times the minimum tokens and 25% more than the minimum steps. A larger batch saves steps, so wall-clock time, and spends tokens. Mistral’s choice of 1× to 4× of its predicted batch over 0.25× to 2× is that trade, made because larger batches ran at higher MFU.

Qwen’s stress test compares the new recipe with the old one on the same proxy, at the same multiples of the optimum. Why is that design stronger than reporting the new recipe’s spike count alone, and what can it still not tell you?

Anything about the proxy that makes it more or less stable than the full model affects both recipes, so the comparison between them is fairer than either count alone. It can’t tell you that the margin holds at full size, where sensitivity to the learning rate is larger, or anything about failures that don’t come from the learning rate, such as precision or data bugs.

In the map

Methods

Taxonomy nodes

Further reading

↑