Open Models Almanac

Foundations · 01

The anatomy of a training recipe

Labs agree on the rough order of stages and a few habits inside them. Each stage spends tokens on what the last one could not teach, and each can lose some of what came before

Illustration of a long workshop bench. At the left stands a huge rough block of ink-blue stone. Along the bench, smaller and more finished pieces follow, ending in a small carved teal hummingbird with a coral throat. Gold chips lie on the floor beneath every station.

Kolibri, Aleph Alpha’s 78-billion-parameter model, read 23.63T tokens (T for trillion) in three stages before it became a finished base model, and the first of those stages alone took a net 377,000 GPU hours. On a mean of 12 benchmarks covering math, knowledge, tool use, code, instruction following and long context, that base scores 30.0%. One further run of supervised fine-tuning (SFT), about 268B packed tokens, or 1.1% of what the base had read (computed), reaches 58.9%. Averaging the weights of two such runs reaches 63.4%. The winners were picked on these same benchmarks, so the report calls those scores optimistic.

So about 1% more tokens moved that mean about as far as all the tokens before them could reach. A base model only continues documents, and several of those benchmarks expect a chat answer or a tool call. We read the jump as largely that, though the report prints only two of the base’s twelve scores, both on math competition tests it can already attempt: 74.6 ± 10.7 on AIME 2026 and the same on a variant.

Modern training runs are sequences of stages for this reason. Each has its own data, and often its own loss, sequence length and learning rate, and each teaches something the previous stage could not. A report summarizes them in a phrase like pre-training, then SFT, RL and MOPD (reinforcement learning, then a way of distilling several models into one). This article explains that phrase stage by stage, then covers what can be compared across labs and which claims to check.

Why a recipe has stages

A small model usually gets one dataset, one loss and one schedule, and training stops when validation loss flattens. At this scale that plan fails. The reports don’t list the reasons together, so the grouping here is ours.

Long sequences are expensive, and most data doesn’t need them. Full attention costs the square of a document’s length (Aleph Alpha makes this point about packing SFT sequences). Moonshot trains Kimi K3 mostly at 8K to 64K tokens (K is 1,024 here) and moves to 1M near the end: “Concentrating the costly long-sequence computation within a small fraction of the overall training budget keeps the curriculum economical”.

Good data is scarce, and labs spend it late. Web text exists in the trillions of tokens, and is mixed in quality. Curated reasoning, code and instruction data is far smaller, and Kolibri caps each dataset at four epochs across pre-training and mid-training, because repeating a small set stops paying. Aleph Alpha cites Hu et al. (2024) for saving higher-quality data for later phases, and NVIDIA cites Feng et al. (2024). Feng et al., who are at NVIDIA, report that two-phase ordering beats random ordering by 3.4% in average accuracy on their data blends. None of the model reports in our corpus tests the order for its own run. The order resembles curriculum learning, but it runs from broad to targeted, not from easy to hard.

Some behaviours need earlier ones. Reinforcement learning can only improve what a model sometimes does. Aleph Alpha says SFT “establishes desired behaviours that the model must produce at a sufficient rate for RL to become effective in improving them”. The RL section below shows why that rate matters.

The model or its machinery changes. DeepSeek V3.2 continues a dense-attention checkpoint with sparse attention on the same data. Mistral modified “an existing checkpoint” before its second pre-training phase. Xiaomi switches optimizer in mid-training “to prepare the model for subsequent large-batch RL”. RL also needs a different cluster shape, with GPUs for generation and judging as well as training.

A checkpoint can branch. Poolside values a schedule because “a single stable-phase checkpoint can be reused across data-mix iterations”, and Kolibri fans out several SFT runs from one base. NVIDIA goes further: Nemotron 3.5 Super starts from Nemotron 3 Super’s base and adds about 1.2T tokens of its own (computed). Its card still says “more than 20 trillion”, which we read as counting the older base.

One recipe from end to end: Kolibri

This section follows Kolibri through every stage. It has the longest report in our corpus, rich in sizes, thresholds and failures. That makes it the most detailed public account, and several of its choices are minority positions, among them no learning-rate decay and no final distillation stage. Everything in this article comes from labs’ own documents: reports first, then model cards and blogs, because training recipes aren’t in the code and configs that the architecture articles can check.

Two words carry the structure. The first three stages produce a base model, which predicts next tokens and continues documents. Everything after it, which teaches the base to behave as an assistant, is post-training. Labs draw the line differently: Poolside files mid-training under post-training.

Tokens trained, log scale (axis starts at 10B, not zero)

Sequence length (tokens)

16K
64K
256K
256K packed
up to 256K

Loss computed on

next-token
next-token
next-token
assistant output only
policy gradient plus smaller terms

Muon learning rate

1e-3
1e-3
1e-3
5e-4 peak
1e-5
  • pre-training. Data: broad pool of 78 datasets, about 29.3T tokens. Batch 4,608 sequences, 75.5M tokens per step, 264,750 steps.
  • mid-training. Data: moved toward instruction, reasoning, STEM and code. Batch 1,536 sequences, 101M tokens per step, 34,200 steps.
  • long-context. Data: one third naturally long documents, two thirds mid-training data. Batch 768 sequences, 201M tokens per step, 1,000 steps.
  • SFT. Data: demonstrations, recorded conversations packed to length. Batch 256 sequences, 67.1M tokens per step, 4,000 steps.
  • RL. Data: the model's own rollouts in 38 environments. Batch 2,048 sequences, tokens per step not printed, 1,000 steps.
Figure 1.Kolibri's five stages, with tokens on a log axis. RL has no bar because the report prints no token total for it. The SFT bar is one of the two runs that were averaged.

Pre-training: breadth

Pre-training covers 20T tokens at a sequence length of 16,384, in batches of 4,608 sequences: 75.5M tokens per step, for 264,750 steps. The objective is ordinary next-token prediction on a pool of 78 datasets holding about 29.3T tokens. German is 21.3% of the tokens, and about 24% is synthetic in origin, much of it rephrased or translated by another model (the synthetic data article). The pool has been through a long filtering pipeline, and its mixing weights were fitted on thousands of small proxy models, smaller models trained on a smaller budget to stand in for a run that can’t be repeated (the data mixture article).

The run used 768 B200 GPUs for about 21 days (the blog’s figure), with 38 unplanned interruptions. The learning rate warms up over the first 100B tokens, then stays constant through this stage and the next two.

Mid-training: from breadth to skill

Mid-training raises the sequence length to 65,536 and moves the data toward reasoning, math, code and instruction following: “English instruction and reasoning, STEM (QA-style), and Code hold 73 % of the budget together, and the two web domains hold 3.4 %”. The mix interpolates from the pre-training mix over the first 100B tokens of the 3.44T. It resumes the optimizer state, so there is no new warmup. Aleph Alpha chose this mix on a proxy, with a penalty for straying from a hand-set mix, and without that penalty the solve zeroes out four of its nine domains.

The stage trades one kind of skill for another. Scores below come from averaged checkpoints at the end of each stage, which the next section explains. English math rises from 79.4 to 84.0, and German math from 71.4 to 75.2. TriviaQA, a test of recalled facts, falls from 82.7 to 75.9, and HellaSwag in German, a commonsense-completion test, from 91.4 to 85.8. These are the largest moves. The means over all knowledge benchmarks fall by only 1.0 (English) and 1.4 (German), since five others move by at most 0.7 and two rise. The German drop follows the mix, because German web text fell to 2.1% of it. For TriviaQA, the report says the cause “remains open”.

Long-context extension: reach

The third base stage trains for 201B tokens at 262,144 tokens per sequence. Its data is one third naturally long documents, one third mid-training data reweighted toward length, and one third natural mid-training data. The aim is to extend context “while preserving the capabilities of the mid-trained model”, which is the job of the latter two thirds. Aleph Alpha removed synthetic documents from the long pool “to avoid artificially favourable results on synthetic long-context benchmarks such as RULER” (a long-context retrieval test). Reweighting had a side effect, which the report calls a consequence and not a design choice: one English language domain rose from 1.3% to 24%.

The effect is hard to detect. To test the stage, the report ran a short test SFT from the checkpoints before and after it. The post-training benchmarks moved “within their eval-noise intervals”, and those intervals are wide: 63.3 [52.0, 74.0] against 73.5 [62.9, 83.0] on German AIME. The long-context benchmarks in that table have no interval at all (LongBench Pro 47.7 against 49.3). In proxy runs, a control with no long-context data “remains competitive”, and the clearest benefit appears “on retrieval tasks at the longest context lengths”. So the evidence is weak in both directions. It is the only base stage at 256K, the longest length Kolibri trains on, and the model is offered up to 1M beyond that.

The released base is the average of the last 20 saved checkpoints, 10B tokens apart. Such a checkpoint merge takes the mean of each parameter over saved points of one run. Writing each checkpoint as the sum of the updates before it shows what this does:

θˉ=θ0−η∑iwiui,wi=fraction of the averaged checkpoints saved at or after step i\bar\theta = \theta_0 - \eta \sum_i w_i u_i, \qquad w_i = \text{fraction of the averaged checkpoints saved at or after step } i

with θ0\theta_0 the starting weights, η\eta the constant learning rate and uiu_i the update at step ii, weight decay included. Weights are 1 before the window and fall in a staircase across it. So a uniform merge is a linear learning-rate cooldown applied after the fact, except that the gradients were computed without slowing down. That is why merging can stand in for a cooldown, and why one run yields many candidate cooldowns without new training. It damps noise and gives up progress: the merged point lags the end of the window by half its length. Kolibri’s window spans about 190B of the 201B stage (computed), so its base has taken only about half of that stage’s updates, and this isn’t an average of a settled tail.

k = 20: window 95% of the stage, mean effective weight 0.525. k = 4: window 15%, mean 0.925. Checkpoints every 50 steps.

Figure 2.What a uniform merge does to past updates, for a stage of Kolibri's length. It has the shape of a linear cooldown, but the gradients were computed at the constant rate.

Why not simply decay? Aleph Alpha reports that “a constant learning rate with checkpoint merging improves post-trained performance over a learning-rate cooldown”, citing its experiments and a paper of its own (Maskey et al., 2026). On a 30B model, after the same SFT, that paper’s aggregate reads 0.363 merged and 0.360 constant against 0.247 for a cooldown. So after SFT the gain belongs to the constant rate, and merging and not merging are effectively tied. Before SFT, merging lifts base scores over a raw constant-rate endpoint by 3.4 to 11.5 points in Kolibri’s proxy runs, and a wider window isn’t uniformly better: moving from 4 to 20 checkpoints on mid-training lifts knowledge scores and drops German code from 84.4 to 80.3. Kolibri’s rate itself was tuned on training loss alone. Tian et al. call this recipe warmup-stable-merge, and it descends from iterate averaging (Polyak and Juditsky, 1992) and stochastic weight averaging (Izmailov et al., 2018). Kolibri is the only model we found using it in production. Poolside ships an exponential moving average of its last 10 checkpoints after a decay, and NVIDIA uses merges to evaluate and select.

Merges don’t feed forward inside the base: mid-training and long-context start from raw checkpoints, and only SFT starts from the merged base.

SFT: a new kind of data, the same kind of loss

Supervised fine-tuning teaches instruction following, reasoning at a requested effort, and tool calling. It trains on demonstrations, recorded conversations showing the wanted behaviour. Part of them were written by other models, called teachers here; for Kolibri these include GLM-5.2, GLM-5.3 and Qwen3.8-27B. The loss is still cross-entropy, computed “only on the tokens the model produces at inference: the reasoning, answer and tool calls of each assistant turn”. User turns and tool responses remain as context and carry no loss, and clearly wrong tool calls are kept in context with their loss masked, so “the model learns the correction that follows without being trained to produce the error”.

Figure 3.Schematic of packing. Filled segments carry loss. In Kolibri's SFT pool, 42.4% of tokens carry loss, and in its long-context data 1.8% (computed from Table 21).

Sequences are packed: several conversations are concatenated to fill 256K tokens, with attention kept inside each. The run is 4,000 steps of 256 such sequences, about 268B tokens, at half the pre-training learning rate.

The mix matters beyond format. AIME 2026 sits at 74.6 ± 10.7 on the base, and after full SFT runs on three different mixes it reads 69.4, 94.0 and 65.2. The 94.0 comes from a mix with only 4.9% math tokens, so more demonstrations of a skill need not mean more of that skill. The SFT article takes this up.

Aleph Alpha then averages two of its mixes equally, a soup, and uses that as the starting point for RL. Unlike a checkpoint merge, a soup averages separate runs. The report’s reason is that both runs start from one base and stay close in weight space (Wortsman et al., 2022). It’s related to linear mode connectivity (Frankle et al., 2020), shown for runs from a late enough shared checkpoint that differ only in optimizer noise; these two also differ in data. To first order a soup’s logits are the mean of its parents’, so it behaves like an ensemble that costs one model at inference. The report doesn’t measure the barrier, and averaging isn’t always kind: while every soup beats its parents on the mean, one soup scores below both parents on a long-context benchmark (43.0 against 44.0 and 50.0).

RL: learning from its own attempts

In the last stage the model generates its own attempts, called rollouts, in 38 environments, each a source of prompts plus a way of scoring them. A verifier, usually a program such as a test suite, gives each attempt a reward. For language tasks without a program to check them, a separate frozen model acts as judge.

Attempts at one prompt form a group, 8 per prompt here. Each attempt’s tokens get a cross-entropy loss with the attempt itself as the target, multiplied by its reward minus the group’s mean reward rˉ\bar r:

g^=1G∑i=1G(ri−rˉ) ∇θlog⁡π(yi∣x),E[g^]=G−1G ∇θE[r]\hat g = \frac{1}{G}\sum_{i=1}^{G} (r_i - \bar r)\,\nabla_\theta \log\pi(y_i \mid x), \qquad \mathbb{E}[\hat g] = \frac{G-1}{G}\,\nabla_\theta \mathbb{E}[r]

This is REINFORCE (Williams, 1992), with a baseline from the group itself, and in expectation the true policy gradient up to a scale, before Kolibri’s corrections for stale rollouts. Rewards 1, 1, 0, 0 give signals +0.5, +0.5, −0.5, −0.5, so tokens of better attempts become more likely and those of worse ones less likely. Rewards 0, 0, 0, 0 give all zeros. A tied group isn’t wrong, just empty, and at very low or very high pass rates almost all groups are: with 8 attempts and a 1% pass rate, 7.7% of groups have any signal (computed). That is what “at a sufficient rate” means, and it explains why SFT precedes RL. Imitation before reinforcement is an old pattern in other fields too (AlphaGo, Silver et al., 2016).

Toy: invented setting, one pass-or-fail outcome per attempt.

8 attempts at one prompt: reward minus mean

pass fail

Example: 3 of 8 pass, mean 0.375, signals +0.625 on a pass and −0.375 on a fail.

Figure 4.A toy with one pass-or-fail outcome per attempt. With 8 attempts, 7.7% of groups have any signal at a 1% pass rate, 57% at 10% and 99% at 50%. Real rewards can be richer than pass or fail.

Kolibri generally drops tied groups, with two exceptions. All-pass groups stay when format penalties differ, and failed attempts in tool-use and terminal environments can learn from a hinted copy of the model, a self-distillation term. RL also runs for 1,000 steps of 2,048 sequences on 256 B300 GPUs, split into 96 for training, 144 for generation and 16 for judging. It is asynchronous, so rollouts may come from slightly older weights (the article on RL at scale). Muon’s learning rate falls a hundredfold from pre-training, to 1e-5.

The report prints the gain from SFT to RL for 13 selected benchmarks, and all 13 rise. Nine of them:

BenchmarkTestsSFTRL
AIME 2026Math92.196.0
GPQA DiamondGraduate science79.884.3
MMLU-ProKnowledge78.580.0
τ3-bench bankingTool use in a service task13.138.1
BrowseCompWeb search22.029.4
Terminal-Bench 2.1Command-line tasks23.227.7
LiveCodeBench v6Code75.385.9
IFBenchInstruction following48.278.1
LongBench ProLong context57.164.5

These come from Kolibri’s Table 25, “a selection”, with a different benchmark suite and protocol from the base-stage scores, so the two can’t be joined into one line. Where the rewards come from, and how they get gamed, is the RLHF article.

Distillation: the stage Kolibri doesn’t have

Most recipes end one step later. A single RL run over every domain has a problem that NVIDIA states: “each domain contributes only a relatively small number of samples to any given training batch”. So six labs train several teachers, usually one RL specialist per domain (Mistral’s differ by reasoning effort), then fold them into one student by multi-teacher on-policy distillation (MOPD). The student generates, and the teacher for that task scores what it generated. NVIDIA: “Every generated token receives a dense advantage from a single teacher forward pass, with no verifier and no sampling from the teacher”. Teacher here means a trained specialist, which is a different sense from SFT’s.

Nemotron 3 Ultra’s report shows the sequence on one model: Terminal-Bench 2.0 goes 34.5 at SFT, 44.5 after RL, 54.0 after distillation; BrowseComp goes 14.3, 31.0, 44.4. The teachers are themselves RL-trained, so part of the last gain is also RL’s. Aleph Alpha lists MOPD as future work. The distillation article continues from here. Don’t confuse this with the offline distillation some labs use on a base model (NVIDIA’s 1T tokens at Nemotron 3.5), which trains on fixed data with no student rollouts.

The knobs of a stage

Kolibri’s stages differ along a handful of settings, and most recipes can be read the same way. Two need a sentence first. BF16 and FP8 are 16-bit and 8-bit number formats: fewer bits make generation and serving cheaper and round the weights. Quantization-aware training (QAT) applies that rounding during training, so that what’s trained matches what runs. And Muon is the optimizer on Kolibri’s attention and expert weights, with Adam-family optimizers on the embedding, norm scales, router and output head (the optimizer article, the precision article).

KnobPre-trainingMid-trainingLong-contextSFTRL
DataBroad poolInstruction, STEM, codeOne third naturally longDemonstrations38 environments
Loss onNext tokenSame (not stated to change)Same (not stated to change)Assistant outputOwn rollouts: policy gradient, smaller terms
Sequence length16K64K256K256K, packedUp to 256K
Batch (sequences)4,6081,5367682562,048
Muon learning rate1e-31e-31e-35e-4 peak (computed)1e-5
Sum of Muon rates (computed, ignores weight decay)264341.01.80.01
PrecisionBF16 (card)SameSameNot printedFP8 rounding, computing in BF16
Reported lossTriviaQA −6.8, German HellaSwag −5.6Within noise in a test SFTNone on the meanNone among 13 shown

Tokens don’t say how far a stage can move the model. Kolibri’s Muon takes steps of a fixed size, the learning rate in the report’s RMS operator norm, on attention and expert weights. In RL, with weight decay off, the sum of rates therefore caps how far those matrices can travel: 0.01. In earlier stages, decoupled weight decay (1−2−121 - 2^{-12} per step) also moves them and limits their norm, so the sum is only a rough guide there. Allowing for decay, worst-case reach is roughly 8 in mid-training, 2 in long-context, 4 in one SFT run and 0.01 in RL (our arithmetic, assuming the norm limit holds at the start of each stage). So RL has several hundred times less reach than SFT, and it shows some of the largest late gains. A bound on movement says nothing about how much a small move changes behaviour. Don’t compare these rates across labs, because Muon has more than one scaling convention.

What each stage costs the others

A later stage moves the weights that earlier stages produced. With a constant rate, stochastic gradient descent has a finite memory: on a quadratic, a direction of curvature λ\lambda reflects roughly the last 1/(ηλ)1/(\eta\lambda) steps of data. Data overlap and a small learning rate limit the damage, and Kolibri’s weight decay adds its own memory of about 4,096 steps at any rate. The word for what remains is forgetting, after catastrophic forgetting in the continual-learning literature (French, 1999), used loosely here, since most stages keep some old data. Labs use similar language. Mistral: “Midtraining trades off acquiring new capabilities (e.g., agentic), against retaining the long tail of pretraining knowledge.” GLM-5: “sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities.”

Case A. Kolibri mid-training, M4 merges (4 checkpoints)

rising falling

Case B. Nemotron 3.5 Super, RULER-1M

falling rising in the recovery stage Score, 0 to 100, linear.

Figure 5.Two stage transitions. Kolibri's mid-training gains in math and loses on three of ten knowledge benchmarks, while the knowledge means fall by 1.0 (English) and 1.4 (German). Nemotron 3.5 Super's mid-training at 256K lost its 1M long-context score until a short recovery stage. Benchmarks and protocols differ between cases.

NVIDIA reports the largest single drop. Nemotron 3.5 Super mid-trains at 256K on software-engineering data after an earlier stage at 1M. RULER-1M falls from 79.9 to 33.5, and “a short long-context recovery phase” of 9B tokens returns it to 81.0. That is 14% of the tokens that lost it (computed), which suggests a shallow loss. Kolibri’s TriviaQA, by contrast, is 77.5 at the end of the base, against 82.7 where pre-training left it (the report’s own comparison, across merge windows of different sizes), and no stage targeted it. Poolside reports the same pattern one stage later: without instruction-following supervision, “RL fine-tuning tends to exhibit catastrophic forgetting of these behaviors”.

A before-and-after drop isn’t yet a cost. The cost is the difference from a run that continued as before. Kolibri has that arm only at proxy scale: at 400B tokens, its production mix scores 55.0 on TriviaQA against 63.5 for continued pre-training, though the two also differ in sequence length. Part of what’s measured may be the model following its mix. Across its 84B runs, German HellaSwag rises from 65.7 at no German web text to 79.4 at 19%, which looks more like tracking than erasing.

Labs respond by keeping a skill in the data, which the continual-learning literature calls rehearsal. Nemotron 3 Ultra’s long-context stage spends 8% of its iterations at 4K “to maintain the accuracy of the short benchmarks”. Kolibri’s RL has a long-context environment and German ones “to prevent the model from regressing”. GLM-5 goes further, distilling from its own earlier stages, close to what the literature calls Learning without Forgetting (Li and Hoiem, 2016).

Measuring a skill doesn’t protect it. In Kolibri’s proxy runs, the production mix gives up 5.5 points of German HellaSwag against a reference mix for 0.09 points of the aggregate score the mix was optimized on, about 61 to 1 (computed). A weighted sum will sell any skill with a small weight, and Aleph Alpha concludes: “Preserving a capability thus requires its own term or constraint in the objective.” Separately, a skill that no benchmark tests can decline unnoticed.

Stages also interact the other way, since a base is judged by what it becomes. Kolibri scores mid-training mixes after SFT, and they separate only at the larger proxy budget. When it tried to predict SFT mixes by merging 20 specialists, “merged and trained scores do not correlate”, at a Spearman ρ of −0.09 over six weightings, though a smaller validation ranked them correctly. NVIDIA warns that “very long SFT stages can reduce how much the model later gains from RL”.

Kinds of token

Most recipes state a token count, and two counts can’t be compared until you know their kinds. Two questions sort them out: how far along the pipeline was this count taken, and does it count every token or only those with a term in the loss?

Kolibri pre-training

Raw is at least 10x trained.

Kolibri SFT pool

42.4% of tokens carry loss.

Kolibri long-context SFT data

1.8% of tokens carry loss: 55x fewer than all tokens.

Figure 6.Token counts for one stage. Each panel has its own scale, because the counts differ by orders of magnitude.
KindMeaningExample
RawWhat the pipeline read before curatingOver 200T for Kolibri (blog only)
PoolAvailable for the mix after curating29.3T materialised for Kolibri, about 27T unique for Laguna
TrainedPassed through training, counting repeats20T for Kolibri, over 30T for Laguna
PackedTrained, including context with no loss268B per SFT run for Kolibri
LossTokens with a term in the loss142.4B of the 335.8B in Kolibri’s SFT pool

Repeats are why a run can train on more than it has. Laguna samples over 30T from about 27T unique tokens. Nemotron 3 Ultra’s card gives 14.8T for its dataset and its report gives 20T trained, which implies repeats (our inference).

Loss tokens are why a volume can mislead. In Kolibri’s SFT pool 42.4% of tokens carry loss, and in its long-context data 1.8%, a gap of 55x between all tokens and loss tokens (computed), because the long document is context and only the short answer is trained on. Those are pool figures, and no source prints loss tokens for one run. A context token still costs compute and affects every other term through attention.

Reported totals also differ in scope. Kolibri’s report labels 24T “Pre-training tokens” in one table, when the pre-training stage proper is 20T. DeepSeek’s 32T for V4 includes its context extension, which other labs count separately. And NVIDIA’s Indic stage is a nominal 100B of which 10B is Indic data.

What labs share, and what they don’t

Labs that document a recipe use the same vocabulary. They warm up the learning rate, shift the data toward higher-quality or agentic material later on, and run SFT before RL. Six labs state a gradient-norm clip, most at 1.0, not all for the same stage. They differ on the questions below, each developed in a later article. Mostly there’s no head-to-head result, and the reasons labs give come from different models at different budgets.

QuestionPositionsArticle
Start from scratch?Stated by Aleph Alpha, Mistral, Poolside, DeepSeek V4.1. Nemotron 3.5 starts from older bases, DeepSeek V3.2 from V3.1. Most others don’t sayS1
Optimizer in pre-trainingMuon family at 7 of the 9 labs that state one (Aleph Alpha, Moonshot, DeepSeek, Z.ai, Poolside, Qwen, Thinking Machines). AdamW at Mistral and Xiaomi, who move to Muon or a variant laterO1
Learning-rate shapeSix of the seven labs that describe one decay it, with a long hold first (NVIDIA, Poolside, DeepSeek, Xiaomi) or as a cosine (Moonshot, Z.ai). Aleph Alpha holds a constant rate and mergesS1
Grow the batch during pre-training?Yes at Mistral, DeepSeek V4, Xiaomi, Z.ai. Qwen tested it and declined. DeepSeek V4.1 keeps it fixedS1
What “mid-training” meansEight labs use the word for a stage, for at least four operations (our grouping)S2
Long context, when?After mid-training (Kolibri, MiMo-V2-Flash), inside it (GLM-5, MiMo-V2.6, LFM3), before it (Laguna, Nemotron 3.5 Super), inside pre-training (DeepSeek), near the end at K3S2
Precision in pre-trainingNVFP4 on most layers (NVIDIA), FP8 (Mistral, Xiaomi), BF16 (Aleph Alpha, Poolside). QAT starts anywhere from mid-training to RL, and others quantize after trainingO2
Synthetic text in pre-trainingUsed at Aleph Alpha (24%), NVIDIA, Moonshot, Poolside. Filtered or avoided in specific scopes at DeepSeek and Z.ai. MiniMax prefers interleaved dataD3
RewardRLHF by name on a text model at NVIDIA, with the reward from a generative reward model. Verifiers, judges and other reward models elsewhereS4
DistillationMulti-teacher at 6 labs (NVIDIA Ultra and 3.5 Super, Xiaomi, DeepSeek, Moonshot, Mistral, Liquid). Z.ai distills across its own stages. None described at Poolside or Thinking Machines, and Aleph Alpha lists it as future work. LFM3 runs RL after itS6

Stated sizes differ too. Pre-training budgets span about 5x, from 10T to 48T. The stage called mid-training spans about 61x on one kind of token, from 65.4B at Nemotron 3.5 Super to 4T at MiMo-V2-Flash. Size alone doesn’t prove the word covers different operations, but the operations differ too. Kolibri’s shifts data toward reasoning and code. NVIDIA’s primes the model on software-engineering data at long context, mostly without chat templates. Poolside’s teaches a chat format. Mistral’s and Xiaomi’s change a learning rate or optimizer as well.

Figure 7.Stated stage sizes on a log axis, in tokens trained, except the hollow dot. A model with no stated size for a stage has no dot, and the stage may still have run. Some pre-training totals include multimodal tokens or context extension.

Similar budgets also hide different training densities. Kolibri trains 256 tokens per parameter by its own figure, and LFM3-1.7B about 10,900 (computed from 18.5T). And not every number is a plan: NVIDIA says it “made the practical decision to cut the total pretraining token horizon down to 20T tokens” after a divergence near 16T.

Where the models in the map sit

One row per model, in the source’s own order, using its sizes. A missing size means we did not find one; the stage may still have run. CPT is continued pretraining, a further pre-training run on a finished checkpoint.

LabModelBase stagesPost-training
Aleph AlphaKolibriPre 20T, mid 3.44T, long-context 201BSFT, RL with a self-distillation term
NVIDIANemotron 3 UltraPre 20T (15T + 5T), long-context CPT 33BSFT in 2 stages, RL, specialists, MOPD
NVIDIANemotron 3.5 SuperFrom Super 3’s base: distill 1T, Indic CPT 100B, long-context 35B, mid 65.4B, recovery 9BSFT in 2 stages, RL, MOPD
MistralLarge 4Pre 65% + 35% (total not stated), midSFT, RL, MOPD with 3 reasoning-effort teachers
XiaomiMiMo-V2.6 Pro, FlashPre text then multimodal: 27T + 3T, 26T + 22T. Mid at 256K, then 1MShort SFT, one mixed RL run, MOPD
MoonshotKimi K3Pre 8K to 64K, “cooldown” 256K to 1M (no token budget stated)SFT, 9 RL specialists, MOPD
PoolsideLaguna XS.2Pre over 30T at 4K, context 100B + 100BMid about 60B, SFT 3 x 40B, RL
DeepSeekV4One stretch of over 32T, including context extensionPer-domain SFT and RL, then distillation replacing mixed RL
DeepSeekV4.1-FlashOne stretch of 45TSFT, RL, distillation with over 40 teachers
Z.aiGLM-5Pre 27T, mid 1T + 500B + 50B, sparse adaptation 20BSFT, three RL stages, distillation across stages
LiquidLFM3-1.7B22T including 3.5T midSFT 1.75T, mixed RL, MOPD with 6 teachers, agentic RL

NVIDIA also fine-tunes prediction heads for speculative decoding and quantizes after this, and those steps are left out here. We found no usable recipe for StepFun’s Step models, Tencent’s Hy models (qualitative lines only) or MiniMax’s M3 models. The Qwen3.5 and 3.6 cards say “Pre-training & Post-training”. gpt-oss, Gemma 4 and Inkling give little beyond a token count or a reference to an earlier generation. These say too little to support or contradict anything above.

What the reports leave out

No lab in our corpus states what each stage costs. There are fragments. Mistral’s pre-training took 45 days and about 4.15M GPU hours on 3,840 GB200s. Kolibri’s took a net 377,000 GPU hours on B200s, and its final RL run 16,900 hours weighted to B300 equivalents. gpt-oss-120b’s “training run” took 2.1M H100 hours. DeepSeek V3.2’s RL budget “already exceeds 10% of the pre-training cost”. Xiaomi spent $2.6M on RL for MiMo-V2.6-Pro, split into 43.8% rollout, 43.5% training and 12.7% grading.

These can’t be stacked. They use different hardware and scope, and few say whether they include proxy runs, ablations (experiments that remove one piece) or failed runs. For that reason we don’t chart compute shares across labs.

Reports describe what shipped. Failure stories exist at about five labs, covering pre-training stability (Nemotron 3 Ultra’s two divergences, Laguna M.1’s expert collapse), architecture and RL infrastructure. Apart from Kolibri’s account of its long-context stage, we found no lab saying that a stage gave little. When a table shows gains, check the baseline (an SFT checkpoint, an earlier model, a proxy), the model (a small test model or the release) and the budget.

Easy to get wrong

  • Kolibri is the example because it’s best documented. Use it to learn the vocabulary; the labs table shows which of its choices are common.
  • Mid-training sizes of 65.4B and 3.44T differ by 53x and are different operations, so neither model is “more mid-trained”. Check what changed before comparing.
  • A jump in one stage isn’t that stage’s power. We read the rise from 30.0 to 58.9 at SFT as partly a format the base couldn’t produce, and the report doesn’t break it down. On AIME 2026, where format matters less, results still range from 65.2 to 94.0 across mixes.
  • Kolibri’s base scores 30.0 on the 12-benchmark mean and 81.1 on the English overall score of its base-model suite, and the released model scores 75.5 on the overall score of a post-training suite. None of this is a regression, since these are different benchmarks. Never join them into one line.
  • Mistral’s report gives no total pre-training tokens and K3’s gives no token budget. That says nothing about the size of either stage.
  • “Pre-training” need not mean from scratch or only the first stage: Nemotron 3.5 begins from older bases, and Kolibri’s Table 1 uses the label for its three base stages together.

Check yourself

  1. Kolibri’s SFT uses 256 sequences of 262,144 tokens per step, for 4,000 steps. Roughly how many packed tokens is that? Why can’t you multiply it by 42.4% to get its loss tokens exactly?
    Answer256×262,144≈67.1256 \times 262{,}144 \approx 67.1M per step, times 4,000 is about 268B. The 42.4% is for the whole SFT pool, and a run trains on a mix drawn from it, with different shares of domains whose loss fractions differ widely.
  2. A report says “mid-training: 65B tokens”. Another says “mid-training: 3.4T”. What can you conclude about which model was trained more thoroughly?
    AnswerNothing yet. The word can mean priming on agentic data, a shift to higher-quality data, or a stage that teaches a chat format. Check what changed and which kind of token each number counts.
  3. A model never produces a valid tool call on a prompt. What does the group-relative term do with its 8 attempts? What about a 1% pass rate?
    AnswerAll rewards tie, so the term is zero for that group. At 1%, about 7.7% of groups have any signal. SFT raises the rate until most groups contain both passes and fails. Kolibri also has a separate self-distillation term for some such cases.
  4. Kolibri’s RL has a sum of Muon rates of 0.01 and no weight decay. What does that tell you, and what doesn’t it?
    AnswerAttention and expert matrices can move by at most 0.01 in the report’s norm over the whole stage, far less than in SFT. It doesn’t say how much a small move changes behaviour, and IFBench still rises from 48.2 to 78.1. It also doesn’t cover the other weights, which use other optimizers.
  5. Why isn’t a 6.8-point drop in TriviaQA during mid-training automatically the cost of mid-training?
    AnswerThe cost is the difference from what continuing as before would have given. Kolibri has that comparison only at proxy scale, and there sequence length also differs.

In the map

Methods

Taxonomy nodes

Further reading

↑