Open Models Almanac

Training stages · S3

Supervised fine-tuning: a cold start for what comes next

SFT has to put in place every behaviour RL will later improve, and each token of it teaches the model to imitate a teacher; the measured effects are in what is kept, masked and averaged

Illustration of a young ink-blue sapling in a pot, tied loosely to a light teal frame, with a few coral cuttings from a much larger tree laid beside the pot and a gold watering can at rest.

Kolibri’s supervised fine-tuning runs 4,000 steps of 256 packed sequences, each 256K tokens long: about 268B tokens per run. LFM3’s card lists 1.75T tokens of SFT for a 1.7B model, after 22T of pre-training. Thinking Machines says only that the SFT before Inkling’s reinforcement learning “accounts for a small fraction of compute”. The first two are more than six-fold apart, and only Kolibri’s says what kind of token it counts; the third gives no number. Of the 16 labs on this site, 13 describe an SFT stage, two of them only in passing, and four say it is there so that RL has something to work on.

A base model continues documents. Ask it a question and it may answer, or write the next question of an exam paper. Reinforcement learning (RL) can improve a behaviour only if the model already produces it some of the time, because a reward that is never earned teaches nothing. Poolside describes what happens otherwise in the report on its Laguna models: without instruction-following data in fine-tuning, “over half of agentic tasks receive zero reward because the model violates system-message constraints before any coding work is evaluated.” SFT closes that gap by training on demonstrations of the behaviour, with the same cross-entropy loss as pre-training. Aleph Alpha calls the stage “a cold start for RL”, Moonshot “a high-quality cold-start policy”.

Background

The primer sets out the stages. Pre-training teaches a base model to predict text, and mid-training and long-context extension (S2) shift its data and lengthen its sequences. Supervised fine-tuning (SFT) then trains it on demonstrations: conversations in which the assistant’s turns show the behaviour wanted, written by people or, far more often now, by other models. The model that writes them is a teacher and the model trained on them the student (D3 covers how such data is generated). In classical terms SFT is behaviour cloning, and training on a teacher’s generated outputs is sequence-level knowledge distillation (Kim and Rush, 2016, who used the teacher’s beam-search output). The data is off-policy: written by another model, not sampled from the student.

RL follows (S4, S5). An RL environment is a task with a program that scores attempts at it, such as a repository and its tests or a search tool and a reference answer. The model produces its own attempts, called rollouts, each gets a reward, and the update makes high-reward attempts more likely. A cold start is a model good enough at a behaviour for RL to begin improving it.

A demonstration reaches the model through a chat template: special tokens that mark where the system prompt, each user turn, each assistant turn and each tool result begin and end. Reasoning models also write a reasoning trace between markers such as <think> and </think> before the answer. An agent calls tools (a shell, a search engine, a file editor) and reads their output, and one recorded episode of that is a trajectory.

Packed tokens fill the training sequences, several conversations to a sequence, with attention masked at each boundary so no conversation sees another (Kolibri and NVIDIA both say so). Loss tokens are the ones the loss is computed on, a minority, as the next sections show.

Nothing about SFT is in a model’s configuration or code, so we read technical reports first, then model cards and official blogs. We count labs, not models. A blank means we searched and found nothing, which doesn’t mean the lab didn’t do it. Kolibri is the worked example because Aleph Alpha’s report describes its SFT in more detail than any other. That makes it the best-documented recipe, not necessarily a typical one.

The article is about the labs’ own training. Model cards also tell users how to adapt a released model, and 37 of the 93 corpus documents that mention fine-tuning carry only that kind of text, licence terms, demos, safety tests or unrelated uses of the word; counting them would add MiniMax and OpenAI, which describe no SFT stage.

Why RL needs a cold start

Many of the labs here train with group-relative RL updates (S5). For each prompt the model makes GG attempts, and each attempt’s advantage is its reward minus the group’s mean. The update sums each attempt’s advantage times the gradient of its log-probability, so if all GG attempts score the same, all zero or all one, the prompt contributes no gradient. With rewards of 0 or 1 and a per-attempt success rate qq, the chance that a prompt teaches anything is

1−(1−q)G−qG≈1−(1−q)Gfor small q.1 - (1 - q)^G - q^G \approx 1 - (1 - q)^G \quad \text{for small } q .

At q=1%q = 1\% and G=8G = 8 that is 7.7%; at q=20%q = 20\% it is 83% (computed). Over prompts of mixed difficulty the share is lower than the formula gives at their mean: half at q=0q = 0 and half at q=40%q = 40\% give 49%, not 83%. Poolside’s zero-reward tasks are such a subpopulation, with qq near zero. Aleph Alpha states the requirement in words: SFT “establishes desired behaviours that the model must produce at a sufficient rate for RL to become effective in improving them.”

Only one lab in the corpus reports RL with and without SFT on the same base. For MiMo-7B, a 7B model from 2025, Xiaomi’s repository lists AIME 2025 (a 30-problem math competition, averaged over 32 attempts) at 24.3 for the base, 46.3 after RL from the base, 44.3 after SFT alone and 55.4 after SFT then RL. That shows SFT then RL ahead of RL alone, by 9.1 points, but not why. The base already solves about a quarter of the AIME problems, so RL from the base was not starting from zero reward everywhere, and the gap fits SFT passing on its teacher’s knowledge as well as it fits a cold start. Each arm is one run, and whether the two RL runs used the same compute isn’t stated.

Prompting can raise qq too: DeepSeek’s “cold-start” for reasoning with tools in V3.2 is a set of system prompts under which the model “is occasionally able to generate the desired trajectories”. And SFT doesn’t always raise qq far enough: Aleph Alpha found many agentic environments “difficult to solve from the SFT checkpoint” and gave the model hints during RL instead.

SFT’s cost is that everything it teaches is imitation: RL starts from the teachers’ style, length and mistakes.

What the loss is on

The mask

SFT minimises the same next-token loss as pre-training, but only on some tokens. With mt=1m_t = 1 on a token that carries loss and 00 elsewhere, summed over every token in the batch,

L(θ)=−1∑tmt∑tmtlog⁡πθ(xt∣x<t).\mathcal{L}(\theta) = -\frac{1}{\sum_t m_t} \sum_t m_t \log \pi_\theta(x_t \mid x_{<t}).

The masked tokens are still in the input, so the model conditions on them, but they are not targets. This loss mask is separate from the attention mask that keeps packed conversations apart. Kolibri’s rule, which NVIDIA states too: “We train with cross-entropy loss only on the tokens the model produces at inference: the reasoning, answer and tool calls of each assistant turn. User turns, tool responses and the special tokens of the chat format are masked out.” Kolibri normalises over the whole batch, so within a batch every supervised token has the same weight. Averaging per conversation first would give each token of a two-line reply more weight than each token of a 20,000-token reasoning trace.

The template has to match what inference renders, and labs differ on one point: whether earlier turns keep their reasoning. Kolibri’s template drops it, as gpt-oss’s format does. To train a later turn the context must show earlier turns without their reasoning, but to train an earlier turn its reasoning must be there, so one sequence can’t supervise every turn. Aleph Alpha splits each conversation into one sample per user turn, “with loss on the last turn only”, 2.27 samples per conversation on average. Laguna, Kimi K3 and GLM-5’s “preserved” mode keep earlier reasoning, so one sample can supervise every turn, but the serving stack has to agree: Poolside found vLLM’s reasoning parser misreading the earlier </think> tags in Laguna’s history, and fixed it upstream.

Loss tokens end up a minority. Kolibri’s SFT pool holds 335.8B tokens and 142.4B loss tokens (42.4%). In its long-context data the share is 0.9B of 49.8B, since a 200K-token document is context and the answer is short. When Xiaomi fine-tuned a Qwen3.5-9B student on MiMo-generated data, the mix held 77.4B tokens, 27.2B of them loss tokens (35%, computed).

Keeping mistakes in context

An agent’s trajectory often contains a wrong step: a call to a tool that doesn’t exist, a malformed argument, a command repeated after it failed. Dropping the trajectory loses the recovery that follows, and training on all of it teaches the error. Kolibri masks the loss on “clearly wrong tool calls: calls whose tool response reports an unknown tool or invalid arguments, and exact repeats of the previous call that never receive a different response. These turns stay in the context, so the model learns the correction that follows without being trained to produce the error”. GLM-5 does the same for coding and agent data: “Erroneous segments within trajectories are retained but masked out in the loss function”. Nemotron 3 Ultra applies the idea to chat, building conversations whose earlier assistant turns “may contain suboptimal responses rather than the best-ranked candidates” and training “only on the final assistant response”.

losscontext onlytoy token counts

Loss on 820 of 2,270 tokens (36.1%).

Figure 1.A toy agent trajectory and which of its tokens carry loss. Pre-training trains on all of them. SFT trains on the assistant's reasoning, calls and answer. Kolibri and GLM-5 also mask a clearly wrong step but keep it in the context. Token counts are invented; in Kolibri's real SFT pool 42.4% of tokens carry loss.

In imitation learning, a policy trained only on an expert’s states never sees where its own mistakes lead, and its errors compound over a long episode (Ross and Bagnell, 2010). DAgger (Ross et al., 2011) has an expert label the states the learner visits; DART (Laskey et al., 2017) perturbs the expert’s rollouts so they contain recoveries. A wrong call kept in context is closer to DART, since both the error and the recovery are the teacher’s (our reading). The DAgger step comes later in several pipelines, as on-policy distillation, where the student generates and a teacher scores its tokens (S6). NVIDIA found that a very light SFT on data from the teacher’s training distribution before that stage raised Ultra’s GDPVal score after distillation from 35.3 to 46.7 and BrowseComp from 33.0 to 44.4, while HLE barely moved (26.3 to 26.7); no repeats are reported.

One outside paper tests masking errors directly. Step rejection fine-tuning (Slinko et al., 2026), which Kolibri cites, has a second model read each software-engineering trajectory and label every step good, harmful or unnecessary, then masks the harmful ones. For a Qwen2.5-Coder-32B student on SWE-bench Verified (real GitHub issues, scored by whether the patch passes the tests), adding failed trajectories with their harmful steps masked raised resolved issues from 30.9% (successful trajectories only) to 32.2%. Adding the same failed trajectories unmasked lowered it to 28.5%. Each is one trained model, scored over seven evaluation runs. The paper’s one test across training seeds, on the failed trajectories alone, gives masking 1.1 points (95% interval 0.4 to 1.8). No arm matches Kolibri’s case of passing trajectories with errors masked; the nearest, masking harmful steps in successful trajectories too, scored 29.4. Aleph Alpha reports no result for its loss masking alone.

Two other masks are tied to specific behaviours. To let users cap how long the model thinks, NVIDIA cuts reasoning traces at random lengths, closes them with </think> and keeps the answer, masking </think> “to prepare the model for inference-time budget control”: at inference the serving code inserts it when the budget runs out, and the mask keeps the model from learning to end its own reasoning early (our reading). Tool responses are masked by both labs that describe their full loss, Aleph Alpha and NVIDIA. NVIDIA’s one measurement of that is from mid-training, where training on them “lowered SWE-bench Verified pass@1 by about 10 points” (solved on the first attempt) in a single research run.

Masking costs loss tokens per step and needs a rule for what counts as wrong. Kolibri’s rule is mechanical; Slinko et al.’s labeller agreed with hand labels on 59% of steps and found 28% of the steps people called harmful.

Where the demonstrations come from

Most SFT data is written by models. Nine labs say where theirs comes from: three name the outside models that wrote it, five say their own specialist or expert models did, and Poolside names neither:

LabTeachers named for SFT dataOwn models used
NVIDIA, Nemotron 3 Ultraby domain: GPT-OSS-120B, DeepSeek-V3.2 and V3.2-Speciale, MiniMax M2.5, GLM-5, Qwen3-Coder-480B and others
NVIDIA, Nemotron 3.5Kimi K3, Kimi K2.6, DeepSeek V4 Pro and Flash, GLM-5.2, Qwen3.5
Aleph Alpha, KolibriGLM-5.2, GLM-5.3, Qwen3.8-27Ba smaller 8B model trained in RL environments
Thinking Machines, Inkling“open-weights models including Kimi K2.5”
Poolside, Lagunaan unnamed open-source teacher for the 30% of tokens that are agentic coding without reasoning; the 45% with reasoning, source unstatedin-house synthetic tasks
Moonshot, Kimi K3“domain-specialized models from the prior Kimi series”, RL experts
Xiaomi, MiMo-V2-Flash“in-house domain-specialized model checkpoints”
Z.ai, GLM-5expert RL models, rejection sampling
DeepSeek V3.2 and V4; Qwen3.5-Omnispecialists fine-tuned and RL-trained from the same base (V4 and Omni say SFT, then RL); V4 says its final model learns from them by on-policy distillation

A specialist in the last rows is a teacher: a copy of the base trained hard on one domain, by SFT or RL (an RL expert), whose outputs then train the general model. DeepSeek calls this “Specialist Distillation” and reports that the general model trained on it lands “only marginally below” the specialists, with the gap “effectively eliminated through subsequent RL” (no numbers given). The word means something else in Kolibri’s mix search below.

Rejection sampling draws several answers per prompt and keeps the ones that pass a check: a matching reference answer, passing tests, a judge’s approval. Fine-tuning on what survives is rejection-sampling fine-tuning (RFT, Yuan et al., 2023), a relative of STaR’s train-on-your-own-correct-answers loop. It fits the teacher’s outputs conditioned on passing, so a check on the final answer also keeps right answers reached by wrong reasoning. Z.ai and NVIDIA name it, and most of the filters in the next section do the same thing without the name.

Some SFT data comes from RL environments, rolled out by another model. Kolibri’s retrieval data are GLM-5.2’s rollouts in the retrieval environments, and its hallucination data come from an 8B model trained in its Merlin–Arthur hallucination environments: “Both are rollouts of RL environments and provide a cold start for RL.” Moonshot trains RL experts per domain and effort level and collects their trajectories “for supervised fine-tuning and multi-teacher on-policy distillation”, without saying which SFT run uses them.

Teachers have costs. Aleph Alpha writes that “several of our teacher models were built in China, so they carry known political biases on sensitive topics”, and filters for them. Licences constrain the data too: for Ultra, NVIDIA kept only the OpenResearcher search trajectories whose source licence cleared commercial use, about 21.7K.

Which data is kept

Filters are where most labs say the recipe is. Nemotron 3.5 removes “reasoning responses shorter than 5,000 tokens”. GLM-5 keeps, for math and science, “only problems that are challenging for the GLM-4.7 model”. Poolside removes from its math data “overly easy samples based on solve rates from open-weight LLMs”. Kolibri keeps a generated German answer only if it matches its seed’s reference and a software-engineering trajectory only if it passes the tests. None of them reports what a filter did to the scores. Kolibri also reports a filter it chose not to apply: chat answers go unfiltered, “as it adds compute without a known benefit”, and its benchmark decontamination removed 0.005% of rows with “no measurable difference”.

Liquid orders LFM2’s SFT data as a curriculum (Bengio et al., 2009), easy items first. Difficulty is the mean solve rate of 12 models, four of them Liquid’s own, and a model trained on prompt features predicts the order. Liquid reports no run without the curriculum, and if the order spans the whole run (LFM2 trains for three epochs and doesn’t say), an easy-to-hard order also puts the hard items where the learning rate decays.

Choosing the mix

A mix is a set of shares, one per cluster of the data pool, and finding good shares is a search over training runs (D2). Fitting shares from trained candidates needs at least 3(m+1)3(m+1) runs for mm clusters (Chen et al., 2026, as Aleph Alpha cites them): with 20 clusters, 63 SFT runs of 16.78B tokens, 1,057B in all.

MergeMix (Wang et al., 2026, a method for mid-training) replaces most of those runs with weight averages. Fine-tune one specialist per cluster from the same base, briefly, then for a candidate mix with shares αk\alpha_k that sum to one form

θ(α)=θ0+∑kαk(θk−θ0)\theta(\alpha) = \theta_0 + \sum_k \alpha_k (\theta_k - \theta_0)

and score the merged model on benchmarks as a stand-in for a model trained on that mix. Here a specialist is only a scoring device. The argument: a short run on cluster kk moves the weights by about η\eta times the sum of its gradients, and a mixed run’s gradients are the α\alpha-weighted sum, so for plain gradient descent the two moves agree to first order, differing in curvature terms. Two caveats apply to Kolibri: Muon, its optimizer for matrices, doesn’t update linearly in the gradient, nor do the Adam-type optimizers of its other parameters, and to first order the score is linear in the shares, so whatever makes a mixture beat its best specialist lies in the terms the argument drops.

Kolibri trained 20 specialists on 4.50B tokens each, 90B in place of 1,057B, scored 69 merges, and trained eight weightings for real at 16.8B tokens. The merges didn’t predict the trained runs. Over the six weightings with both scores the Spearman correlation is −0.09, and “The best merge trains worst, at 50.0 %.” Six points can’t show the absence of a correlation, but the 20-specialist merge gave no usable ranking. An earlier check on the smaller proxy model, with two or three specialists at 20B tokens each, had ranked better (0.77 to 0.94 on two benchmarks, 0.26 to 0.32 on MATH-500, over four to six weightings). Aleph Alpha can’t say which change broke it, more clusters or shorter specialist runs; the check also ran on a smaller model.

Aleph Alpha then chose with full runs: four searched candidates and the baseline B0 at 168B tokens each, and three of those continued to 268B. Those three are B0, which samples the pool roughly in proportion with German upweighted, and C1 and C2, two searched mixes.

Figure 2.Training tokens behind Kolibri's SFT, by step (computed from the report's run lengths). The merge search was cheap; the decision came from full-length runs. Two of them, C1 and C2, were averaged into the released model; the rest went to choosing them.
Figure 3.Kolibri's three full-length SFT mixes after 4,000 steps. C1 has less math than B0 but the best AIME 2026 score; C2, with the least math, scores lowest; B0, sampled close to the pool's natural proportions, is within noise of the model before SFT. Single runs on a 30-problem benchmark, and C1 was chosen on benchmarks that include AIME 2026. ± is the report's interval (not defined in its §3.1.4.3), printed only for base and B0.

C1 scored 94.0% on AIME 2026 with math at 4.9% of its tokens, against 69.4 ± 9.3 for B0 with 18.3%. The report doesn’t define this section’s ±. For its fine-tuning ablations it gives 95% intervals from runs that differ only in seed, and counts two runs as different when their scores are about 1.4 half-widths apart; applied here (our step), C1’s 24.6-point lead holds. B0 is within noise of the model before SFT (74.6 ± 10.7) and ends 7.3% of its AIME 2026 answers in repetition loops or at the context limit, against 2.1% for the base and none for C1. What C1 changed was which math it kept: it “drops or downsamples in particular the math datasets with on average easier problems and longer reasoning traces.” Aleph Alpha adds: “Other causes are possible, as C1 and B0 also differ in other domains.” These are single runs on 30 problems, where one problem is 3.3 points, and C1 was picked on a mean of benchmarks that includes AIME 2026.

Stages, lengths and reasoning effort

Of the labs that describe the structure, only NVIDIA names two SFT stages for one model. It runs two for Nemotron 3 Ultra, the second extending packed sequences from 294,912 to 515,000 tokens with long-context data up to 512K. Nemotron 3.5 Super’s base was trained on text only, and vision first enters at SFT, in two stages “because learning vision and text capabilities simultaneously is challenging”. Nemotron 3.5 Lightning, text only, has one. GLM-5 “extends the maximum context length to 202,752 tokens during SFT”. Poolside’s word differs: its roughly 60B-token “mid-training” is instruction data spanning chat, reasoning and agentic coding, at SFT’s learning rate, which other labs would call SFT, so by that reading Poolside’s comes in two parts too.

Many reasoning models let the user choose how long they think. A line in the system prompt, such as Kolibri’s “Reasoning effort is set to low. Think briefly through only the essential steps in the user’s language, then proceed directly to the answer.”, selects an effort level (low, medium or high), and the model is expected to reason for a shorter or longer time to match. In Kolibri, SFT is where the model first learns to obey that line; Mistral leaves it to RL (below).

Kolibri’s traces have no effort label, so Aleph Alpha assigns one after the fact. A single length threshold would fail: median reasoning length per dataset runs “from 27 to 90 223 tokens”, and hard prompts reach “up to 31.3 times” the length of easy ones. So each reference group gg, one dataset at one difficulty (graded by a small model distilled from GLM-5.2), gets its own 30th and 70th percentiles of reasoning length, ℓg30\ell^{30}_g and ℓg70\ell^{70}_g, and a trace of length LL is placed on [0,1][0, 1] by

ug(L)=clip[0,1] ⁣(0.3+0.4 log⁡L−log⁡ℓg30log⁡ℓg70−log⁡ℓg30).u_g(L) = \mathrm{clip}_{[0,1]}\!\left(0.3 + 0.4\,\frac{\log L - \log \ell^{30}_g}{\log \ell^{70}_g - \log \ell^{30}_g}\right).

The percentiles land on 0.3 and 0.7, the edges between low, medium and high. Each label ee has a centre mem_e (0.15, 0.5, 0.85) and a width wew_e (0.3, 0.4, 0.3), and is drawn with probability

P(e∣L,g)∝exp⁡ ⁣(−12(ug(L)−meβ we)2),β=0.15.P(e \mid L, g) \propto \exp\!\left(-\tfrac12 \left(\frac{u_g(L) - m_e}{\beta\, w_e}\right)^{2}\right), \qquad \beta = 0.15 .

At each edge the two neighbouring labels are equally likely. The draw is seeded per row so the label doesn’t switch at one exact token count, “which the model could otherwise learn as a length at which to stop.”

lowmediumhigh30th and 70th percentiles in use

Toy groups: 60 traces each, log-normal lengths with invented medians. Each dot draws its label once, with a fixed seed. Shares below are expected shares under Equation 12.

chat: low 30% · medium 40% · high 30%

code: low 30% · medium 40% · high 30%

math proofs: low 30% · medium 40% · high 30%

Figure 4.A toy version of Kolibri's effort labels. With one threshold for all data, 80% of the short chat traces are "low" and 80% of the long proofs "high". Per-group thresholds give each group the same 30/40/30 split. The soft draw mixes labels only close to a threshold. Distributions are invented; the rule is the report's Equation 12.

Labelling after the fact has one risk: the model might never need the label, since a hard prompt has a long trace whatever the label says. What makes it read the label is repetition. 9.5% of Kolibri’s opening prompts appear twice with reasoning, and Aleph Alpha estimates that about 37.3% of the pairs receive two different labels (the report doesn’t say whether of all pairs or of those whose lengths differ two-fold); for those, the label is the only difference in the input. In the released SFT model the median reasoning length orders low, medium and high correctly on 23 of 24 benchmarks. Scores move less: the mean over 14 benchmarks is 64.9 at low and 67.8 at high, with a paired standard error of 0.8 points for high against low, and medium to high adds 0.7 points.

NVIDIA takes effort from a teacher’s mode instead, GPT-OSS-120B’s medium mode for Ultra, and mode names don’t carry over: Nemotron 3.5 Super’s medium effort is taught “with data generated by Kimi K3’s low-effort mode”. Mistral uses “a single reasoning effort during SFT” and gets effort levels later, from RL. Moonshot trains its effort levels in RL experts and collects their trajectories for SFT and distillation.

How much, and how fast

Volumes are printed in four units, and the table gives each in its own.

ModelSFT volume as printedUnitEasily confused with
LFM3-1.7B1.75Ttokens, kind not stated
Kolibriabout 268B per run, two runs averagedpacked tokensthe summary’s 537B, which matches two runs; the 335.8B pool
Laguna M.1, XS.2 (Poolside)3 epochs of about 40B, stopped earlytokensat most 120B seen (computed)
Nemotron 3 Ultra204,800 + 19,200 packed sequencessequencesat most 60.4B + 9.9B token positions (computed); the 40B of its STEM teacher’s own SFT
LFM2 (Liquid)5.39M (dense) or 9.24M (8B-A1B), 3 epochssamples
MiMo-V2.6“a short supervised fine-tuning (SFT) stage”77.4B / 27.2B loss, which is a Qwen3.5-9B student’s SFT
Inkling“a small fraction of compute”compute

Learning rates sit below pre-training’s peak and usually decay: Kolibri uses “Half of pre-training in every group”, Nemotron 3 Ultra starts at 6% of its pre-training peak, and Laguna at a fiftieth of XS.2’s. MiMo-V2-Flash’s SFT starts at 5×10⁻⁵, under its pre-training peak of 3.2×10⁻⁴ but five times the rate its pre-training ended at. The values don’t compare across labs, since Kolibri and Laguna train matrices with Muon, whose learning rate is scaled differently from AdamW’s (O1).

Two labs say how they chose: Mistral sets SFT hyperparameters “with a scaling ladder instead of searching at ML4 scale”, and Poolside stops early “based on evaluation scores”, without saying which; if those are the benchmarks it reports, they carry the same selection optimism as Kolibri’s mix choice. Aleph Alpha adds an observation from early experiments: learning rates and step counts for SFT “can be set independently” of RL, while “behaviours that RL should reinforce need to be introduced in SFT”.

Averaging the runs

Kolibri’s released SFT model is an average of two runs. Aleph Alpha averaged every pair of its three full-length models, parameter by parameter with equal weights, and shipped the average of C1 and C2.

Figure 5.Kolibri's three full-length SFT runs (diagonal) and the equal-weight average of each pair (below), on the mean of 12 benchmarks. Every average scores above both parents; by our application of the report's noise rule, the gain is well outside the noise for the released average of C1 and C2 and inside it for B0 with C1. The mixes were chosen on the same benchmarks, so Aleph Alpha calls these scores optimistic.

The average of C1 and C2 scored 63.4 ± 1.8% against 58.9 ± 1.8% for C1, the best single model, and 30.0 ± 2.0% for the base before SFT. “Every soup scores above both of its models on the mean”, by 4.5, 3.4 and 2.0 points; with half-widths near 1.8, the report’s rule for its ablations (App. D.2.2) would need about 2.5, so by our application of it the smallest gain is within noise; the report itself doesn’t say. On AIME 2026 the soup lands below C1, also within noise. There is one run per mix, and Aleph Alpha adds the caveat itself: “Selection and these results use the same benchmarks, so the scores of the selected soup are optimistic.”

Averaging works “because both models are fine-tuned from the same base checkpoint and stay close to each other in weight space”. This is the model soup (Wortsman et al., 2022), built on the observation that models fine-tuned from one pre-trained start lie in one basin of the loss (Neyshabur et al., 2020), so the loss along the line between them stays low; two models trained from scratch generally don’t. That explains why the average isn’t worse than its parents. That it beats both is an empirical result, also seen in the soup paper, which varied hyperparameters on one dataset where Kolibri’s parents saw different mixes.

What SFT does to RL

NVIDIA’s Nemotron 3.5 report warns that “very long SFT stages can reduce how much the model later gains from RL.” The companion paper (Singhal et al., 2026), which argues for moving some of what SFT teaches into mid-training, cites three papers for it and adds “we have observed this in our own training as well”, with no data; its own RL run is 20 steps on one seed, and the authors write that it does “not show that midtraining increases what RL can add”. Aleph Alpha reports the opposite impression, also without a measurement: “we have not seen checkpoints collapse into a hard-to-train state through SFT.”

Too little SFT has a cost as well. Poolside writes that without instruction-following supervision in SFT, “RL fine-tuning tends to exhibit catastrophic forgetting of these behaviors”. Z.ai ends GLM-5’s pipeline with distillation “to swiftly recover the skills acquired in earlier SFT and RL stages”. RL’s gains over a fine-tuned model vary widely: over 11 benchmarks, Nemotron 3 Ultra’s RL changed scores by −0.6 to +41.5 points, and over 13, Kolibri’s by +1.5 to +36.0 (single checkpoints). No lab in the corpus reports RL gains after SFT runs of different lengths.

Comparison

MethodWhat it changesWhat it costsMeasured whereLabs
Assistant-only losswhich tokens are targetsfewer loss tokens per stepNVIDIA, tool responses, in mid-training (single run)Aleph Alpha, NVIDIA describe it in full
Masking wrong steps in contextrecovery without the errora rule or a labeller that decides “wrong”Slinko et al.: +1.1 points (0.4 to 1.8) across seeds, failed trajectories only, one benchmarkAleph Alpha, Z.ai; NVIDIA for chat turns
Teacher-generated datawhere demonstrations come fromthe teachers’ biases, habits and licencesDeepSeek V3.2: near the specialists (no numbers)9 say where it comes from, 3 name outside teachers
Rejection sampling and filterswhich demonstrations are keptgeneration and judging computeKolibri: decontamination, no effectZ.ai and NVIDIA by name; Aleph Alpha, Poolside, Liquid filter
Merge-based mix searchhow shares are foundspecialist runs; a ranking that may failKolibri: no usable ranking at 20 clusters (6 points)Aleph Alpha
Curriculumorder of the dataa difficulty estimatenoneLiquid
Post-hoc effort labelscontrol of reasoning lengththresholds per group, a difficulty graderKolibri: lengths ordered on 23 of 24 benchmarksAleph Alpha
Soup of runsthe final weightsone extra full runKolibri: +2.0 to +4.5 over the better parent, chosen on the same benchmarksAleph Alpha

Where the labs sit

Quoted from the reports, cards and blogs, with blanks where we searched and found nothing.

LabRole statedDataStagesVolumeLearning rateLoss and merging
Aleph Alpha“a cold start for RL”GLM-5.2/5.3, Qwen3.8-27B; RL-environment rollouts1~268B packed per runhalf of pre-training, warm-up, constant, then decayassistant only; wrong calls masked; soup of two runs
NVIDIA“foundational capabilities”many named teachers; rejection samplingUltra 2, 3.5 Super 2, Lightning 1Ultra 224,000 packed sequencesUltra 1.5×10⁻⁵ → 1×10⁻⁶</think> masked in truncated traces; chat final turn only
Poolsideagentic coding; instruction following so RL rewards aren’t zerounnamed teacher; in-house tasks1, after a chat-format “mid-training”3 × ~40B, early stop1×10⁻⁵ → 2×10⁻⁷earlier reasoning kept in context
Thinking Machines“bootstrap” before RLopen models incl. Kimi K2.51“small fraction of compute”
Mistral AImid-training, then SFTfrom a scaling laddersingle effort level
Liquid AIteach the chat template67 to 79 sources; 12-model curriculum1LFM3 1.75T; LFM2 5.39M samples3×10⁻⁵ → 1×10⁻⁷merges after alignment
Xiaomi (MiMo-V2-Flash unless noted)“activating the model’s latent capabilities”own specialists1; V2.5 extends 32K to 1M within it“millions of training samples”5×10⁻⁵ → 5×10⁻⁶
Moonshot AI“a high-quality cold-start policy”prior Kimi specialists, RL experts1earlier reasoning kept
Z.ai“interleaved thinking modes”expert RL, rejection sampling1, to 202,752 tokenserroneous segments masked
Qwen(teachers only)specialists from the same base1
DeepSeekper-specialist SFT, then RL or distillationown RL specialistsper specialist
Tencent“joint optimization of SFT and RL”
Google DeepMind“Instruction Tuning”IT model ends turns with <turn|>

MiniMax, OpenAI and StepFun describe no SFT stage in the corpus. OpenAI’s gpt-oss card describes post-training as RL, and StepFun’s technical blog for Step 5 is linked but not in the corpus.

Easy to get wrong

  • Counting tutorials. A card’s “quick single-GPU fine-tuning” (LFM3) is advice to users, not the lab’s recipe.
  • Training on a different template from the one inference renders. Whether earlier reasoning is kept, and which token ends a turn, must match: Gemma 4’s pre-trained and instruction-tuned models end with different tokens, and fine-tuning either “requires adding their respective end tokens”.
  • Reading C1 as “less math helps”. C2 had even less math and scored lowest; C1 dropped easy math with long traces.
  • Reading “specialist” one way. In DeepSeek, Moonshot, Xiaomi and Qwen it is a teacher; in MergeMix it is a scoring device that never teaches.
  • Treating the warning about long SFT as a result. It is a clause with citations and an unquantified “we have observed this”.

Check yourself

1. A model solves a new agentic task 5% of the time. With groups of 8 attempts and 0/1 rewards, what share of such prompts gives a group-relative update any gradient? How large must the success rate be for half of them to?

1−0.958−0.058≈0.3371 - 0.95^8 - 0.05^8 \approx 0.337, so about a third. For a half, (1−q)8≈0.5(1-q)^8 \approx 0.5 gives q=1−0.51/8≈8.3%q = 1 - 0.5^{1/8} \approx 8.3\% (the all-success term is negligible there).

2. Kolibri’s long-context SFT data holds 49.8B tokens and 0.9B loss tokens. Why is the share so small, and what does it mean for the mix shares in the figure, which are shares of trained tokens?

The documents, around 200K tokens each, are context, and only the answer carries loss: 0.9 / 49.8 = 1.8%. With the loss normalised over the batch, a domain’s share of the gradient is its share of loss tokens, so a token share overstates a domain whose loss share is low. Long context is 14.9% of C1’s tokens, but if C1’s other domains have the pool’s average loss share (49.5%), it carries about 0.6% of C1’s loss tokens (computed).

3. Why mask a wrong tool call rather than drop the trajectory or train on all of it?

Dropping it loses the recovery that follows, so the model never sees a state after an error. Training on it teaches the model to make the error. Masking keeps the state in the input and puts the loss only on the correction.

4. Why can Kolibri average C1 and C2 at all, when averaging two models trained from scratch generally fails? Does the same argument explain why the average beat both?

Both were fine-tuned from the same base and stayed close in weight space, where the loss along the line between them stays low, so their midpoint isn’t worse than they are. Models trained from scratch end in different basins, and their midpoint usually has high loss. That the average scored above both parents is an empirical result, not something the basin argument guarantees.

In the map

Methods

Taxonomy nodes

Further reading

  • Kolibri technical report, §3.1 (Aleph Alpha): the pool, masking rules, MergeMix at SFT, effort labels and the soup, with every table. https://aleph-alpha.com/downloads/tech-report.pdf
  • Nemotron 3 Ultra technical report, §3.1, and Nemotron 3.5 report, §3.1 and §3.9 (NVIDIA): per-domain teachers, two-stage SFT, effort data and </think> masking.
  • Laguna M.1/XS.2 technical report, §4 (Poolside): the SFT mix, preserved reasoning and its parser fix, and the instruction-following failure.
  • GLM-5, arXiv:2602.15763, §3.1: masking erroneous segments, difficulty filtering, SFT to 202,752 tokens.
  • LFM2, arXiv:2511.23404, §4: the curriculum by ensemble solve rate and the SFT filtering pipeline.
  • Slinko et al., Step Rejection Fine-Tuning, arXiv:2605.10674: the one controlled test of masking harmful steps.
  • Wang et al., MergeMix, arXiv:2601.17858: the method and its first-order argument, for mid-training.
  • Wortsman et al., Model Soups, arXiv:2203.05482: why fine-tunes from one start can be averaged, and when they can’t.
  • Singhal et al., arXiv:2610.22019: NVIDIA’s case for moving learning out of SFT.
↑