Kolibri’s supervised fine-tuning runs 4,000 steps of 256 packed sequences, each 256K tokens long: about 268B tokens per run. LFM3’s card lists 1.75T tokens of SFT for a 1.7B model, after 22T of pre-training. Thinking Machines says only that the SFT before Inkling’s reinforcement learning “accounts for a small fraction of compute”. The first two are more than six-fold apart, and only Kolibri’s says what kind of token it counts; the third gives no number. Of the 16 labs on this site, 13 describe an SFT stage, two of them only in passing, and four say it is there so that RL has something to work on.
A base model continues documents. Ask it a question and it may answer, or write the next question of an exam paper. Reinforcement learning (RL) can improve a behaviour only if the model already produces it some of the time, because a reward that is never earned teaches nothing. Poolside describes what happens otherwise in the report on its Laguna models: without instruction-following data in fine-tuning, “over half of agentic tasks receive zero reward because the model violates system-message constraints before any coding work is evaluated.” SFT closes that gap by training on demonstrations of the behaviour, with the same cross-entropy loss as pre-training. Aleph Alpha calls the stage “a cold start for RL”, Moonshot “a high-quality cold-start policy”.
Background
The primer sets out the stages. Pre-training teaches a base model to predict text, and mid-training and long-context extension (S2) shift its data and lengthen its sequences. Supervised fine-tuning (SFT) then trains it on demonstrations: conversations in which the assistant’s turns show the behaviour wanted, written by people or, far more often now, by other models. The model that writes them is a teacher and the model trained on them the student (D3 covers how such data is generated). In classical terms SFT is behaviour cloning, and training on a teacher’s generated outputs is sequence-level knowledge distillation (Kim and Rush, 2016, who used the teacher’s beam-search output). The data is off-policy: written by another model, not sampled from the student.
RL follows (S4, S5). An RL environment is a task with a program that scores attempts at it, such as a repository and its tests or a search tool and a reference answer. The model produces its own attempts, called rollouts, each gets a reward, and the update makes high-reward attempts more likely. A cold start is a model good enough at a behaviour for RL to begin improving it.
A demonstration reaches the model through a chat template: special tokens that mark where the system prompt, each user turn, each assistant turn and each tool result begin and end. Reasoning models also write a reasoning trace between markers such as <think> and </think> before the answer. An agent calls tools (a shell, a search engine, a file editor) and reads their output, and one recorded episode of that is a trajectory.
Packed tokens fill the training sequences, several conversations to a sequence, with attention masked at each boundary so no conversation sees another (Kolibri and NVIDIA both say so). Loss tokens are the ones the loss is computed on, a minority, as the next sections show.
Nothing about SFT is in a model’s configuration or code, so we read technical reports first, then model cards and official blogs. We count labs, not models. A blank means we searched and found nothing, which doesn’t mean the lab didn’t do it. Kolibri is the worked example because Aleph Alpha’s report describes its SFT in more detail than any other. That makes it the best-documented recipe, not necessarily a typical one.
The article is about the labs’ own training. Model cards also tell users how to adapt a released model, and 37 of the 93 corpus documents that mention fine-tuning carry only that kind of text, licence terms, demos, safety tests or unrelated uses of the word; counting them would add MiniMax and OpenAI, which describe no SFT stage.
Why RL needs a cold start
Many of the labs here train with group-relative RL updates (S5). For each prompt the model makes attempts, and each attempt’s advantage is its reward minus the group’s mean. The update sums each attempt’s advantage times the gradient of its log-probability, so if all attempts score the same, all zero or all one, the prompt contributes no gradient. With rewards of 0 or 1 and a per-attempt success rate , the chance that a prompt teaches anything is
At and that is 7.7%; at it is 83% (computed). Over prompts of mixed difficulty the share is lower than the formula gives at their mean: half at and half at give 49%, not 83%. Poolside’s zero-reward tasks are such a subpopulation, with near zero. Aleph Alpha states the requirement in words: SFT “establishes desired behaviours that the model must produce at a sufficient rate for RL to become effective in improving them.”
Only one lab in the corpus reports RL with and without SFT on the same base. For MiMo-7B, a 7B model from 2025, Xiaomi’s repository lists AIME 2025 (a 30-problem math competition, averaged over 32 attempts) at 24.3 for the base, 46.3 after RL from the base, 44.3 after SFT alone and 55.4 after SFT then RL. That shows SFT then RL ahead of RL alone, by 9.1 points, but not why. The base already solves about a quarter of the AIME problems, so RL from the base was not starting from zero reward everywhere, and the gap fits SFT passing on its teacher’s knowledge as well as it fits a cold start. Each arm is one run, and whether the two RL runs used the same compute isn’t stated.
Prompting can raise too: DeepSeek’s “cold-start” for reasoning with tools in V3.2 is a set of system prompts under which the model “is occasionally able to generate the desired trajectories”. And SFT doesn’t always raise far enough: Aleph Alpha found many agentic environments “difficult to solve from the SFT checkpoint” and gave the model hints during RL instead.
SFT’s cost is that everything it teaches is imitation: RL starts from the teachers’ style, length and mistakes.
What the loss is on
The mask
SFT minimises the same next-token loss as pre-training, but only on some tokens. With on a token that carries loss and elsewhere, summed over every token in the batch,
The masked tokens are still in the input, so the model conditions on them, but they are not targets. This loss mask is separate from the attention mask that keeps packed conversations apart. Kolibri’s rule, which NVIDIA states too: “We train with cross-entropy loss only on the tokens the model produces at inference: the reasoning, answer and tool calls of each assistant turn. User turns, tool responses and the special tokens of the chat format are masked out.” Kolibri normalises over the whole batch, so within a batch every supervised token has the same weight. Averaging per conversation first would give each token of a two-line reply more weight than each token of a 20,000-token reasoning trace.
The template has to match what inference renders, and labs differ on one point: whether earlier turns keep their reasoning. Kolibri’s template drops it, as gpt-oss’s format does. To train a later turn the context must show earlier turns without their reasoning, but to train an earlier turn its reasoning must be there, so one sequence can’t supervise every turn. Aleph Alpha splits each conversation into one sample per user turn, “with loss on the last turn only”, 2.27 samples per conversation on average. Laguna, Kimi K3 and GLM-5’s “preserved” mode keep earlier reasoning, so one sample can supervise every turn, but the serving stack has to agree: Poolside found vLLM’s reasoning parser misreading the earlier </think> tags in Laguna’s history, and fixed it upstream.
Loss tokens end up a minority. Kolibri’s SFT pool holds 335.8B tokens and 142.4B loss tokens (42.4%). In its long-context data the share is 0.9B of 49.8B, since a 200K-token document is context and the answer is short. When Xiaomi fine-tuned a Qwen3.5-9B student on MiMo-generated data, the mix held 77.4B tokens, 27.2B of them loss tokens (35%, computed).
Keeping mistakes in context
An agent’s trajectory often contains a wrong step: a call to a tool that doesn’t exist, a malformed argument, a command repeated after it failed. Dropping the trajectory loses the recovery that follows, and training on all of it teaches the error. Kolibri masks the loss on “clearly wrong tool calls: calls whose tool response reports an unknown tool or invalid arguments, and exact repeats of the previous call that never receive a different response. These turns stay in the context, so the model learns the correction that follows without being trained to produce the error”. GLM-5 does the same for coding and agent data: “Erroneous segments within trajectories are retained but masked out in the loss function”. Nemotron 3 Ultra applies the idea to chat, building conversations whose earlier assistant turns “may contain suboptimal responses rather than the best-ranked candidates” and training “only on the final assistant response”.
losscontext onlytoy token counts
Loss on 820 of 2,270 tokens (36.1%).
In imitation learning, a policy trained only on an expert’s states never sees where its own mistakes lead, and its errors compound over a long episode (Ross and Bagnell, 2010). DAgger (Ross et al., 2011) has an expert label the states the learner visits; DART (Laskey et al., 2017) perturbs the expert’s rollouts so they contain recoveries. A wrong call kept in context is closer to DART, since both the error and the recovery are the teacher’s (our reading). The DAgger step comes later in several pipelines, as on-policy distillation, where the student generates and a teacher scores its tokens (S6). NVIDIA found that a very light SFT on data from the teacher’s training distribution before that stage raised Ultra’s GDPVal score after distillation from 35.3 to 46.7 and BrowseComp from 33.0 to 44.4, while HLE barely moved (26.3 to 26.7); no repeats are reported.
One outside paper tests masking errors directly. Step rejection fine-tuning (Slinko et al., 2026), which Kolibri cites, has a second model read each software-engineering trajectory and label every step good, harmful or unnecessary, then masks the harmful ones. For a Qwen2.5-Coder-32B student on SWE-bench Verified (real GitHub issues, scored by whether the patch passes the tests), adding failed trajectories with their harmful steps masked raised resolved issues from 30.9% (successful trajectories only) to 32.2%. Adding the same failed trajectories unmasked lowered it to 28.5%. Each is one trained model, scored over seven evaluation runs. The paper’s one test across training seeds, on the failed trajectories alone, gives masking 1.1 points (95% interval 0.4 to 1.8). No arm matches Kolibri’s case of passing trajectories with errors masked; the nearest, masking harmful steps in successful trajectories too, scored 29.4. Aleph Alpha reports no result for its loss masking alone.
Two other masks are tied to specific behaviours. To let users cap how long the model thinks, NVIDIA cuts reasoning traces at random lengths, closes them with </think> and keeps the answer, masking </think> “to prepare the model for inference-time budget control”: at inference the serving code inserts it when the budget runs out, and the mask keeps the model from learning to end its own reasoning early (our reading). Tool responses are masked by both labs that describe their full loss, Aleph Alpha and NVIDIA. NVIDIA’s one measurement of that is from mid-training, where training on them “lowered SWE-bench Verified pass@1 by about 10 points” (solved on the first attempt) in a single research run.
Masking costs loss tokens per step and needs a rule for what counts as wrong. Kolibri’s rule is mechanical; Slinko et al.’s labeller agreed with hand labels on 59% of steps and found 28% of the steps people called harmful.
Where the demonstrations come from
Most SFT data is written by models. Nine labs say where theirs comes from: three name the outside models that wrote it, five say their own specialist or expert models did, and Poolside names neither:
| Lab | Teachers named for SFT data | Own models used |
|---|---|---|
| NVIDIA, Nemotron 3 Ultra | by domain: GPT-OSS-120B, DeepSeek-V3.2 and V3.2-Speciale, MiniMax M2.5, GLM-5, Qwen3-Coder-480B and others | |
| NVIDIA, Nemotron 3.5 | Kimi K3, Kimi K2.6, DeepSeek V4 Pro and Flash, GLM-5.2, Qwen3.5 | |
| Aleph Alpha, Kolibri | GLM-5.2, GLM-5.3, Qwen3.8-27B | a smaller 8B model trained in RL environments |
| Thinking Machines, Inkling | “open-weights models including Kimi K2.5” | |
| Poolside, Laguna | an unnamed open-source teacher for the 30% of tokens that are agentic coding without reasoning; the 45% with reasoning, source unstated | in-house synthetic tasks |
| Moonshot, Kimi K3 | “domain-specialized models from the prior Kimi series”, RL experts | |
| Xiaomi, MiMo-V2-Flash | “in-house domain-specialized model checkpoints” | |
| Z.ai, GLM-5 | expert RL models, rejection sampling | |
| DeepSeek V3.2 and V4; Qwen3.5-Omni | specialists fine-tuned and RL-trained from the same base (V4 and Omni say SFT, then RL); V4 says its final model learns from them by on-policy distillation |
A specialist in the last rows is a teacher: a copy of the base trained hard on one domain, by SFT or RL (an RL expert), whose outputs then train the general model. DeepSeek calls this “Specialist Distillation” and reports that the general model trained on it lands “only marginally below” the specialists, with the gap “effectively eliminated through subsequent RL” (no numbers given). The word means something else in Kolibri’s mix search below.
Rejection sampling draws several answers per prompt and keeps the ones that pass a check: a matching reference answer, passing tests, a judge’s approval. Fine-tuning on what survives is rejection-sampling fine-tuning (RFT, Yuan et al., 2023), a relative of STaR’s train-on-your-own-correct-answers loop. It fits the teacher’s outputs conditioned on passing, so a check on the final answer also keeps right answers reached by wrong reasoning. Z.ai and NVIDIA name it, and most of the filters in the next section do the same thing without the name.
Some SFT data comes from RL environments, rolled out by another model. Kolibri’s retrieval data are GLM-5.2’s rollouts in the retrieval environments, and its hallucination data come from an 8B model trained in its Merlin–Arthur hallucination environments: “Both are rollouts of RL environments and provide a cold start for RL.” Moonshot trains RL experts per domain and effort level and collects their trajectories “for supervised fine-tuning and multi-teacher on-policy distillation”, without saying which SFT run uses them.
Teachers have costs. Aleph Alpha writes that “several of our teacher models were built in China, so they carry known political biases on sensitive topics”, and filters for them. Licences constrain the data too: for Ultra, NVIDIA kept only the OpenResearcher search trajectories whose source licence cleared commercial use, about 21.7K.
Which data is kept
Filters are where most labs say the recipe is. Nemotron 3.5 removes “reasoning responses shorter than 5,000 tokens”. GLM-5 keeps, for math and science, “only problems that are challenging for the GLM-4.7 model”. Poolside removes from its math data “overly easy samples based on solve rates from open-weight LLMs”. Kolibri keeps a generated German answer only if it matches its seed’s reference and a software-engineering trajectory only if it passes the tests. None of them reports what a filter did to the scores. Kolibri also reports a filter it chose not to apply: chat answers go unfiltered, “as it adds compute without a known benefit”, and its benchmark decontamination removed 0.005% of rows with “no measurable difference”.
Liquid orders LFM2’s SFT data as a curriculum (Bengio et al., 2009), easy items first. Difficulty is the mean solve rate of 12 models, four of them Liquid’s own, and a model trained on prompt features predicts the order. Liquid reports no run without the curriculum, and if the order spans the whole run (LFM2 trains for three epochs and doesn’t say), an easy-to-hard order also puts the hard items where the learning rate decays.
Choosing the mix
A mix is a set of shares, one per cluster of the data pool, and finding good shares is a search over training runs (D2). Fitting shares from trained candidates needs at least runs for clusters (Chen et al., 2026, as Aleph Alpha cites them): with 20 clusters, 63 SFT runs of 16.78B tokens, 1,057B in all.
MergeMix (Wang et al., 2026, a method for mid-training) replaces most of those runs with weight averages. Fine-tune one specialist per cluster from the same base, briefly, then for a candidate mix with shares that sum to one form
and score the merged model on benchmarks as a stand-in for a model trained on that mix. Here a specialist is only a scoring device. The argument: a short run on cluster moves the weights by about times the sum of its gradients, and a mixed run’s gradients are the -weighted sum, so for plain gradient descent the two moves agree to first order, differing in curvature terms. Two caveats apply to Kolibri: Muon, its optimizer for matrices, doesn’t update linearly in the gradient, nor do the Adam-type optimizers of its other parameters, and to first order the score is linear in the shares, so whatever makes a mixture beat its best specialist lies in the terms the argument drops.
Kolibri trained 20 specialists on 4.50B tokens each, 90B in place of 1,057B, scored 69 merges, and trained eight weightings for real at 16.8B tokens. The merges didn’t predict the trained runs. Over the six weightings with both scores the Spearman correlation is −0.09, and “The best merge trains worst, at 50.0 %.” Six points can’t show the absence of a correlation, but the 20-specialist merge gave no usable ranking. An earlier check on the smaller proxy model, with two or three specialists at 20B tokens each, had ranked better (0.77 to 0.94 on two benchmarks, 0.26 to 0.32 on MATH-500, over four to six weightings). Aleph Alpha can’t say which change broke it, more clusters or shorter specialist runs; the check also ran on a smaller model.
Aleph Alpha then chose with full runs: four searched candidates and the baseline B0 at 168B tokens each, and three of those continued to 268B. Those three are B0, which samples the pool roughly in proportion with German upweighted, and C1 and C2, two searched mixes.
released runs (C1, C2)trained, not releasedspecialists, used only to score merges
1,366B in all; 537B released; 829B choosing.
chatmathcodesciencetool usesafety and alignmentlong context
Bars: domains in this order, left to right; shares of 4% or more printed. Whiskers: the report's 95% interval, printed only for base and B0. Dashed line: the model before SFT.
C1 scored 94.0% on AIME 2026 with math at 4.9% of its tokens, against 69.4 ± 9.3 for B0 with 18.3%. The report doesn’t define this section’s ±. For its fine-tuning ablations it gives 95% intervals from runs that differ only in seed, and counts two runs as different when their scores are about 1.4 half-widths apart; applied here (our step), C1’s 24.6-point lead holds. B0 is within noise of the model before SFT (74.6 ± 10.7) and ends 7.3% of its AIME 2026 answers in repetition loops or at the context limit, against 2.1% for the base and none for C1. What C1 changed was which math it kept: it “drops or downsamples in particular the math datasets with on average easier problems and longer reasoning traces.” Aleph Alpha adds: “Other causes are possible, as C1 and B0 also differ in other domains.” These are single runs on 30 problems, where one problem is 3.3 points, and C1 was picked on a mean of benchmarks that includes AIME 2026.
Stages, lengths and reasoning effort
Of the labs that describe the structure, only NVIDIA names two SFT stages for one model. It runs two for Nemotron 3 Ultra, the second extending packed sequences from 294,912 to 515,000 tokens with long-context data up to 512K. Nemotron 3.5 Super’s base was trained on text only, and vision first enters at SFT, in two stages “because learning vision and text capabilities simultaneously is challenging”. Nemotron 3.5 Lightning, text only, has one. GLM-5 “extends the maximum context length to 202,752 tokens during SFT”. Poolside’s word differs: its roughly 60B-token “mid-training” is instruction data spanning chat, reasoning and agentic coding, at SFT’s learning rate, which other labs would call SFT, so by that reading Poolside’s comes in two parts too.
Many reasoning models let the user choose how long they think. A line in the system prompt, such as Kolibri’s “Reasoning effort is set to low. Think briefly through only the essential steps in the user’s language, then proceed directly to the answer.”, selects an effort level (low, medium or high), and the model is expected to reason for a shorter or longer time to match. In Kolibri, SFT is where the model first learns to obey that line; Mistral leaves it to RL (below).
Kolibri’s traces have no effort label, so Aleph Alpha assigns one after the fact. A single length threshold would fail: median reasoning length per dataset runs “from 27 to 90 223 tokens”, and hard prompts reach “up to 31.3 times” the length of easy ones. So each reference group , one dataset at one difficulty (graded by a small model distilled from GLM-5.2), gets its own 30th and 70th percentiles of reasoning length, and , and a trace of length is placed on by
The percentiles land on 0.3 and 0.7, the edges between low, medium and high. Each label has a centre (0.15, 0.5, 0.85) and a width (0.3, 0.4, 0.3), and is drawn with probability
At each edge the two neighbouring labels are equally likely. The draw is seeded per row so the label doesn’t switch at one exact token count, “which the model could otherwise learn as a length at which to stop.”
lowmediumhigh30th and 70th percentiles in use
Toy groups: 60 traces each, log-normal lengths with invented medians. Each dot draws its label once, with a fixed seed. Shares below are expected shares under Equation 12.
chat: low 30% · medium 40% · high 30%
code: low 30% · medium 40% · high 30%
math proofs: low 30% · medium 40% · high 30%
Labelling after the fact has one risk: the model might never need the label, since a hard prompt has a long trace whatever the label says. What makes it read the label is repetition. 9.5% of Kolibri’s opening prompts appear twice with reasoning, and Aleph Alpha estimates that about 37.3% of the pairs receive two different labels (the report doesn’t say whether of all pairs or of those whose lengths differ two-fold); for those, the label is the only difference in the input. In the released SFT model the median reasoning length orders low, medium and high correctly on 23 of 24 benchmarks. Scores move less: the mean over 14 benchmarks is 64.9 at low and 67.8 at high, with a paired standard error of 0.8 points for high against low, and medium to high adds 0.7 points.
NVIDIA takes effort from a teacher’s mode instead, GPT-OSS-120B’s medium mode for Ultra, and mode names don’t carry over: Nemotron 3.5 Super’s medium effort is taught “with data generated by Kimi K3’s low-effort mode”. Mistral uses “a single reasoning effort during SFT” and gets effort levels later, from RL. Moonshot trains its effort levels in RL experts and collects their trajectories for SFT and distillation.
How much, and how fast
Volumes are printed in four units, and the table gives each in its own.
| Model | SFT volume as printed | Unit | Easily confused with |
|---|---|---|---|
| LFM3-1.7B | 1.75T | tokens, kind not stated | |
| Kolibri | about 268B per run, two runs averaged | packed tokens | the summary’s 537B, which matches two runs; the 335.8B pool |
| Laguna M.1, XS.2 (Poolside) | 3 epochs of about 40B, stopped early | tokens | at most 120B seen (computed) |
| Nemotron 3 Ultra | 204,800 + 19,200 packed sequences | sequences | at most 60.4B + 9.9B token positions (computed); the 40B of its STEM teacher’s own SFT |
| LFM2 (Liquid) | 5.39M (dense) or 9.24M (8B-A1B), 3 epochs | samples | |
| MiMo-V2.6 | “a short supervised fine-tuning (SFT) stage” | 77.4B / 27.2B loss, which is a Qwen3.5-9B student’s SFT | |
| Inkling | “a small fraction of compute” | compute |
Learning rates sit below pre-training’s peak and usually decay: Kolibri uses “Half of pre-training in every group”, Nemotron 3 Ultra starts at 6% of its pre-training peak, and Laguna at a fiftieth of XS.2’s. MiMo-V2-Flash’s SFT starts at 5×10⁻⁵, under its pre-training peak of 3.2×10⁻⁴ but five times the rate its pre-training ended at. The values don’t compare across labs, since Kolibri and Laguna train matrices with Muon, whose learning rate is scaled differently from AdamW’s (O1).
Two labs say how they chose: Mistral sets SFT hyperparameters “with a scaling ladder instead of searching at ML4 scale”, and Poolside stops early “based on evaluation scores”, without saying which; if those are the benchmarks it reports, they carry the same selection optimism as Kolibri’s mix choice. Aleph Alpha adds an observation from early experiments: learning rates and step counts for SFT “can be set independently” of RL, while “behaviours that RL should reinforce need to be introduced in SFT”.
Averaging the runs
Kolibri’s released SFT model is an average of two runs. Aleph Alpha averaged every pair of its three full-length models, parameter by parameter with equal weights, and shipped the average of C1 and C2.
The average of C1 and C2 scored 63.4 ± 1.8% against 58.9 ± 1.8% for C1, the best single model, and 30.0 ± 2.0% for the base before SFT. “Every soup scores above both of its models on the mean”, by 4.5, 3.4 and 2.0 points; with half-widths near 1.8, the report’s rule for its ablations (App. D.2.2) would need about 2.5, so by our application of it the smallest gain is within noise; the report itself doesn’t say. On AIME 2026 the soup lands below C1, also within noise. There is one run per mix, and Aleph Alpha adds the caveat itself: “Selection and these results use the same benchmarks, so the scores of the selected soup are optimistic.”
Averaging works “because both models are fine-tuned from the same base checkpoint and stay close to each other in weight space”. This is the model soup (Wortsman et al., 2022), built on the observation that models fine-tuned from one pre-trained start lie in one basin of the loss (Neyshabur et al., 2020), so the loss along the line between them stays low; two models trained from scratch generally don’t. That explains why the average isn’t worse than its parents. That it beats both is an empirical result, also seen in the soup paper, which varied hyperparameters on one dataset where Kolibri’s parents saw different mixes.
What SFT does to RL
NVIDIA’s Nemotron 3.5 report warns that “very long SFT stages can reduce how much the model later gains from RL.” The companion paper (Singhal et al., 2026), which argues for moving some of what SFT teaches into mid-training, cites three papers for it and adds “we have observed this in our own training as well”, with no data; its own RL run is 20 steps on one seed, and the authors write that it does “not show that midtraining increases what RL can add”. Aleph Alpha reports the opposite impression, also without a measurement: “we have not seen checkpoints collapse into a hard-to-train state through SFT.”
Too little SFT has a cost as well. Poolside writes that without instruction-following supervision in SFT, “RL fine-tuning tends to exhibit catastrophic forgetting of these behaviors”. Z.ai ends GLM-5’s pipeline with distillation “to swiftly recover the skills acquired in earlier SFT and RL stages”. RL’s gains over a fine-tuned model vary widely: over 11 benchmarks, Nemotron 3 Ultra’s RL changed scores by −0.6 to +41.5 points, and over 13, Kolibri’s by +1.5 to +36.0 (single checkpoints). No lab in the corpus reports RL gains after SFT runs of different lengths.
Comparison
| Method | What it changes | What it costs | Measured where | Labs |
|---|---|---|---|---|
| Assistant-only loss | which tokens are targets | fewer loss tokens per step | NVIDIA, tool responses, in mid-training (single run) | Aleph Alpha, NVIDIA describe it in full |
| Masking wrong steps in context | recovery without the error | a rule or a labeller that decides “wrong” | Slinko et al.: +1.1 points (0.4 to 1.8) across seeds, failed trajectories only, one benchmark | Aleph Alpha, Z.ai; NVIDIA for chat turns |
| Teacher-generated data | where demonstrations come from | the teachers’ biases, habits and licences | DeepSeek V3.2: near the specialists (no numbers) | 9 say where it comes from, 3 name outside teachers |
| Rejection sampling and filters | which demonstrations are kept | generation and judging compute | Kolibri: decontamination, no effect | Z.ai and NVIDIA by name; Aleph Alpha, Poolside, Liquid filter |
| Merge-based mix search | how shares are found | specialist runs; a ranking that may fail | Kolibri: no usable ranking at 20 clusters (6 points) | Aleph Alpha |
| Curriculum | order of the data | a difficulty estimate | none | Liquid |
| Post-hoc effort labels | control of reasoning length | thresholds per group, a difficulty grader | Kolibri: lengths ordered on 23 of 24 benchmarks | Aleph Alpha |
| Soup of runs | the final weights | one extra full run | Kolibri: +2.0 to +4.5 over the better parent, chosen on the same benchmarks | Aleph Alpha |
Where the labs sit
Quoted from the reports, cards and blogs, with blanks where we searched and found nothing.
| Lab | Role stated | Data | Stages | Volume | Learning rate | Loss and merging |
|---|---|---|---|---|---|---|
| Aleph Alpha | “a cold start for RL” | GLM-5.2/5.3, Qwen3.8-27B; RL-environment rollouts | 1 | ~268B packed per run | half of pre-training, warm-up, constant, then decay | assistant only; wrong calls masked; soup of two runs |
| NVIDIA | “foundational capabilities” | many named teachers; rejection sampling | Ultra 2, 3.5 Super 2, Lightning 1 | Ultra 224,000 packed sequences | Ultra 1.5×10⁻⁵ → 1×10⁻⁶ | </think> masked in truncated traces; chat final turn only |
| Poolside | agentic coding; instruction following so RL rewards aren’t zero | unnamed teacher; in-house tasks | 1, after a chat-format “mid-training” | 3 × ~40B, early stop | 1×10⁻⁵ → 2×10⁻⁷ | earlier reasoning kept in context |
| Thinking Machines | “bootstrap” before RL | open models incl. Kimi K2.5 | 1 | “small fraction of compute” | ||
| Mistral AI | mid-training, then SFT | from a scaling ladder | single effort level | |||
| Liquid AI | teach the chat template | 67 to 79 sources; 12-model curriculum | 1 | LFM3 1.75T; LFM2 5.39M samples | 3×10⁻⁵ → 1×10⁻⁷ | merges after alignment |
| Xiaomi (MiMo-V2-Flash unless noted) | “activating the model’s latent capabilities” | own specialists | 1; V2.5 extends 32K to 1M within it | “millions of training samples” | 5×10⁻⁵ → 5×10⁻⁶ | |
| Moonshot AI | “a high-quality cold-start policy” | prior Kimi specialists, RL experts | 1 | earlier reasoning kept | ||
| Z.ai | “interleaved thinking modes” | expert RL, rejection sampling | 1, to 202,752 tokens | erroneous segments masked | ||
| Qwen | (teachers only) | specialists from the same base | 1 | |||
| DeepSeek | per-specialist SFT, then RL or distillation | own RL specialists | per specialist | |||
| Tencent | “joint optimization of SFT and RL” | |||||
| Google DeepMind | “Instruction Tuning” | IT model ends turns with <turn|> |
MiniMax, OpenAI and StepFun describe no SFT stage in the corpus. OpenAI’s gpt-oss card describes post-training as RL, and StepFun’s technical blog for Step 5 is linked but not in the corpus.
Easy to get wrong
- Counting tutorials. A card’s “quick single-GPU fine-tuning” (LFM3) is advice to users, not the lab’s recipe.
- Training on a different template from the one inference renders. Whether earlier reasoning is kept, and which token ends a turn, must match: Gemma 4’s pre-trained and instruction-tuned models end with different tokens, and fine-tuning either “requires adding their respective end tokens”.
- Reading C1 as “less math helps”. C2 had even less math and scored lowest; C1 dropped easy math with long traces.
- Reading “specialist” one way. In DeepSeek, Moonshot, Xiaomi and Qwen it is a teacher; in MergeMix it is a scoring device that never teaches.
- Treating the warning about long SFT as a result. It is a clause with citations and an unquantified “we have observed this”.
Check yourself
1. A model solves a new agentic task 5% of the time. With groups of 8 attempts and 0/1 rewards, what share of such prompts gives a group-relative update any gradient? How large must the success rate be for half of them to?
, so about a third. For a half, gives (the all-success term is negligible there).
2. Kolibri’s long-context SFT data holds 49.8B tokens and 0.9B loss tokens. Why is the share so small, and what does it mean for the mix shares in the figure, which are shares of trained tokens?
The documents, around 200K tokens each, are context, and only the answer carries loss: 0.9 / 49.8 = 1.8%. With the loss normalised over the batch, a domain’s share of the gradient is its share of loss tokens, so a token share overstates a domain whose loss share is low. Long context is 14.9% of C1’s tokens, but if C1’s other domains have the pool’s average loss share (49.5%), it carries about 0.6% of C1’s loss tokens (computed).
3. Why mask a wrong tool call rather than drop the trajectory or train on all of it?
Dropping it loses the recovery that follows, so the model never sees a state after an error. Training on it teaches the model to make the error. Masking keeps the state in the input and puts the loss only on the correction.
4. Why can Kolibri average C1 and C2 at all, when averaging two models trained from scratch generally fails? Does the same argument explain why the average beat both?
Both were fine-tuned from the same base and stayed close in weight space, where the loss along the line between them stays low, so their midpoint isn’t worse than they are. Models trained from scratch end in different basins, and their midpoint usually has high loss. That the average scored above both parents is an empirical result, not something the basin argument guarantees.
In the map
Methods
- Supervised fine-tuning 23 sources
- Two-stage supervised fine-tuning 4 sources
- Rejection sampling 2 sources
- SFT bootstrapping with synthetic data 2 sources
- Mask loss on incorrect tool calls and terminal actions 2 sources
- Mask tool responses from the SFT loss 1 source
- MergeMix 2 sources
- Model souping 2 sources
- Difficulty-ordered SFT curriculum based on ensemble solve rates 1 source
- Self-correction cold start 2 sources
- Evaluation-based early stopping during SFT 1 source
Taxonomy nodes
- supervised fine-tuning 110 methods
Further reading
- Kolibri technical report, §3.1 (Aleph Alpha): the pool, masking rules, MergeMix at SFT, effort labels and the soup, with every table. https://aleph-alpha.com/downloads/tech-report.pdf
- Nemotron 3 Ultra technical report, §3.1, and Nemotron 3.5 report, §3.1 and §3.9 (NVIDIA): per-domain teachers, two-stage SFT, effort data and
</think>masking. - Laguna M.1/XS.2 technical report, §4 (Poolside): the SFT mix, preserved reasoning and its parser fix, and the instruction-following failure.
- GLM-5, arXiv:2602.15763, §3.1: masking erroneous segments, difficulty filtering, SFT to 202,752 tokens.
- LFM2, arXiv:2511.23404, §4: the curriculum by ensemble solve rate and the SFT filtering pipeline.
- Slinko et al., Step Rejection Fine-Tuning, arXiv:2605.10674: the one controlled test of masking harmful steps.
- Wang et al., MergeMix, arXiv:2601.17858: the method and its first-order argument, for mid-training.
- Wortsman et al., Model Soups, arXiv:2203.05482: why fine-tunes from one start can be averaged, and when they can’t.
- Singhal et al., arXiv:2610.22019: NVIDIA’s case for moving learning out of SFT.
