For V4, DeepSeek changed the last stage of its post-training: “the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation”. Instead of one reinforcement-learning run over every domain, DeepSeek trained “more than ten teacher models covering various domains”, each with its own fine-tuning and RL, and then trained a single student to match them on text the student wrote itself. Its next report, for V4.1, used “over 40 teacher models”.
NVIDIA’s Nemotron 3 Ultra report describes the problem with one RL run across many environments: “as the number of environments continues to grow, each domain contributes only a relatively small number of samples to any given training batch, diluting the per-domain learning signal”. Training one model per domain removes the dilution and leaves ten models to serve. On-policy distillation keeps the specialists for training only. The student writes a response, the teacher that owns the prompt’s domain gives its own probability for every token the student wrote, and the student moves toward the teacher wherever the two disagree. Seven of the 16 labs in our corpus run a stage of this kind.
Background
The primer lays out the stages of a recipe, S3 covers supervised fine-tuning (SFT), S4 rewards, including RL from verifiable rewards (RLVR), and S5 the RL machinery this stage reuses. Nothing in this article shows in a model’s configuration or code, so every claim comes from the labs’ technical reports first, then model cards and blogs. Papers outside our corpus are named as such.
Distillation trains a student model to imitate a teacher. Labs use the word for three operations:
| Operation | Who writes the tokens | What the student learns from | Where it appears |
|---|---|---|---|
| Logit distillation | nobody: fixed text | the teacher’s next-token distribution on that text | pre-training smaller models from larger ones: the Nemotron 3.5 Super and Lightning bases (1T tokens), LFM2 |
| Data distillation | the teacher | the teacher’s text, as SFT targets | SFT data (S3); DeepSeek V3.2’s and Qwen3.5-Omni’s “specialist distillation” |
| On-policy distillation | the student | the teacher’s probability of each student token | a late post-training stage, after RL, at seven labs |
The first is Hinton et al.’s 2015 recipe: soften the teacher’s output distribution with a temperature and train the student to match it, on a fixed set of inputs. The second is ordinary SFT on teacher output. In both, the student trains on text it did not write.
Specialist has three meanings in this series: a merged model used only to score data mixes (S3), a model that writes training data (S3), and here, a model whose probabilities supervise the student. Teacher has the last two.
Why the student’s own samples
A model trained only on someone else’s text meets, at generation time, prefixes it never saw in training: its own. One early mistake leads to a state the teacher never wrote, and errors compound from there. Ross and Bagnell (2010) made this precise: with per-step error , a learner trained on the expert’s states can fall behind by up to over steps, against about for one trained on its own, when the expert recovers quickly from a mistake. DAgger (Ross et al., 2011) addressed it by letting the learner act and asking the expert what it would have done in the states the learner reached.
Agarwal et al. carried the idea to language models as generalized knowledge distillation (GKD, ICLR 2024, outside our corpus): “using a fixed dataset can lead to distribution mismatch between output sequences seen during training and the sequences generated by the student auto-regressively during inference”. Gu et al.’s MiniLLM (ICLR 2024, also outside it, and cited by DeepSeek V4) treated the same problem as policy-gradient RL on the reverse KL. Both sample mainly from the student (MiniLLM mixes in a fifth of the teacher) and have the teacher score those samples. Distilling several teachers into one policy is older still: Rusu et al.’s policy distillation (2016) compressed several Atari game-playing networks into one.
Thinking Machines’ post on the method (Lu and Thinking Machines Lab, 2025, outside our corpus) places it on two axes. SFT is off-policy and dense: every token gets a target, on someone else’s text. RL is on-policy and sparse: one reward per episode of the student’s own text. On-policy distillation is on-policy and dense.
Sampling from the student and the direction of the KL divergence are separate choices. With the student and a teacher (E for expert, DeepSeek’s letter), the forward KL penalizes the student for missing anything the teacher considers likely, and the reverse KL penalizes it for putting mass where the teacher puts little. When the student can’t represent the teacher exactly, forward KL is mass-covering and reverse KL mode-seeking. GKD tried both, plus Jensen-Shannon mixtures, on student samples, and found the “optimal divergence seems to be task-dependent”: its headline results used forward KL, and for instruction tuning reverse KL “performs much better”. Every lab with a distillation stage that writes out its objective uses the reverse KL. Per position, it has an estimate that needs only the two models’ probabilities of the one token the student sampled; the forward KL, at the same prefixes, needs both full distributions, a token sampled from the teacher, or an unbounded importance weight.
The per-token signal
Write a prompt as , the student’s response as and the prefix before token as . The objective is the reverse KL at every position of the student’s own response, summed:
By the chain rule of KL, this equals the reverse KL between the two models’ whole responses. Four of the six labs that state their objective (NVIDIA, Moonshot, Xiaomi, Z.ai) estimate each position’s KL from the one token the student sampled there and feed the result to the policy-gradient machinery of RL, with a per-token advantage
where stops the gradient, so the log-ratio is a fixed weight. The update follows , as in S5. In NVIDIA’s Nemotron 3.5 report: “Every generated token receives a dense advantage from a single teacher forward pass, with no verifier and no sampling from the teacher; the direct contribution is zero where student and teacher agree, so the signal concentrates where the student departs from the specialist.”
A token both models give similar probability gets an advantage near zero. A token the student chose and the teacher would not have gets a large negative one, and a token the teacher likes more than the student a positive one. Averaged over the student’s choices at one position, is exactly that position’s KL, so is a single-sample estimate of it: unbiased, noisy, and negative on some tokens although the KL never is. It is S5’s estimator with the teacher in the reference model’s place.
Shown: the plain teacher (α = 1), with “probably” selected.
“probably”: student pS = 0.30, teacher pT = 0.02.
At = ln 0.02 − ln 0.30 = −2.71 nats.
Click or tap a token to read out its numbers.
The per-token update is not the gradient of . Differentiating through the sampling weights each token by the sum of its own log-ratio and those of every token after it (in RL terms, a reward-to-go), since an early token changes which later prefixes the student reaches. The labs drop the future terms: they take the gradient with the sampled prefixes held fixed, then sample again, which is DAgger’s step. NVIDIA’s Nemotron 3.5 report names this, “treating the sampled contexts as fixed during each update”, and Thinking Machines states it as a discount factor of zero, noting that “more mathematically correct” positive discounts did not “improve performance in practice”. Both updates stop at ; they differ where the student can’t match the teacher everywhere. MiniLLM kept the future term and normalized it by length, against a bias toward short responses; Xie et al.’s trust-region paper (2026, outside our corpus) calls the cumulative form a source of “an inherent length bias as well as high variance”.
Every student token needs a forward pass of its domain’s teacher, on top of the generation itself. Unlike classical distillation, the teachers are mostly as large as the student, since in most recipes they are branched from it, so the cost is several more copies of the model to serve. Nemotron 3.5 Super kept them on “independent, non-colocated inference fleets”; the whole two-stage run used 128 GB200 nodes.
In return, a prompt needs one rollout instead of a group. GRPO compares a rollout’s reward with its siblings’ (S5), so Ultra’s RLVR stage drew 16 rollouts for each of 8,192 prompts per step. In distillation the teacher supplies the reference for each token, so Ultra used “1,024 prompts, with one rollout per prompt”, and “using multiple rollouts did not yield additional benefits”. Z.ai set GRPO’s group size to 1 in its distillation stage “to increase data throughput”. The units differ, though: for most of Ultra’s agentic tasks, a distillation rollout is one turn, not a whole episode (see below).
The teacher must also assign a probability to each student token, which in practice means a shared tokenizer. NVIDIA’s Nemotron-Cascade 2 paper (outside our corpus) notes that teachers trained from the student’s SFT checkpoint “share the same tokenizer and vocabulary as the student”. DeepSeek’s V4.1 teachers “may differ architecturally”; the report doesn’t say whether their vocabularies do.
Many teachers, one student
The multi-teacher version adds routing. Each training prompt belongs to a domain, and the teacher that owns that domain scores the response. NVIDIA’s Ultra objective weights the domains with mixing coefficients , and DeepSeek V4 writes (with in the report), the weight “typically determined by the relative importance of the expert”. The student never sees two teachers on one prompt. Xiaomi named the method MOPD, “Multi-Teacher On-Policy Distillation”, for MiMo-V2-Flash. NVIDIA’s reports and Moonshot’s use the same expansion, while NVIDIA’s Ultra model card and Liquid’s LFM3 documents read the same letters as “Multi-Domain On-Policy Distillation”.
RL-trainedSFTbeing distilledfrozen
Teacher counts run from Mistral’s three, split by reasoning effort, to DeepSeek V4.1’s “over 40”, where “the best teacher for each domain may come from a different stage of model development”. NVIDIA’s Ultra used ten per round over two rounds, 15 distinct checkpoints (counted from the report’s Figure 10), plus the student “as a self-teacher for areas not covered by specialized teachers”. Moonshot’s nine for K3 cross three domains with three reasoning efforts, and the student is told the effort level while the teacher is not.
Most teachers are RL-trained specialists, but not all. Xiaomi’s teacher “can be a specialized RL-derived model with strong capabilities, a different SFT model, or even the student model itself”, and for MiMo-V2.6 it trains SFT teachers for “open-domain tasks where reliable RL rewards are difficult to design”. Ultra’s office-work and search teachers are SFT-trained. Three labs use the student’s own starting checkpoint as a teacher: Xiaomi (“Self” in its tables), NVIDIA (the self-teacher, and Super’s self-anchor below) and Mistral (its high-effort teacher). Z.ai’s general-RL checkpoint is probably both a teacher and the student’s start, though the report doesn’t say where the student starts. With an SFT teacher, the stage can train tasks that have no reward at all, a second reason to run it besides dilution. MiMo-V2-Flash also adds, by default, a scaled advantage from an outcome reward model (one score per response, S4) to the per-token log-ratio, so its version is not reward-free.
Sampled token or full distribution
The labs disagree about the single sample per position. Instead of scoring the one token the student drew, the trainer can compute each position’s KL exactly from both models’ full next-token distributions, or over the teacher’s top tokens. The exact KL is the expectation of the sampled estimate at that position, so it removes the noise of drawing one token there (a Rao-Blackwell step, in classical terms), though not the noise of which prefix the student wrote. In expectation the two give the same update. A difference in a run comes from that noise, from the clipping and masks the sampled-token versions carry, from truncating to the top , or from matching a different divergence.
DeepSeek V4 chose the full vocabulary. Using the sampled estimate as an RL advantage, its report says, “leads to high variance in gradient estimation and often causes training instability”; full logits give “more stable gradient estimates”. No ablation is printed. Mistral also uses the full vocabulary, without giving a reason. NVIDIA tested the other way for Ultra and stayed with the sampled token: top- and full-vocabulary objectives “did not improve MOPD performance and consistently underperformed the sampled-token objective on some agentic benchmarks such as Terminal Bench”. The report calls these “preliminary experiments” that “should not be interpreted as evidence that these approaches are fundamentally ineffective”, and offers a hypothesis rather than a variance argument: full matching “may impose an overly strong local constraint on prefixes sampled from the student policy, particularly when those prefixes have limited support under the teacher distribution”. It doesn’t say which divergence its full-distribution variant matched. Moonshot tried top- for K3 and “observed no clear advantage in either convergence speed or final performance in our setting”. Xiaomi’s MOPD paper (Ma et al., 2026, outside our corpus) found top- “comparable on Math and slightly worse on IF and SWE” (instruction following, software engineering) on Qwen3-30B-A3B.
Top- is not the full objective with the noise removed. A reverse KL truncated to the teacher’s top tokens can’t see the student’s mass outside them, which is where the reverse KL does its penalizing (our reading; neither source says so). No two labs ran the comparison on a shared benchmark, and the four accounts concern different models and teachers, so none of them settles the question.
Full logits take enough memory that both labs using them store something smaller. For a teacher with DeepSeek-V4-Pro’s shapes (vocab_size: 129280, hidden_size: 7168), one token’s logits in BF16 (2 bytes per number) take 258,560 bytes and its last hidden state 14,336 bytes, 18 times less (computed). Over a million student tokens that is 259 GB of logits per teacher against 14 GB of hidden states. DeepSeek therefore caches “only the last-layer teacher hidden states in a centralized buffer” and passes them through each teacher’s output layer (the matrix from hidden state to vocabulary logits) “to reconstruct the full logits on the fly”. Mistral does the same, “inspired by the DeepSeek-V4 report”.
What goes wrong, and the fixes
Each problem below comes with the fix of a lab that reported it. None reports the student exploiting a miscalibrated teacher, the distillation counterpart of reward hacking. Of the benchmarks named below, BrowseComp, Terminal Bench, GDPVal and SWE-Bench Verified test agents that browse, use a shell, do office tasks or fix code; HLE (Humanity’s Last Exam) and IMOAnswerBench are hard reasoning questions.
Teachers from different pipelines
NVIDIA found for Ultra that “teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward MOPD merge”. Its hypothesis is that teachers fine-tuned on different SFT data find the student’s responses out of distribution. The fix is a warm-up: “the student undergoes a very light SFT on data drawn from the teacher’s training distribution” before distillation. In the ablation (Ultra Table 4, single numbers with no repeats reported), the warm-up lifted GDPVal from 35.3 to 46.7 and BrowseComp from 33.0 to 44.4, against teachers at 49.5 and 51.0, and HLE only from 26.3 to 26.7. For GDPVal the warm-up data came from the office teacher’s own training data, so that row measures the data as well as the alignment. The MOPD paper saw the same failure: swapping its math teacher for Qwen3-235B-A22B, stronger and of the same family but not trained from the student’s checkpoint, made the student’s math worse, and the paper concludes that “same-origin teachers” are “critical”.
States the teacher never sees, and long episodes
The student’s prefixes may be ones the teacher would never write, and its scores there carry little information; NVIDIA’s hypothesis about the full-vocabulary objective rests on this. Long agent episodes add a cost problem. Mixing them with single-turn reasoning prompts left Ultra’s training inefficient “because rollout times can differ dramatically”, so “for most of the agentic tasks” NVIDIA used “single-turn rollouts”: the student writes one turn after a recorded history. Moonshot says its distillation reward fits K3’s partial rollouts, which continue unfinished long episodes across steps (S5).
Xiaomi’s MiMo-V2.6 makes recorded histories a method, Multi-Prefix MOPD (MOPD2 there; at NVIDIA, “MOPD2” names the checkpoint after the second round). It keeps full student rollouts where an RL-trained teacher exists, and adds prefix-conditioned ones chiefly for its SFT teachers on open-domain tasks. Each multi-turn trajectory from a teacher, or from SFT data, gives one prefix before each assistant turn: “The student samples one new turn from each prefix without regenerating earlier interactions”, and the teacher scores that turn. The student then trains in states the teacher has seen, at the price of being less on-policy for the earlier turns.
Prompts no teacher owns
An environment with no specialist would otherwise get no signal, and the student may drift on it. Nemotron 3.5 Super routes such prompts “to a self-anchor, a frozen copy of the student’s starting checkpoint, which provides no signal while the student still matches it and regularizes departures from it”. On those prompts it is S5’s KL penalty to a reference model, applied to a share of the batch: 5% in the first stage, beside agentic (35%), reasoning (25%), alignment (20%) and vision tool use (15%). The anchor can only keep what the student already had.
Benchmarks that peak and then fall
NVIDIA reports that under distillation, benchmarks “improve at different rates”, and that “some peak within a few steps and then decline with further training”. Super’s answer is two short stages. The first, weighted toward fast-moving domains, runs 30 steps of 512 prompts. The second retires the agentic and reasoning teachers, folds their environments into the self-anchor (now the stage-1 model) and runs 40 steps of 1,024 prompts, one pass: 56,320 prompts in all (computed).
The second stage also softens the teacher into a proximal teacher, after Xie et al.’s trust-region distillation, which keeps each update close to the current student. The teacher is replaced by the mixture with , where is the student as it was when the batch was sampled, held fixed during the update. Since the mixture is at least , the advantage can’t fall below : a token the teacher considers impossible gets a bounded negative signal. Near agreement the mixture also shrinks every signal to about times its size, so part of the effect is a smaller step. In the toy above, the hedge’s advantage moves from to , and “42“‘s from to . Distilling toward this mixture is, with a frozen copy of the student inside it, the skew reverse KL of Ko et al. (2024) at skew 0.8 in their notation (our reading; NVIDIA cites Xie et al.). Super’s pre-training distillation used the same divergence at skew 0.1, close to the teacher.
Stale rollouts
The stage runs on RL infrastructure, so it inherits RL’s staleness (S5): a rollout can come from weights a step or two old. Ultra handles this as S5’s RL does, reweighting each token by how much likelier the current weights make it than the weights that wrote it, clipped as in PPO around a “proximal policy” (S5’s anchor for the update, unrelated to the proximal teacher above), with IcePop’s mask. Super masks tokens “with weights outside [0.2, 5]”, and NVIDIA notes that this corrects “action probabilities at the sampled contexts”, not the prefix, and that masking introduces bias. A full-vocabulary version needs no token weight, since it computes each position exactly.
Teachers split by effort or by stage
Two of the seven labs split their teachers by something other than domain.
Mistral splits by effort. Length-penalized RL runs from the high-reasoning checkpoint produce medium- and low-effort teachers; “The high-effort teacher is the initialization checkpoint itself”. Distillation then puts three reasoning efforts into one model, and SFT needed only “a single reasoning effort”.
Z.ai’s GLM-5 uses its own earlier stages as teachers, in cross-stage distillation. Its RL runs in sequence, reasoning, then agentic, then general, and “sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities”. So “the final checkpoints from the preceding training stages serve as teacher models”, to recover “skills acquired in earlier SFT and RL stages”, with the sampled-token log-ratio in place of GRPO’s advantage, group size 1 and batch 1,024. DeepSeek V3.2 had addressed the same forgetting the other way, by merging “reasoning, agent, and human alignment training into one RL stage”.
Two labs outside the seven use related ideas for other purposes, both forms of context distillation, where the teacher is the same model given more or easier context. Aleph Alpha’s Kolibri has no distillation stage and lists MOPD as future work. Inside its single RL run it uses on-policy self-distillation: for a group of rollouts that all failed (best reward below 0.2) in eight tool-use and terminal environments, the teacher is the live policy given a remedial hint about the error, and the student is the same policy without it. The loss is a forward KL over the full vocabulary, and a group trained this way gets no RL loss. It touched 0.10% of tokens in those environments early in the run and 0.034% late. Qwen3.5-Omni’s “On-Policy Distillation (OPD)” stage uses the model’s response to a text query as “the distillation target for the corresponding audio-conditioned query”; no teacher scores student tokens.
Distillation or weight merging
The other way to fold specialists into one model is to average their weights. Xiaomi wrote for MiMo-V2-Flash that MOPD, unlike parameter merging, “preserves peak performance of the strongest teacher across all domains”, and DeepSeek V4 that distillation avoids “the performance degradation often encountered in traditional weight-merging or mixed RL techniques”. Neither report includes a comparison. Liquid’s LFM2 ended post-training with a merging stage, and LFM3 lists distillation and no merging; no Liquid document says one replaced the other. DeepSeek V4.1 does both: it uses “model merging to reinitialize successive RL runs” and then distills.
The one controlled comparison is outside our corpus. Xiaomi’s MOPD paper (Qwen3-30B-A3B; math, instruction following, software engineering) scores each method by the share of the gap from the SFT student to the teachers that it closes, averaged over domains: MOPD 0.937, a single mixed RL run 0.882, task-arithmetic merging 0.857, SFT on teacher outputs 0.824, RL run one domain after another 0.775, plain weight averaging 0.328. It is one model at one scale, with no repeats reported, from the lab that proposed the method, and MOPD’s arm, like the merging and SFT arms, builds on the RL runs that made its teachers.
How much of each teacher the student keeps
NVIDIA’s Ultra report is the one source with a column for every checkpoint. It defines “Recovery rate” as
the share of the gap between the student before distillation (the RLVR checkpoint) and the domain’s teacher that two rounds of distillation close. It is a ratio of score differences, not an accuracy. On BrowseComp, the teacher leads by 20.0 points, the student gains 13.4, and recovery is 67% (computed, matching the printed 67.0%). On HLE, the gap is 6.5 points and the student gains 1.1: 17%.
A ratio over a small gap is fragile. The four most extreme rows, both above 100% and the two lowest, all have gaps of 6.5 points or less, where one point of score moves the ratio by 15 percentage points or more. The report gives no error bars or repeats.
SFTRLVRMOPD1MOPD2teacher
Recovery = (MOPD2 − RLVR) / (Teacher − RLVR), as NVIDIA prints it.
The report’s reading is that distillation works best “when the teacher’s advantage can be expressed as token-level preferences over trajectories that the student is already able to sample”. On HLE, the reasoning teacher had been trained further on data “the student has not directly seen”, and NVIDIA reads the small recovery as “a limitation of the on-policy distillation setting rather than a failure of the teacher”. Three rows the report leaves uncommented cut against a tidy reading: LiveCodeBench falls in the second round (90.0 to 89.0), GDPVal doesn’t move in it, and IMOAnswerBench’s RL checkpoint sits below its SFT one (84.5 against 85.1).
Both NVIDIA and the MOPD paper measure from the checkpoint distillation starts from: after RL at Ultra, after SFT in the MOPD paper, which has no RL stage in between. Measured from Ultra’s SFT checkpoint instead, HLE’s figure would be 56% rather than 17% (computed), but that credits distillation with what RL did first.
MiMo-V2-Flash’s Table 7 is the other same-model table, with the student before and after distillation (the starting checkpoint isn’t named; the pipeline implies SFT) and the best teacher per benchmark. The student ends below its best teacher on 4 of 12 benchmarks, by 6.3 points on BrowseComp, whose best teacher was an SFT model. On five benchmarks no specialist beat the student’s own starting checkpoint, and on two of them distillation left the student below where it started: GPQA-Diamond by 0.6, creative writing by 3.9. The abstract says MOPD enables “the student model to perfectly master teacher expertise”; the section text says “matches or exceeds the strongest teacher in most domains”, which is what the table shows.
Stage order across the series
As the series’ last stage, the article closes with each lab’s whole post-training order as its reports state it. A blank means we searched the lab’s documents and found no order.
| Lab (model) | Post-training order |
|---|---|
| Aleph Alpha (Kolibri) | SFT → one RL run, with self-distillation for failed groups |
| DeepSeek (V3.2) | per-domain specialists write data → SFT on it → one mixed RL run |
| DeepSeek (V4) | per-domain specialists (SFT, then RL) → distillation from more than ten |
| DeepSeek (V4.1) | SFT → successive RL runs, reinitialized by merging → distillation from over 40 |
| Google DeepMind (Gemma 4) | “similar post-training approach as in Gemma 3” |
| Liquid AI (LFM3) | SFT → mixed RL → distillation from six → agentic RL |
| MiniMax | |
| Mistral AI (Large 4) | mid-training and SFT → RL → length-penalized RL teachers → distillation from three |
| Moonshot AI (K3) | SFT → nine RL experts → distillation |
| NVIDIA (Nemotron 3 Ultra) | SFT → RLVR → light SFT warm-up → (teachers → distillation) twice → MTP boosting |
| NVIDIA (Nemotron 3.5 Super / Lightning) | SFT → RLVR and RLHF → teachers → distillation in two stages → MTP boosting → QAD / SFT → RLVR → MTP boosting |
| OpenAI (gpt-oss) | “CoT RL techniques as OpenAI o3”; no order |
| Poolside (Laguna) | mid-training → SFT → RL → QAD |
| Qwen (Qwen3.5-Omni) | data from specialists → text-to-audio “OPD” → RL |
| StepFun | |
| Tencent (Hy3) | “joint optimization of SFT and RL”; no order |
| Thinking Machines (Inkling) | SFT on synthetic data → large-scale RL |
| Xiaomi (MiMo-V2-Flash / V2.6) | SFT → domain teachers → MOPD / mid-training → short SFT → one mixed RL run → Multi-Prefix MOPD |
| Z.ai (GLM-5) | SFT → reasoning RL → agentic RL → general RL → cross-stage distillation |
MTP boosting trains only the multi-token prediction head, which drafts tokens ahead for faster decoding, to match the frozen model; QAD (quantization-aware distillation) trains a low-precision copy to match the full one on fixed data. Neither adds a capability.
mid-trainingSFTRLon-policy distillation (OPD)parallel specialists or teachers, trained with RLend cap: MTP boosting, quantization-aware distillation
Equal block widths: the order of stages, not their length in time or tokens. Read left to right.
DeepSeek V3.2 and Z.ai answered forgetting across stages in opposite ways (above), and DeepSeek V4 then replaced V3.2’s mixed stage with distillation from per-domain specialists, the shape Kimi K3 and MiMo-V2-Flash also use. No report in our corpus compares two orders on one model; the MOPD paper’s scores above (one domain after another 0.775, mixed 0.882, distillation 0.937) are the only such comparison, on its 30B test model.
Kimi K3 and DeepSeek V4 also run quantization-aware training through post-training. DeepSeek’s covers “teacher and reference models” too; Kimi K3 doesn’t say whether its teachers are quantized.
Gains measured stage by stage
Most published gains compare model generations and can’t be assigned to a stage. Four sources print a measured checkpoint from before RL or distillation, SFT or implied SFT, beside a later one of the same model:
| Source | Model | Benchmark: SFT → RL (→ distillation) |
|---|---|---|
| Ultra report Table 5 | Nemotron 3 Ultra | Terminal Bench 2.0 34.5 → 44.5 → 54.0; BrowseComp 14.3 → 31.0 → 44.4; OmniScience non-hallucination 4.8 → 46.3 → 78.7; SWE-Bench Verified 63.5 → 65.8 → 71.7; IMOAnswerBench 85.1 → 84.5 → 88.6 |
| MiMo-V2-Flash report Table 7 | MiMo-V2-Flash | SFT → distillation, no mixed RL: AIME 2025 89.3 → 94.1; SWE-Bench Verified 67.8 → 73.4; BrowseComp 42.5 → 45.4; GPQA-Diamond 84.9 → 84.3 (starting checkpoint not named) |
| Kolibri report Table 25 | Kolibri | BrowseComp 22.0 → 29.4; IFBench 48.2 → 78.1; MMLU-Pro 78.5 → 80.0 (RL includes self-distillation) |
| Mistral blog | Mistral Large 4 | BrowseComp 75% → 84.5% (“our RL pipeline”; RL and distillation not split) |
At Ultra, the share of the post-SFT gain that arrives during distillation ranges from 16% (HLE) to 76% (GDPVal) on ten of the eleven rows, about half on BrowseComp and Terminal Bench (computed); on IMOAnswerBench, where RL lowered the score, distillation brought more than the whole gain. That share is measured in this order; what distillation would add straight after SFT isn’t. Part of it is the teachers’ own RL, passed on. Kolibri’s RL raised all 13 benchmarks in its table, by 1.5 points (MMLU-Pro) to 36.0 (AA-Omniscience).
None of these sources reports a stage that gave nothing, and none is a randomized comparison; publications select for what shipped. The nulls that are reported are ablations of the stage’s variants (more rollouts, top- and full-vocabulary objectives, the warm-up on HLE), not of the stage itself.
Comparison
| Variant | Who writes the tokens | Teacher signal | Main cost or risk | Labs |
|---|---|---|---|---|
| Sampled-token on-policy distillation | student | log-ratio on the sampled token | teacher pass per token; drops future terms; partial, uneven recovery | NVIDIA, Moonshot, Xiaomi, Z.ai |
| Full-vocabulary on-policy distillation | student | exact KL per position | logits memory; NVIDIA’s preliminary runs found it worse on agentic tasks | DeepSeek, Mistral |
| Proximal teacher | student | log-ratio to | weaker signal everywhere; bounded below by | NVIDIA (Super, stage 2) |
| Prefix-conditioned | recorded history up to a turn, student after | teacher on the new turn | less on-policy for earlier turns | NVIDIA (Ultra, most agentic tasks), Xiaomi (V2.6) |
| Cross-stage | student | earlier-stage checkpoints | each teacher is capped at its stage’s final checkpoint | Z.ai |
| Self-distillation in RL | student | the policy with a hint, forward KL | needs a useful hint; few tokens | Aleph Alpha |
Where the labs sit
Seven of the 16 labs run a post-training stage in which a teacher scores the student’s own samples: DeepSeek, Liquid, Mistral, Moonshot, NVIDIA, Xiaomi and Z.ai. Five of them split teachers by domain (all but Mistral and Z.ai). Blank cells mean searched and not stated.
| Lab | Teachers | Objective | Student starts from | Rollouts per step | After it |
|---|---|---|---|---|---|
| DeepSeek (V4 / V4.1) | >10 / >40, by domain | full-vocabulary reverse KL | not stated / after RL | none | |
| Liquid AI (LFM3) | 6, by domain | after mixed RL | agentic RL | ||
| Mistral AI (Large 4) | 3, by effort | full-vocabulary reverse KL | high-effort RL checkpoint | none | |
| Moonshot AI (K3) | 9, domain × effort | sampled token, clipped | none | ||
| NVIDIA (Ultra / Super) | 10 per round, 2 rounds / 4 per stage, 2 stages, plus self | sampled token | RLVR checkpoint / RL (RLVR and RLHF) checkpoint | 1,024 × 1 / 512 × 1, then 1,024 prompts | MTP boosting / MTP boosting, QAD |
| Xiaomi (V2-Flash / V2.6) | domain RL, SFT and self teachers | sampled token + outcome reward / standard plus prefix-conditioned | SFT / after mixed RL | none | |
| Z.ai (GLM-5) | earlier stages’ checkpoints | sampled token in GRPO | after general RL (implied) | 1,024 × 1 | none |
Aleph Alpha distills inside RL; Qwen uses the name for a text-to-audio stage; Poolside distills only for quantization. Google DeepMind, MiniMax, OpenAI, StepFun, Tencent and Thinking Machines state no distillation in post-training. Thinking Machines wrote the post that DeepSeek, NVIDIA, Moonshot and Mistral cite for the method, and its Inkling documents describe SFT and RL only.
Easy to get wrong
- Reading “distillation replaces RL”. The teachers are mostly RL-trained, so the stage passes on RL gains rather than doing without them. DeepSeek V4, Kimi K3 and MiMo-V2-Flash run RL only on the teachers; MiMo-V2.6, Mistral, Liquid, NVIDIA and DeepSeek V4.1 distill a student that was itself RL-trained, Z.ai after sequential RL stages, and Liquid runs agentic RL after distillation.
- Comparing recovery figures across sources without their baselines. Both NVIDIA and the MOPD paper start from the student that enters distillation, an RL checkpoint at NVIDIA and an SFT one in the MOPD paper; HLE’s 17% becomes 56% if Ultra’s is moved back to SFT.
- Taking “specialist distillation” for this stage. At DeepSeek V3.2 and in Qwen3.5-Omni’s first stage, specialists write SFT data.
- Treating “on-policy” and “reverse KL” as one choice. GKD’s headline on-policy results used forward KL, and Kolibri’s self-distillation is on-policy with a forward KL.
- Comparing “MOPD2” across labs. At NVIDIA it is a checkpoint, at Xiaomi a method.
Check yourself
Ultra’s BrowseComp scores are 14.3 after SFT, 31.0 after RLVR, 44.4 after two rounds of distillation, and 51.0 for the teacher. What is the recovery with NVIDIA’s baseline, and with the SFT baseline?
With RLVR: , the printed value. With SFT: . The second credits distillation with what RL did too.
At one position the student samples a token with probability 0.5 that the teacher gives 0.001. What advantage does the token get under plain distillation, and under Super’s stage-2 proximal teacher with α = 0.2?
Plain: . Proximal: , essentially the bound . The signal still says “less of this”, but no longer dominates the update.
Ultra’s RLVR stage drew 16 rollouts per prompt, its distillation stage one. Why can distillation do with one?
GRPO needs several rollouts of the same prompt to estimate a baseline and turn rewards into advantages. In distillation, the teacher’s probability of each token is the reference, so one rollout already gives every token an unbiased signal; more rollouts would only average its noise, and NVIDIA found they “did not yield additional benefits”.
In the map
Methods
- On-Policy Distillation 6 sources
- Multi-Teacher On-Policy Distillation 21 sources
- Specialist Distillation 3 sources
- Light supervised fine-tuning on teacher-distribution data 1 source
- Frozen Student Self-Anchor 1 source
- Two-Stage Multi-Teacher On-Policy Distillation 1 source
- Proximal-Teacher Trust-Region Policy Distillation 1 source
- Prefix-Conditioned On-Policy Distillation 3 sources
- On-Policy Cross-Stage Distillation 1 source
- On-Policy Self-Distillation 2 sources
- Full-vocabulary logit distillation 3 sources
- Reverse KL divergence 2 sources
- Hidden-state caching for KL distillation 1 source
- Parameter-space model merging 1 source
Taxonomy nodes
- policy distillation 32 methods
- distillation objective 25 methods
Further reading
- Agarwal et al., On-policy distillation of language models (GKD), ICLR 2024, arXiv:2306.13649: the imitation-learning argument, and divergences compared on student samples.
- Gu et al., MiniLLM, ICLR 2024, arXiv:2306.08543: reverse KL as a policy-gradient problem, with the exact single-step term, teacher-mixed sampling and length normalization.
- Lu and Thinking Machines Lab, On-policy distillation, doi:10.64434/tml.20251026: the dense/sparse framing, the zero-discount choice and compute comparisons at their scale.
- Ma et al., MOPD, arXiv:2606.30406: the controlled comparison with mixed RL, merging and SFT on teacher data, top- objectives and the external-teacher failure.
- Xie et al., Trust region policy distillation, arXiv:2607.04751: the proximal teacher and its variance bound. Liao et al., arXiv:2607.04763: replayed prefixes and the “prefix trap”.
- Z. Yang et al., Nemotron-Cascade 2, arXiv:2603.19220 §4.4: distillation inside a staged RL pipeline, with step counts.
- Hinton et al., arXiv:1503.02531, and Ko et al., DistiLLM, arXiv:2402.03898: logit distillation and the skew reverse KL. Rusu et al., Policy distillation, ICLR 2016: several teachers into one policy, before language models.
- The labs’ accounts: Nemotron 3 Ultra report §3.3 (the most detailed, with Tables 4 and 5), Nemotron 3.5 report §3.6, DeepSeek V4 §5.1 to §5.2 and V4.1 §5, MiMo-V2-Flash §4 and MiMo-V2.6 §5.6, Kimi K3 §4.1, Mistral Large 4 §4.5, GLM-5 §3.5, Kolibri §3.2.1.
