Open Models Almanac

Training stages · S6

On-policy distillation: folding specialists back into one model

Distilling RL-trained specialists on the student's own samples gives every token a signal and ships one model, and the student keeps only part of each teacher, unevenly by task

Illustration of five small teal and coral funnels of different shapes hanging in a row above a single ink-blue bowl on the left; thin streams run from each funnel into the bowl, and a muted-gold sieve rests on the bowl's rim, with a few drops left on it.

For V4, DeepSeek changed the last stage of its post-training: “the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation”. Instead of one reinforcement-learning run over every domain, DeepSeek trained “more than ten teacher models covering various domains”, each with its own fine-tuning and RL, and then trained a single student to match them on text the student wrote itself. Its next report, for V4.1, used “over 40 teacher models”.

NVIDIA’s Nemotron 3 Ultra report describes the problem with one RL run across many environments: “as the number of environments continues to grow, each domain contributes only a relatively small number of samples to any given training batch, diluting the per-domain learning signal”. Training one model per domain removes the dilution and leaves ten models to serve. On-policy distillation keeps the specialists for training only. The student writes a response, the teacher that owns the prompt’s domain gives its own probability for every token the student wrote, and the student moves toward the teacher wherever the two disagree. Seven of the 16 labs in our corpus run a stage of this kind.

Background

The primer lays out the stages of a recipe, S3 covers supervised fine-tuning (SFT), S4 rewards, including RL from verifiable rewards (RLVR), and S5 the RL machinery this stage reuses. Nothing in this article shows in a model’s configuration or code, so every claim comes from the labs’ technical reports first, then model cards and blogs. Papers outside our corpus are named as such.

Distillation trains a student model to imitate a teacher. Labs use the word for three operations:

OperationWho writes the tokensWhat the student learns fromWhere it appears
Logit distillationnobody: fixed textthe teacher’s next-token distribution on that textpre-training smaller models from larger ones: the Nemotron 3.5 Super and Lightning bases (1T tokens), LFM2
Data distillationthe teacherthe teacher’s text, as SFT targetsSFT data (S3); DeepSeek V3.2’s and Qwen3.5-Omni’s “specialist distillation”
On-policy distillationthe studentthe teacher’s probability of each student tokena late post-training stage, after RL, at seven labs

The first is Hinton et al.’s 2015 recipe: soften the teacher’s output distribution with a temperature and train the student to match it, on a fixed set of inputs. The second is ordinary SFT on teacher output. In both, the student trains on text it did not write.

Specialist has three meanings in this series: a merged model used only to score data mixes (S3), a model that writes training data (S3), and here, a model whose probabilities supervise the student. Teacher has the last two.

Why the student’s own samples

A model trained only on someone else’s text meets, at generation time, prefixes it never saw in training: its own. One early mistake leads to a state the teacher never wrote, and errors compound from there. Ross and Bagnell (2010) made this precise: with per-step error ϵ\epsilon, a learner trained on the expert’s states can fall behind by up to n2ϵn^2\epsilon over nn steps, against about nϵn\epsilon for one trained on its own, when the expert recovers quickly from a mistake. DAgger (Ross et al., 2011) addressed it by letting the learner act and asking the expert what it would have done in the states the learner reached.

Agarwal et al. carried the idea to language models as generalized knowledge distillation (GKD, ICLR 2024, outside our corpus): “using a fixed dataset can lead to distribution mismatch between output sequences seen during training and the sequences generated by the student auto-regressively during inference”. Gu et al.’s MiniLLM (ICLR 2024, also outside it, and cited by DeepSeek V4) treated the same problem as policy-gradient RL on the reverse KL. Both sample mainly from the student (MiniLLM mixes in a fifth of the teacher) and have the teacher score those samples. Distilling several teachers into one policy is older still: Rusu et al.’s policy distillation (2016) compressed several Atari game-playing networks into one.

Thinking Machines’ post on the method (Lu and Thinking Machines Lab, 2025, outside our corpus) places it on two axes. SFT is off-policy and dense: every token gets a target, on someone else’s text. RL is on-policy and sparse: one reward per episode of the student’s own text. On-policy distillation is on-policy and dense.

Sampling from the student and the direction of the KL divergence are separate choices. With the student πθ\pi_\theta and a teacher πE\pi_E (E for expert, DeepSeek’s letter), the forward KL KL(πE∥πθ)\mathrm{KL}(\pi_E \| \pi_\theta) penalizes the student for missing anything the teacher considers likely, and the reverse KL KL(πθ∥πE)\mathrm{KL}(\pi_\theta \| \pi_E) penalizes it for putting mass where the teacher puts little. When the student can’t represent the teacher exactly, forward KL is mass-covering and reverse KL mode-seeking. GKD tried both, plus Jensen-Shannon mixtures, on student samples, and found the “optimal divergence seems to be task-dependent”: its headline results used forward KL, and for instruction tuning reverse KL “performs much better”. Every lab with a distillation stage that writes out its objective uses the reverse KL. Per position, it has an estimate that needs only the two models’ probabilities of the one token the student sampled; the forward KL, at the same prefixes, needs both full distributions, a token sampled from the teacher, or an unbounded importance weight.

The per-token signal

Write a prompt as xx, the student’s response as y=(y1,…,yn)y = (y_1, \dots, y_n) and the prefix before token tt as st=(x,y<t)s_t = (x, y_{<t}). The objective is the reverse KL at every position of the student’s own response, summed:

L(θ)=Ey∼πθ[∑tKL(πθ(⋅∣st) ∥ πE(⋅∣st))].\mathcal{L}(\theta) = \mathbb{E}_{y \sim \pi_\theta}\Big[\sum_t \mathrm{KL}\big(\pi_\theta(\cdot \mid s_t) \,\|\, \pi_E(\cdot \mid s_t)\big)\Big].

By the chain rule of KL, this equals the reverse KL between the two models’ whole responses. Four of the six labs that state their objective (NVIDIA, Moonshot, Xiaomi, Z.ai) estimate each position’s KL from the one token the student sampled there and feed the result to the policy-gradient machinery of RL, with a per-token advantage

At=sg⁡[log⁡πE(yt∣st)−log⁡πθ(yt∣st)],A_t = \operatorname{sg}\big[\log \pi_E(y_t \mid s_t) - \log \pi_\theta(y_t \mid s_t)\big],

where sg⁡\operatorname{sg} stops the gradient, so the log-ratio is a fixed weight. The update follows ∑tAt∇θlog⁡πθ(yt∣st)\sum_t A_t \nabla_\theta \log \pi_\theta(y_t \mid s_t), as in S5. In NVIDIA’s Nemotron 3.5 report: “Every generated token receives a dense advantage from a single teacher forward pass, with no verifier and no sampling from the teacher; the direct contribution is zero where student and teacher agree, so the signal concentrates where the student departs from the specialist.”

A token both models give similar probability gets an advantage near zero. A token the student chose and the teacher would not have gets a large negative one, and a token the teacher likes more than the student a positive one. Averaged over the student’s choices at one position, Eyt∼πθ[−At]\mathbb{E}_{y_t \sim \pi_\theta}[-A_t] is exactly that position’s KL, so −At-A_t is a single-sample estimate of it: unbiased, noisy, and negative on some tokens although the KL never is. It is S5’s k1k_1 estimator with the teacher in the reference model’s place.

Shown: the plain teacher (α = 1), with “probably” selected.

“probably”: student pS = 0.30, teacher pT = 0.02.

At = ln 0.02 − ln 0.30 = −2.71 nats.

Figure 1.A toy response and its per-token signal. Where student and teacher give the sampled token similar probability, the advantage is close to zero; the hedge "probably", which the teacher would rarely write, gets a large negative one. The alpha slider anticipates the proximal teacher described below: mixing the student back into the teacher bounds the negative signal at log(1 − alpha) and shrinks every other bar too, as in Nemotron 3.5 Super's second distillation stage (alpha = 0.2). The numbers are invented.

The per-token update is not the gradient of L\mathcal{L}. Differentiating L\mathcal{L} through the sampling weights each token by the sum of its own log-ratio and those of every token after it (in RL terms, a reward-to-go), since an early token changes which later prefixes the student reaches. The labs drop the future terms: they take the gradient with the sampled prefixes held fixed, then sample again, which is DAgger’s step. NVIDIA’s Nemotron 3.5 report names this, “treating the sampled contexts as fixed during each update”, and Thinking Machines states it as a discount factor of zero, noting that “more mathematically correct” positive discounts did not “improve performance in practice”. Both updates stop at πθ=πE\pi_\theta = \pi_E; they differ where the student can’t match the teacher everywhere. MiniLLM kept the future term and normalized it by length, against a bias toward short responses; Xie et al.’s trust-region paper (2026, outside our corpus) calls the cumulative form a source of “an inherent length bias as well as high variance”.

Every student token needs a forward pass of its domain’s teacher, on top of the generation itself. Unlike classical distillation, the teachers are mostly as large as the student, since in most recipes they are branched from it, so the cost is several more copies of the model to serve. Nemotron 3.5 Super kept them on “independent, non-colocated inference fleets”; the whole two-stage run used 128 GB200 nodes.

In return, a prompt needs one rollout instead of a group. GRPO compares a rollout’s reward with its siblings’ (S5), so Ultra’s RLVR stage drew 16 rollouts for each of 8,192 prompts per step. In distillation the teacher supplies the reference for each token, so Ultra used “1,024 prompts, with one rollout per prompt”, and “using multiple rollouts did not yield additional benefits”. Z.ai set GRPO’s group size to 1 in its distillation stage “to increase data throughput”. The units differ, though: for most of Ultra’s agentic tasks, a distillation rollout is one turn, not a whole episode (see below).

The teacher must also assign a probability to each student token, which in practice means a shared tokenizer. NVIDIA’s Nemotron-Cascade 2 paper (outside our corpus) notes that teachers trained from the student’s SFT checkpoint “share the same tokenizer and vocabulary as the student”. DeepSeek’s V4.1 teachers “may differ architecturally”; the report doesn’t say whether their vocabularies do.

Many teachers, one student

The multi-teacher version adds routing. Each training prompt belongs to a domain, and the teacher that owns that domain scores the response. NVIDIA’s Ultra objective weights the domains with mixing coefficients λi\lambda_i, and DeepSeek V4 writes ∑iλi KL(πθ∥πEi)\sum_i \lambda_i \,\mathrm{KL}(\pi_\theta \| \pi_{E_i}) (with wiw_i in the report), the weight “typically determined by the relative importance of the expert”. The student never sees two teachers on one prompt. Xiaomi named the method MOPD, “Multi-Teacher On-Policy Distillation”, for MiMo-V2-Flash. NVIDIA’s reports and Moonshot’s use the same expansion, while NVIDIA’s Ultra model card and Liquid’s LFM3 documents read the same letters as “Multi-Domain On-Policy Distillation”.

Figure 2.How the multi-teacher stage is wired in NVIDIA's Nemotron 3.5 Super (stage 1 shares shown). Teachers are trained from the same RL checkpoint the student starts from; the student writes every response, and the prompt's domain decides which single teacher scores it. Prompts no specialist owns go to a frozen copy of the starting student. Ultra, below, adds a light SFT warm-up, uses two SFT-trained teachers, and runs the stage twice.

Teacher counts run from Mistral’s three, split by reasoning effort, to DeepSeek V4.1’s “over 40”, where “the best teacher for each domain may come from a different stage of model development”. NVIDIA’s Ultra used ten per round over two rounds, 15 distinct checkpoints (counted from the report’s Figure 10), plus the student “as a self-teacher for areas not covered by specialized teachers”. Moonshot’s nine for K3 cross three domains with three reasoning efforts, and the student is told the effort level while the teacher is not.

Most teachers are RL-trained specialists, but not all. Xiaomi’s teacher “can be a specialized RL-derived model with strong capabilities, a different SFT model, or even the student model itself”, and for MiMo-V2.6 it trains SFT teachers for “open-domain tasks where reliable RL rewards are difficult to design”. Ultra’s office-work and search teachers are SFT-trained. Three labs use the student’s own starting checkpoint as a teacher: Xiaomi (“Self” in its tables), NVIDIA (the self-teacher, and Super’s self-anchor below) and Mistral (its high-effort teacher). Z.ai’s general-RL checkpoint is probably both a teacher and the student’s start, though the report doesn’t say where the student starts. With an SFT teacher, the stage can train tasks that have no reward at all, a second reason to run it besides dilution. MiMo-V2-Flash also adds, by default, a scaled advantage from an outcome reward model (one score per response, S4) to the per-token log-ratio, so its version is not reward-free.

Sampled token or full distribution

The labs disagree about the single sample per position. Instead of scoring the one token the student drew, the trainer can compute each position’s KL exactly from both models’ full next-token distributions, or over the teacher’s top kk tokens. The exact KL is the expectation of the sampled estimate at that position, so it removes the noise of drawing one token there (a Rao-Blackwell step, in classical terms), though not the noise of which prefix the student wrote. In expectation the two give the same update. A difference in a run comes from that noise, from the clipping and masks the sampled-token versions carry, from truncating to the top kk, or from matching a different divergence.

DeepSeek V4 chose the full vocabulary. Using the sampled estimate as an RL advantage, its report says, “leads to high variance in gradient estimation and often causes training instability”; full logits give “more stable gradient estimates”. No ablation is printed. Mistral also uses the full vocabulary, without giving a reason. NVIDIA tested the other way for Ultra and stayed with the sampled token: top-kk and full-vocabulary objectives “did not improve MOPD performance and consistently underperformed the sampled-token objective on some agentic benchmarks such as Terminal Bench”. The report calls these “preliminary experiments” that “should not be interpreted as evidence that these approaches are fundamentally ineffective”, and offers a hypothesis rather than a variance argument: full matching “may impose an overly strong local constraint on prefixes sampled from the student policy, particularly when those prefixes have limited support under the teacher distribution”. It doesn’t say which divergence its full-distribution variant matched. Moonshot tried top-kk for K3 and “observed no clear advantage in either convergence speed or final performance in our setting”. Xiaomi’s MOPD paper (Ma et al., 2026, outside our corpus) found top-kk “comparable on Math and slightly worse on IF and SWE” (instruction following, software engineering) on Qwen3-30B-A3B.

Top-kk is not the full objective with the noise removed. A reverse KL truncated to the teacher’s top tokens can’t see the student’s mass outside them, which is where the reverse KL does its penalizing (our reading; neither source says so). No two labs ran the comparison on a shared benchmark, and the four accounts concern different models and teachers, so none of them settles the question.

Full logits take enough memory that both labs using them store something smaller. For a teacher with DeepSeek-V4-Pro’s shapes (vocab_size: 129280, hidden_size: 7168), one token’s logits in BF16 (2 bytes per number) take 258,560 bytes and its last hidden state 14,336 bytes, 18 times less (computed). Over a million student tokens that is 259 GB of logits per teacher against 14 GB of hidden states. DeepSeek therefore caches “only the last-layer teacher hidden states in a centralized buffer” and passes them through each teacher’s output layer (the matrix from hidden state to vocabulary logits) “to reconstruct the full logits on the fly”. Mistral does the same, “inspired by the DeepSeek-V4 report”.

What goes wrong, and the fixes

Each problem below comes with the fix of a lab that reported it. None reports the student exploiting a miscalibrated teacher, the distillation counterpart of reward hacking. Of the benchmarks named below, BrowseComp, Terminal Bench, GDPVal and SWE-Bench Verified test agents that browse, use a shell, do office tasks or fix code; HLE (Humanity’s Last Exam) and IMOAnswerBench are hard reasoning questions.

Teachers from different pipelines

NVIDIA found for Ultra that “teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward MOPD merge”. Its hypothesis is that teachers fine-tuned on different SFT data find the student’s responses out of distribution. The fix is a warm-up: “the student undergoes a very light SFT on data drawn from the teacher’s training distribution” before distillation. In the ablation (Ultra Table 4, single numbers with no repeats reported), the warm-up lifted GDPVal from 35.3 to 46.7 and BrowseComp from 33.0 to 44.4, against teachers at 49.5 and 51.0, and HLE only from 26.3 to 26.7. For GDPVal the warm-up data came from the office teacher’s own training data, so that row measures the data as well as the alignment. The MOPD paper saw the same failure: swapping its math teacher for Qwen3-235B-A22B, stronger and of the same family but not trained from the student’s checkpoint, made the student’s math worse, and the paper concludes that “same-origin teachers” are “critical”.

States the teacher never sees, and long episodes

The student’s prefixes may be ones the teacher would never write, and its scores there carry little information; NVIDIA’s hypothesis about the full-vocabulary objective rests on this. Long agent episodes add a cost problem. Mixing them with single-turn reasoning prompts left Ultra’s training inefficient “because rollout times can differ dramatically”, so “for most of the agentic tasks” NVIDIA used “single-turn rollouts”: the student writes one turn after a recorded history. Moonshot says its distillation reward fits K3’s partial rollouts, which continue unfinished long episodes across steps (S5).

Xiaomi’s MiMo-V2.6 makes recorded histories a method, Multi-Prefix MOPD (MOPD2 there; at NVIDIA, “MOPD2” names the checkpoint after the second round). It keeps full student rollouts where an RL-trained teacher exists, and adds prefix-conditioned ones chiefly for its SFT teachers on open-domain tasks. Each multi-turn trajectory from a teacher, or from SFT data, gives one prefix before each assistant turn: “The student samples one new turn from each prefix without regenerating earlier interactions”, and the teacher scores that turn. The student then trains in states the teacher has seen, at the price of being less on-policy for the earlier turns.

Prompts no teacher owns

An environment with no specialist would otherwise get no signal, and the student may drift on it. Nemotron 3.5 Super routes such prompts “to a self-anchor, a frozen copy of the student’s starting checkpoint, which provides no signal while the student still matches it and regularizes departures from it”. On those prompts it is S5’s KL penalty to a reference model, applied to a share of the batch: 5% in the first stage, beside agentic (35%), reasoning (25%), alignment (20%) and vision tool use (15%). The anchor can only keep what the student already had.

Benchmarks that peak and then fall

NVIDIA reports that under distillation, benchmarks “improve at different rates”, and that “some peak within a few steps and then decline with further training”. Super’s answer is two short stages. The first, weighted toward fast-moving domains, runs 30 steps of 512 prompts. The second retires the agentic and reasoning teachers, folds their environments into the self-anchor (now the stage-1 model) and runs 40 steps of 1,024 prompts, one pass: 56,320 prompts in all (computed).

The second stage also softens the teacher into a proximal teacher, after Xie et al.’s trust-region distillation, which keeps each update close to the current student. The teacher is replaced by the mixture απE+(1−α)πold\alpha \pi_E + (1-\alpha)\pi_{\text{old}} with α=0.2\alpha = 0.2, where πold\pi_{\text{old}} is the student as it was when the batch was sampled, held fixed during the update. Since the mixture is at least (1−α)πold(1-\alpha)\pi_{\text{old}}, the advantage can’t fall below log⁡(1−α)≈−0.22\log(1-\alpha) \approx -0.22: a token the teacher considers impossible gets a bounded negative signal. Near agreement the mixture also shrinks every signal to about α\alpha times its size, so part of the effect is a smaller step. In the toy above, the hedge’s advantage moves from −2.71-2.71 to −0.21-0.21, and “42“‘s from +0.56+0.56 to +0.14+0.14. Distilling toward this mixture is, with a frozen copy of the student inside it, the skew reverse KL of Ko et al. (2024) at skew 0.8 in their notation (our reading; NVIDIA cites Xie et al.). Super’s pre-training distillation used the same divergence at skew 0.1, close to the teacher.

Stale rollouts

The stage runs on RL infrastructure, so it inherits RL’s staleness (S5): a rollout can come from weights a step or two old. Ultra handles this as S5’s RL does, reweighting each token by how much likelier the current weights make it than the weights that wrote it, clipped as in PPO around a “proximal policy” (S5’s anchor for the update, unrelated to the proximal teacher above), with IcePop’s mask. Super masks tokens “with weights outside [0.2, 5]”, and NVIDIA notes that this corrects “action probabilities at the sampled contexts”, not the prefix, and that masking introduces bias. A full-vocabulary version needs no token weight, since it computes each position exactly.

Teachers split by effort or by stage

Two of the seven labs split their teachers by something other than domain.

Mistral splits by effort. Length-penalized RL runs from the high-reasoning checkpoint produce medium- and low-effort teachers; “The high-effort teacher is the initialization checkpoint itself”. Distillation then puts three reasoning efforts into one model, and SFT needed only “a single reasoning effort”.

Z.ai’s GLM-5 uses its own earlier stages as teachers, in cross-stage distillation. Its RL runs in sequence, reasoning, then agentic, then general, and “sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities”. So “the final checkpoints from the preceding training stages serve as teacher models”, to recover “skills acquired in earlier SFT and RL stages”, with the sampled-token log-ratio in place of GRPO’s advantage, group size 1 and batch 1,024. DeepSeek V3.2 had addressed the same forgetting the other way, by merging “reasoning, agent, and human alignment training into one RL stage”.

Two labs outside the seven use related ideas for other purposes, both forms of context distillation, where the teacher is the same model given more or easier context. Aleph Alpha’s Kolibri has no distillation stage and lists MOPD as future work. Inside its single RL run it uses on-policy self-distillation: for a group of rollouts that all failed (best reward below 0.2) in eight tool-use and terminal environments, the teacher is the live policy given a remedial hint about the error, and the student is the same policy without it. The loss is a forward KL over the full vocabulary, and a group trained this way gets no RL loss. It touched 0.10% of tokens in those environments early in the run and 0.034% late. Qwen3.5-Omni’s “On-Policy Distillation (OPD)” stage uses the model’s response to a text query as “the distillation target for the corresponding audio-conditioned query”; no teacher scores student tokens.

Distillation or weight merging

The other way to fold specialists into one model is to average their weights. Xiaomi wrote for MiMo-V2-Flash that MOPD, unlike parameter merging, “preserves peak performance of the strongest teacher across all domains”, and DeepSeek V4 that distillation avoids “the performance degradation often encountered in traditional weight-merging or mixed RL techniques”. Neither report includes a comparison. Liquid’s LFM2 ended post-training with a merging stage, and LFM3 lists distillation and no merging; no Liquid document says one replaced the other. DeepSeek V4.1 does both: it uses “model merging to reinitialize successive RL runs” and then distills.

The one controlled comparison is outside our corpus. Xiaomi’s MOPD paper (Qwen3-30B-A3B; math, instruction following, software engineering) scores each method by the share of the gap from the SFT student to the teachers that it closes, averaged over domains: MOPD 0.937, a single mixed RL run 0.882, task-arithmetic merging 0.857, SFT on teacher outputs 0.824, RL run one domain after another 0.775, plain weight averaging 0.328. It is one model at one scale, with no repeats reported, from the lab that proposed the method, and MOPD’s arm, like the merging and SFT arms, builds on the RL runs that made its teachers.

How much of each teacher the student keeps

NVIDIA’s Ultra report is the one source with a column for every checkpoint. It defines “Recovery rate” as

recovery=MOPD2−RLVRTeacher−RLVR,\text{recovery} = \frac{\text{MOPD2} - \text{RLVR}}{\text{Teacher} - \text{RLVR}},

the share of the gap between the student before distillation (the RLVR checkpoint) and the domain’s teacher that two rounds of distillation close. It is a ratio of score differences, not an accuracy. On BrowseComp, the teacher leads by 20.0 points, the student gains 13.4, and recovery is 67% (computed, matching the printed 67.0%). On HLE, the gap is 6.5 points and the student gains 1.1: 17%.

A ratio over a small gap is fragile. The four most extreme rows, both above 100% and the two lowest, all have gaps of 6.5 points or less, where one point of score moves the ratio by 15 percentage points or more. The report gives no error bars or repeats.

SFTRLVRMOPD1MOPD2teacher

Recovery = (MOPD2 − RLVR) / (Teacher − RLVR), as NVIDIA prints it.

Figure 3.Nemotron 3 Ultra's checkpoints on eleven benchmarks: after SFT, after RL (RLVR), after one and two rounds of multi-teacher distillation, and the domain's teacher. Recovery is NVIDIA's share of the RLVR-to-teacher gap that distillation closed, with the gap in points beneath it. It runs from 17% on HLE to more than 100% on Terminal Bench and Multi-Challenge, where the student scored above its teacher by 4.0 and 0.5 points. Switching the baseline to SFT changes every percentage (HLE's 17% becomes 56%).

The report’s reading is that distillation works best “when the teacher’s advantage can be expressed as token-level preferences over trajectories that the student is already able to sample”. On HLE, the reasoning teacher had been trained further on data “the student has not directly seen”, and NVIDIA reads the small recovery as “a limitation of the on-policy distillation setting rather than a failure of the teacher”. Three rows the report leaves uncommented cut against a tidy reading: LiveCodeBench falls in the second round (90.0 to 89.0), GDPVal doesn’t move in it, and IMOAnswerBench’s RL checkpoint sits below its SFT one (84.5 against 85.1).

Both NVIDIA and the MOPD paper measure from the checkpoint distillation starts from: after RL at Ultra, after SFT in the MOPD paper, which has no RL stage in between. Measured from Ultra’s SFT checkpoint instead, HLE’s figure would be 56% rather than 17% (computed), but that credits distillation with what RL did first.

MiMo-V2-Flash’s Table 7 is the other same-model table, with the student before and after distillation (the starting checkpoint isn’t named; the pipeline implies SFT) and the best teacher per benchmark. The student ends below its best teacher on 4 of 12 benchmarks, by 6.3 points on BrowseComp, whose best teacher was an SFT model. On five benchmarks no specialist beat the student’s own starting checkpoint, and on two of them distillation left the student below where it started: GPQA-Diamond by 0.6, creative writing by 3.9. The abstract says MOPD enables “the student model to perfectly master teacher expertise”; the section text says “matches or exceeds the strongest teacher in most domains”, which is what the table shows.

Stage order across the series

As the series’ last stage, the article closes with each lab’s whole post-training order as its reports state it. A blank means we searched the lab’s documents and found no order.

Lab (model)Post-training order
Aleph Alpha (Kolibri)SFT → one RL run, with self-distillation for failed groups
DeepSeek (V3.2)per-domain specialists write data → SFT on it → one mixed RL run
DeepSeek (V4)per-domain specialists (SFT, then RL) → distillation from more than ten
DeepSeek (V4.1)SFT → successive RL runs, reinitialized by merging → distillation from over 40
Google DeepMind (Gemma 4)“similar post-training approach as in Gemma 3”
Liquid AI (LFM3)SFT → mixed RL → distillation from six → agentic RL
MiniMax
Mistral AI (Large 4)mid-training and SFT → RL → length-penalized RL teachers → distillation from three
Moonshot AI (K3)SFT → nine RL experts → distillation
NVIDIA (Nemotron 3 Ultra)SFT → RLVR → light SFT warm-up → (teachers → distillation) twice → MTP boosting
NVIDIA (Nemotron 3.5 Super / Lightning)SFT → RLVR and RLHF → teachers → distillation in two stages → MTP boosting → QAD / SFT → RLVR → MTP boosting
OpenAI (gpt-oss)“CoT RL techniques as OpenAI o3”; no order
Poolside (Laguna)mid-training → SFT → RL → QAD
Qwen (Qwen3.5-Omni)data from specialists → text-to-audio “OPD” → RL
StepFun
Tencent (Hy3)“joint optimization of SFT and RL”; no order
Thinking Machines (Inkling)SFT on synthetic data → large-scale RL
Xiaomi (MiMo-V2-Flash / V2.6)SFT → domain teachers → MOPD / mid-training → short SFT → one mixed RL run → Multi-Prefix MOPD
Z.ai (GLM-5)SFT → reasoning RL → agentic RL → general RL → cross-stage distillation

MTP boosting trains only the multi-token prediction head, which drafts tokens ahead for faster decoding, to match the frozen model; QAD (quantization-aware distillation) trains a low-precision copy to match the full one on fixed data. Neither adds a capability.

Figure 4.Post-training order for every lab whose documents state one. Distillation (plum) always follows RL (coral), whether the student's own or only the teachers'; at most labs it is the last stage that trains the whole model. DeepSeek V4, Kimi K3 and MiMo-V2-Flash run RL only on the teachers, and Liquid's LFM3 and Qwen3.5-Omni run RL after distillation. Block widths show order only, not time or tokens.

DeepSeek V3.2 and Z.ai answered forgetting across stages in opposite ways (above), and DeepSeek V4 then replaced V3.2’s mixed stage with distillation from per-domain specialists, the shape Kimi K3 and MiMo-V2-Flash also use. No report in our corpus compares two orders on one model; the MOPD paper’s scores above (one domain after another 0.775, mixed 0.882, distillation 0.937) are the only such comparison, on its 30B test model.

Kimi K3 and DeepSeek V4 also run quantization-aware training through post-training. DeepSeek’s covers “teacher and reference models” too; Kimi K3 doesn’t say whether its teachers are quantized.

Gains measured stage by stage

Most published gains compare model generations and can’t be assigned to a stage. Four sources print a measured checkpoint from before RL or distillation, SFT or implied SFT, beside a later one of the same model:

SourceModelBenchmark: SFT → RL (→ distillation)
Ultra report Table 5Nemotron 3 UltraTerminal Bench 2.0 34.5 → 44.5 → 54.0; BrowseComp 14.3 → 31.0 → 44.4; OmniScience non-hallucination 4.8 → 46.3 → 78.7; SWE-Bench Verified 63.5 → 65.8 → 71.7; IMOAnswerBench 85.1 → 84.5 → 88.6
MiMo-V2-Flash report Table 7MiMo-V2-FlashSFT → distillation, no mixed RL: AIME 2025 89.3 → 94.1; SWE-Bench Verified 67.8 → 73.4; BrowseComp 42.5 → 45.4; GPQA-Diamond 84.9 → 84.3 (starting checkpoint not named)
Kolibri report Table 25KolibriBrowseComp 22.0 → 29.4; IFBench 48.2 → 78.1; MMLU-Pro 78.5 → 80.0 (RL includes self-distillation)
Mistral blogMistral Large 4BrowseComp 75% → 84.5% (“our RL pipeline”; RL and distillation not split)

At Ultra, the share of the post-SFT gain that arrives during distillation ranges from 16% (HLE) to 76% (GDPVal) on ten of the eleven rows, about half on BrowseComp and Terminal Bench (computed); on IMOAnswerBench, where RL lowered the score, distillation brought more than the whole gain. That share is measured in this order; what distillation would add straight after SFT isn’t. Part of it is the teachers’ own RL, passed on. Kolibri’s RL raised all 13 benchmarks in its table, by 1.5 points (MMLU-Pro) to 36.0 (AA-Omniscience).

None of these sources reports a stage that gave nothing, and none is a randomized comparison; publications select for what shipped. The nulls that are reported are ablations of the stage’s variants (more rollouts, top-kk and full-vocabulary objectives, the warm-up on HLE), not of the stage itself.

Comparison

VariantWho writes the tokensTeacher signalMain cost or riskLabs
Sampled-token on-policy distillationstudentlog-ratio on the sampled tokenteacher pass per token; drops future terms; partial, uneven recoveryNVIDIA, Moonshot, Xiaomi, Z.ai
Full-vocabulary on-policy distillationstudentexact KL per positionlogits memory; NVIDIA’s preliminary runs found it worse on agentic tasksDeepSeek, Mistral
Proximal teacherstudentlog-ratio to απE+(1−α)πold\alpha\pi_E + (1-\alpha)\pi_{\text{old}}weaker signal everywhere; bounded below by log⁡(1−α)\log(1-\alpha)NVIDIA (Super, stage 2)
Prefix-conditionedrecorded history up to a turn, student afterteacher on the new turnless on-policy for earlier turnsNVIDIA (Ultra, most agentic tasks), Xiaomi (V2.6)
Cross-stagestudentearlier-stage checkpointseach teacher is capped at its stage’s final checkpointZ.ai
Self-distillation in RLstudentthe policy with a hint, forward KLneeds a useful hint; few tokensAleph Alpha

Where the labs sit

Seven of the 16 labs run a post-training stage in which a teacher scores the student’s own samples: DeepSeek, Liquid, Mistral, Moonshot, NVIDIA, Xiaomi and Z.ai. Five of them split teachers by domain (all but Mistral and Z.ai). Blank cells mean searched and not stated.

LabTeachersObjectiveStudent starts fromRollouts per stepAfter it
DeepSeek (V4 / V4.1)>10 / >40, by domainfull-vocabulary reverse KLnot stated / after RLnone
Liquid AI (LFM3)6, by domainafter mixed RLagentic RL
Mistral AI (Large 4)3, by effortfull-vocabulary reverse KLhigh-effort RL checkpointnone
Moonshot AI (K3)9, domain × effortsampled token, clippednone
NVIDIA (Ultra / Super)10 per round, 2 rounds / 4 per stage, 2 stages, plus selfsampled tokenRLVR checkpoint / RL (RLVR and RLHF) checkpoint1,024 × 1 / 512 × 1, then 1,024 promptsMTP boosting / MTP boosting, QAD
Xiaomi (V2-Flash / V2.6)domain RL, SFT and self teacherssampled token + outcome reward / standard plus prefix-conditionedSFT / after mixed RLnone
Z.ai (GLM-5)earlier stages’ checkpointssampled token in GRPOafter general RL (implied)1,024 × 1none

Aleph Alpha distills inside RL; Qwen uses the name for a text-to-audio stage; Poolside distills only for quantization. Google DeepMind, MiniMax, OpenAI, StepFun, Tencent and Thinking Machines state no distillation in post-training. Thinking Machines wrote the post that DeepSeek, NVIDIA, Moonshot and Mistral cite for the method, and its Inkling documents describe SFT and RL only.

Easy to get wrong

  • Reading “distillation replaces RL”. The teachers are mostly RL-trained, so the stage passes on RL gains rather than doing without them. DeepSeek V4, Kimi K3 and MiMo-V2-Flash run RL only on the teachers; MiMo-V2.6, Mistral, Liquid, NVIDIA and DeepSeek V4.1 distill a student that was itself RL-trained, Z.ai after sequential RL stages, and Liquid runs agentic RL after distillation.
  • Comparing recovery figures across sources without their baselines. Both NVIDIA and the MOPD paper start from the student that enters distillation, an RL checkpoint at NVIDIA and an SFT one in the MOPD paper; HLE’s 17% becomes 56% if Ultra’s is moved back to SFT.
  • Taking “specialist distillation” for this stage. At DeepSeek V3.2 and in Qwen3.5-Omni’s first stage, specialists write SFT data.
  • Treating “on-policy” and “reverse KL” as one choice. GKD’s headline on-policy results used forward KL, and Kolibri’s self-distillation is on-policy with a forward KL.
  • Comparing “MOPD2” across labs. At NVIDIA it is a checkpoint, at Xiaomi a method.

Check yourself

Ultra’s BrowseComp scores are 14.3 after SFT, 31.0 after RLVR, 44.4 after two rounds of distillation, and 51.0 for the teacher. What is the recovery with NVIDIA’s baseline, and with the SFT baseline?

With RLVR: (44.4−31.0)/(51.0−31.0)=13.4/20.0=67%(44.4 - 31.0)/(51.0 - 31.0) = 13.4/20.0 = 67\%, the printed value. With SFT: (44.4−14.3)/(51.0−14.3)=30.1/36.7=82%(44.4 - 14.3)/(51.0 - 14.3) = 30.1/36.7 = 82\%. The second credits distillation with what RL did too.

At one position the student samples a token with probability 0.5 that the teacher gives 0.001. What advantage does the token get under plain distillation, and under Super’s stage-2 proximal teacher with α = 0.2?

Plain: ln⁡0.001−ln⁡0.5=−6.21\ln 0.001 - \ln 0.5 = -6.21. Proximal: ln⁡(0.2×0.001+0.8×0.5)−ln⁡0.5=ln⁡(0.4002/0.5)=−0.223\ln(0.2 \times 0.001 + 0.8 \times 0.5) - \ln 0.5 = \ln(0.4002/0.5) = -0.223, essentially the bound ln⁡0.8\ln 0.8. The signal still says “less of this”, but no longer dominates the update.

Ultra’s RLVR stage drew 16 rollouts per prompt, its distillation stage one. Why can distillation do with one?

GRPO needs several rollouts of the same prompt to estimate a baseline and turn rewards into advantages. In distillation, the teacher’s probability of each token is the reference, so one rollout already gives every token an unbiased signal; more rollouts would only average its noise, and NVIDIA found they “did not yield additional benefits”.

In the map

Methods

Taxonomy nodes

Further reading

  • Agarwal et al., On-policy distillation of language models (GKD), ICLR 2024, arXiv:2306.13649: the imitation-learning argument, and divergences compared on student samples.
  • Gu et al., MiniLLM, ICLR 2024, arXiv:2306.08543: reverse KL as a policy-gradient problem, with the exact single-step term, teacher-mixed sampling and length normalization.
  • Lu and Thinking Machines Lab, On-policy distillation, doi:10.64434/tml.20251026: the dense/sparse framing, the zero-discount choice and compute comparisons at their scale.
  • Ma et al., MOPD, arXiv:2606.30406: the controlled comparison with mixed RL, merging and SFT on teacher data, top-kk objectives and the external-teacher failure.
  • Xie et al., Trust region policy distillation, arXiv:2607.04751: the proximal teacher and its variance bound. Liao et al., arXiv:2607.04763: replayed prefixes and the “prefix trap”.
  • Z. Yang et al., Nemotron-Cascade 2, arXiv:2603.19220 §4.4: distillation inside a staged RL pipeline, with step counts.
  • Hinton et al., arXiv:1503.02531, and Ko et al., DistiLLM, arXiv:2402.03898: logit distillation and the skew reverse KL. Rusu et al., Policy distillation, ICLR 2016: several teachers into one policy, before language models.
  • The labs’ accounts: Nemotron 3 Ultra report §3.3 (the most detailed, with Tables 4 and 5), Nemotron 3.5 report §3.6, DeepSeek V4 §5.1 to §5.2 and V4.1 §5, MiMo-V2-Flash §4 and MiMo-V2.6 §5.6, Kimi K3 §4.1, Mistral Large 4 §4.5, GLM-5 §3.5, Kolibri §3.2.1.
↑