Open Models Almanac
Methodspost-trainingpolicy distillation

general family · filed under post-training

On-Policy Distillation

On-Policy Distillation is a post-training method that transfers the capabilities of teacher (expert) models into a single student model.1 The student learns from the teachers' output distributions on trajectories the student itself generates, optimizing a reverse KL loss so each relevant expert guides the current task context.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. an effective post-training paradigm for efficiently transferring the knowledge and capabilities of domain experts to a single, unified model — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. having the student learn from the output distributions of teacher models on its own generated trajectories — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. Computing the reverse KL loss requires sampling training trajectories from the student to maintain on-policy learning — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called on-policy distillation (OPD).

sources
6
models
4
labs adopt it
3
strongest
core

Documented in

Evidence

8 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 7core 1

DeepSeek-V4-Pro

DeepSeekadopted3 sources

we employ multi-teacher On-Policy Distillation (OPD; lu2025onpolicydistillation, minillm) as the primary technique for merging expert capabilities into the final model.

usedpost trainingin DeepSeek-V4DeepSeek
4 more from this model

a single unified model is trained through on-policy distillation

usedpost trainingin DeepSeek-V4 seriesDeepSeek

post-training pipeline replaces reinforcement learning with On-Policy Distillation from 10+ domain specialists

corepost trainingin DeepSeek-V4DeepSeek

the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation

usedpost trainingin DeepSeek-V4DeepSeek

unified model consolidation via on-policy distillation

usedpost trainingin DeepSeek-V4 seriesDeepSeek

DeepSeek-V4-Flash

DeepSeekadopted2 sources

we employ multi-teacher On-Policy Distillation (OPD; lu2025onpolicydistillation, minillm) as the primary technique for merging expert capabilities into the final model.

usedpost trainingin DeepSeek-V4DeepSeek
3 more from this model

a single unified model is trained through on-policy distillation

usedpost trainingin DeepSeek-V4 seriesDeepSeek

the mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation

usedpost trainingin DeepSeek-V4DeepSeek

unified model consolidation via on-policy distillation

usedpost trainingin DeepSeek-V4-FlashDeepSeek

Qwen3.5-397B-A17B

Alibaba (Qwen)adopted1 source

we introduce a second-stage training procedure based on on-policy distillation (OPD), with the goal of distilling the model’s stronger response capabilities under text inputs into the audio-input setting.

usedpost trainingin Qwen3.5-OmniQwen Team

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑