implementation detail · filed under post-training
Per-Token On-Policy Distillation Reward
A per-token reward computed from the teacher-to-student probability ratio and clipped to a bounded range.
Also called per-token OPD reward.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
the per-token OPD reward evaluated on yt between the teacher π(d,e)teacher and the student πθ is defined as: rdopd(yt |e, x, y<t) = clip(sg(log π(d,e)teacher(yt |x, y<t)/πθ(yt |e, x, y<t)), −Rmax, Rmax)
usedunclearin Kimi K3Moonshot AI
Filed alongside
Other methods under post-training :: policy distillation.
Multi-Teacher On-Policy DistillationOn-Policy DistillationPrefix-Conditioned On-Policy DistillationMulti-Prefix Multi-Teacher On-Policy DistillationSpecialist DistillationAutonomous Student RolloutsLarge-Scale On-Policy DistillationOn-Policy Cross-Stage DistillationAsynchronous Multi-Teacher On-Policy DistillationBehavior–Proximal Policy DecouplingDistillation Fine-Tuning on MiMo-Generated DataDistillation for Post-Training Data GenerationIcePop Token-Level Loss MaskingModel DistillationMulti-Objective Policy DistillationOff-Policy DistillationSFT–RL–On-Policy Distillation PipelineTeacher-Trajectory and SFT-History Reuse