Open Models Almanac
Methodspost-trainingpolicy distillation

single mention · post-training

Full-Vocabulary On-Policy Distillation with Heterogeneous Teachers

Full-vocabulary on-policy distillation is a post-training stage in which the student model is trained on datasets from all domains using teacher models.1 It is heterogeneous in that the best teacher for each domain may come from a different training stage, and teachers may differ architecturally from each other and from the student.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the final full-vocabulary OPD task is trained on datasets from all domains using over 40 teacher models — DeepSeek V41 Tech Report
  2. the best teacher for each domain may come from a different stage of model development. Moreover, the teacher models may differ architecturally from one another and from the student — DeepSeek V41 Tech Report

Also called Large-scale on-policy distillation.

Filed under post-training › policy distillation