single source · training objective
Decoupled Top-K knowledge distillation
A knowledge distillation objective that trains a student using only the teacher's Top-K logits per token.1 It decomposes the KL into a binary term matching the total probability mass on the Top-K set and a temperature-scaled conditional KL within it, avoiding the support mismatch of naive Top-K truncation.2
Trade-off It avoids the support mismatch and unstable losses of naive Top-K distillation while preserving the benefits of temperature-scaled distillation.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we distill the teacher distribution for each token given its context to a student , using only the Top-K teacher logits per token
— LFM2 Technical Reportdecomposing the KL via the chain rule into (i) a binary term that matches the total probability mass assigned to the Top-K set, and (ii) a conditional KL within the Top-K, to which we apply temperature
— LFM2 Technical ReportThis “decoupled, tempered Top-K” objective avoids support mismatch while preserving the benefits of temperature-scaled distillation
— LFM2 Technical Report
Also called Decoupled Top-K objective.
Filed under training objective › distillation objective