Open Models Almanac
Methodstraining objectivedistillation objective

single source · training objective

Decoupled Top-K knowledge distillation

A knowledge distillation objective that trains a student using only the teacher's Top-K logits per token.1 It decomposes the KL into a binary term matching the total probability mass on the Top-K set and a temperature-scaled conditional KL within it, avoiding the support mismatch of naive Top-K truncation.2

Trade-off It avoids the support mismatch and unstable losses of naive Top-K distillation while preserving the benefits of temperature-scaled distillation.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we distill the teacher distribution for each token given its context to a student , using only the Top-K teacher logits per token — LFM2 Technical Report
  2. decomposing the KL via the chain rule into (i) a binary term that matches the total probability mass assigned to the Top-K set, and (ii) a conditional KL within the Top-K, to which we apply temperature — LFM2 Technical Report
  3. This “decoupled, tempered Top-K” objective avoids support mismatch while preserving the benefits of temperature-scaled distillation — LFM2 Technical Report

Also called Decoupled Top-K objective.

Filed under training objective › distillation objective