single source · training objective
Conditional Top-K-only temperature scaling
Conditional Top-K-only temperature scaling applies temperature only to the conditional Top-K distributions during knowledge distillation.1 By conditioning both teacher and student on the Top-K set and tempering only these conditional distributions, the method removes support mismatch.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We apply temperature only to the conditional Top-K distributions and , i.e., we replace the conditional KL in Eq. (11) by , while keeping both the Bernoulli term and the scalar weight untempered
— LFM2 Technical ReportConditioning both teacher and student on and tempering only these conditional distributions makes them both converge to uniform on , removing this support mismatch
— LFM2 Technical Report
Also called Temperature placement.
Filed under training objective › distillation objective