Open Models Almanac
Methodstraining objectivedistillation objective

single source · training objective

Conditional Top-K-only temperature scaling

Conditional Top-K-only temperature scaling applies temperature only to the conditional Top-K distributions during knowledge distillation.1 By conditioning both teacher and student on the Top-K set and tempering only these conditional distributions, the method removes support mismatch.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We apply temperature only to the conditional Top-K distributions and , i.e., we replace the conditional KL in Eq. (11) by , while keeping both the Bernoulli term and the scalar weight untempered — LFM2 Technical Report
  2. Conditioning both teacher and student on and tempering only these conditional distributions makes them both converge to uniform on , removing this support mismatch — LFM2 Technical Report

Also called Temperature placement.

Filed under training objective › distillation objective