single mention · training objective
Chain-rule decomposition of Top-K KL
The Top-K KL is separated into a term matching total Top-K probability mass and a conditional KL within the Top-K set.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See distillation objective for the methods filed beside it.
Also called decomposing the KL via the chain rule.
Filed under training objective › distillation objective