Open Models Almanac
Methodstraining objectivedistillation objective

single mention · training objective

Chain-rule decomposition of Top-K KL

The Top-K KL is separated into a term matching total Top-K probability mass and a conditional KL within the Top-K set.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See distillation objective for the methods filed beside it.

Also called decomposing the KL via the chain rule.

Filed under training objective › distillation objective