single mention · training objective
Hidden-state caching for KL distillation
Teacher final hidden states are cached so teacher and student logits can be reconstructed through the frozen output head.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See distillation objective for the methods filed beside it.
Filed under training objective › distillation objective