implementation detail · filed under training objective
Hidden-state caching for KL distillation
An implementation that caches the teacher’s final hidden states and reconstructs teacher and student logits through a frozen output head.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
We cache final hidden states of the teacher and reconstruct teacher and student logits through the frozen output head.
usedpost trainingin Laguna XS.2Poolside
Filed alongside
Other methods under training objective :: distillation objective.