single mention · training objective
Dense-attention distillation to initialize the QSA indexer
Dense distillation is a training objective that initializes the QSA indexer by distilling the backbone's full-sequence attention distribution into it.1 The teacher distribution sums softmax attention over teacher heads, is max-pooled to the block level, and is matched to indexer scores by a KL loss on complete key blocks.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We first distill the full-sequence attention distribution of the backbone into the indexer
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityThe token-level teacher distribution is obtained by summing the softmax attention distributions over all teacher heads and applying normalization
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Also called dense distillation.
Filed under training objective › distillation objective