Open Models Almanac
Methodstraining objectivedistillation objective

single mention · training objective

Dense-attention distillation to initialize the QSA indexer

Dense distillation is a training objective that initializes the QSA indexer by distilling the backbone's full-sequence attention distribution into it.1 The teacher distribution sums softmax attention over teacher heads, is max-pooled to the block level, and is matched to indexer scores by a KL loss on complete key blocks.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We first distill the full-sequence attention distribution of the backbone into the indexer — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. The token-level teacher distribution is obtained by summing the softmax attention distributions over all teacher heads and applying normalization — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Also called dense distillation.

Filed under training objective › distillation objective