implementation detail · filed under model architecture
Dense attention warm-up with a frozen lightning indexer
Dense attention warm-up with a frozen lightning indexer is a short training stage that initializes the lightning indexer while the model keeps dense attention.1 The indexer is trained with a KL-divergence loss against a target distribution built by summing the main attention scores across heads and L1-normalizing them along the sequence dimension.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We first use a short warm-up stage to initialize the lightning indexer. In this stage, we keep dense attention and freeze all model parameters except for the lightning indexer
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelswe first aggregate the main attention scores by summing across all attention heads. This sum is then L1-normalized along the sequence dimension to produce a target distribution . Based on , we set a KL-divergence loss as the training objective of the indexer
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
Also called Dense Warm-up Stage.