Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

implementation detail · filed under model architecture

Dense attention warm-up with a frozen lightning indexer

Dense attention warm-up with a frozen lightning indexer is a short training stage that initializes the lightning indexer while the model keeps dense attention.1 The indexer is trained with a KL-divergence loss against a target distribution built by summing the main attention scores across heads and L1-normalizing them along the sequence dimension.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We first use a short warm-up stage to initialize the lightning indexer. In this stage, we keep dense attention and freeze all model parameters except for the lightning indexer — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. we first aggregate the main attention scores by summing across all attention heads. This sum is then L1-normalized along the sequence dimension to produce a target distribution . Based on , we set a KL-divergence loss as the training objective of the indexer — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

Also called Dense Warm-up Stage.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2
↑