Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

implementation detail · filed under model architecture

Sparse-attention continued pretraining with joint model and indexer optimization

A continued-pretraining stage that adapts a dense-attention model to sparse attention by introducing fine-grained token selection and optimizing all model parameters.1 The indexer and main model train jointly but separately: the indexer aligns its outputs with the main attention distribution over the selected tokens, while the main model optimizes only the language modeling loss.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to the sparse pattern of DSA — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. we also keep aligning the indexer outputs to the main attention distribution, but considering only the selected token set — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  3. The training signal of the indexer is from only , while the optimization of the main model is according to only the language modeling loss — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

Also called Sparse Training Stage.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

DeepSeek-V3.2

DeepSeekadopted2 sources

Following indexer warm-up, we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to the sparse pattern of DSA.

usedoptimizationin DeepSeek-V3.2DeepSeek
1 more from this model

Following indexer warm-up, we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to the sparse pattern of DSA.

usedoptimizationin DeepSeek-V3.2DeepSeek
↑