implementation detail · filed under model architecture
Sparse-attention continued pretraining with joint model and indexer optimization
A continued-pretraining stage that adapts a dense-attention model to sparse attention by introducing fine-grained token selection and optimizing all model parameters.1 The indexer and main model train jointly but separately: the indexer aligns its outputs with the main attention distribution over the selected tokens, while the main model optimizes only the language modeling loss.2,3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to the sparse pattern of DSA
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelswe also keep aligning the indexer outputs to the main attention distribution, but considering only the selected token set
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsThe training signal of the indexer is from only , while the optimization of the main model is according to only the language modeling loss
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
Also called Sparse Training Stage.