single source · model architecture
Indexer warmup
Indexer warmup is a two-stage training schedule that initializes a sparse-attention indexer before it controls token selection.1 The model first runs full attention in both branches while the index projections are trained, then switches to sparse top-k selection over the chosen positions.2,3
Trade-off Warmup gives better short-context and long-context retrieval than starting sparse selection from step zero.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We use a two-stage training schedule to initialise the Index Branch and avoid early random selections
— MiniMax Sparse AttentionDuring the first few iterations, the model runs full attention in both branches and trains the newly added index projections
— MiniMax Sparse AttentionAfter steps, we enable top- sparse selection and continue training
— MiniMax Sparse AttentionThe warmed-up run achieves better short-context performance and stronger long-context retrieval
— MiniMax Sparse Attention
Also called indexer warmup with full attention.
Filed under model architecture › token mixer › sparse attention › sparse attention indexer