Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

single source · model architecture

Indexer warmup

Indexer warmup is a two-stage training schedule that initializes a sparse-attention indexer before it controls token selection.1 The model first runs full attention in both branches while the index projections are trained, then switches to sparse top-k selection over the chosen positions.2,3

Trade-off Warmup gives better short-context and long-context retrieval than starting sparse selection from step zero.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We use a two-stage training schedule to initialise the Index Branch and avoid early random selections — MiniMax Sparse Attention
  2. During the first few iterations, the model runs full attention in both branches and trains the newly added index projections — MiniMax Sparse Attention
  3. After steps, we enable top- sparse selection and continue training — MiniMax Sparse Attention
  4. The warmed-up run achieves better short-context performance and stronger long-context retrieval — MiniMax Sparse Attention

Also called indexer warmup with full attention.

Filed under model architecture › token mixer › sparse attention › sparse attention indexer