Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

single mention · model architecture

Index value head

An index value head is an extra attention output attached to the sparse attention indexer's Index Branch.1 Its role was to give an early training signal so sparse training can start from scratch; it is not needed once indexer warmup is used.2

Trade-off It helps initial sparse training but adds computation and complexity, and the final design drops it for efficiency.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. providing an additional attention output through the Index Branch — MiniMax Sparse Attention
  2. provide an additional early training signal, rather than to supply essential capacity at convergence — MiniMax Sparse Attention
  3. this index value head introduces additional computation and complexity — MiniMax Sparse Attention

Also called Index Branch value head.

Filed under model architecture › token mixer › sparse attention › sparse attention indexer