single mention · model architecture
Index value head
An index value head is an extra attention output attached to the sparse attention indexer's Index Branch.1 Its role was to give an early training signal so sparse training can start from scratch; it is not needed once indexer warmup is used.2
Trade-off It helps initial sparse training but adds computation and complexity, and the final design drops it for efficiency.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
providing an additional attention output through the Index Branch
— MiniMax Sparse Attentionprovide an additional early training signal, rather than to supply essential capacity at convergence
— MiniMax Sparse Attentionthis index value head introduces additional computation and complexity
— MiniMax Sparse Attention
Also called Index Branch value head.
Filed under model architecture › token mixer › sparse attention › sparse attention indexer