Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

single mention · model architecture

Index Branch output

Index Branch output is a training mechanism that lets the sparse-attention Index Branch contribute an additional attention output to the layer.1 It attaches a value projection to the Index Branch and adds the result through a separate output projection, so the indexer learns block selection via next-token prediction.2

Trade-off It preserves short-context ability but gives little direct pressure on block selection, hurting long-context retrieval.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. The first mechanism lets the Index Branch contribute an additional attention output — MiniMax Sparse Attention
  2. we attach a value projection to the Index Branch and compute . This output is added to the layer output through a separate output projection — MiniMax Sparse Attention
  3. LM Loss only preserves short-context ability but performs poorly on long-context retrieval — MiniMax Sparse Attention

Also called Index Branch value head output.

Filed under model architecture › token mixer › sparse attention › sparse attention indexer