Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionblock selection

single source · model architecture

Main Branch

The Main Branch is the attention branch of MSA that computes standard softmax attention restricted to the key blocks selected by the Index Branch.1 For each query head it applies scaled dot-product attention to the causally visible tokens of the selected blocks, using the key-value head of its GQA group, so per-query attention cost stays fixed as sequence length increases.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the Main Branch computes softmax attention over the tokens in those blocks — MiniMax Sparse Attention
  2. the Main Branch attends only to the causally visible tokens in the selected blocks. For any query head , it applies standard scaled dot-product attention restricted to these tokens — MiniMax Sparse Attention

Filed under model architecture › token mixer › sparse attention › block selection