single source · model architecture
Main Branch
The Main Branch is the attention branch of MSA that computes standard softmax attention restricted to the key blocks selected by the Index Branch.1 For each query head it applies scaled dot-product attention to the causally visible tokens of the selected blocks, using the key-value head of its GQA group, so per-query attention cost stays fixed as sequence length increases.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
the Main Branch computes softmax attention over the tokens in those blocks
— MiniMax Sparse Attentionthe Main Branch attends only to the causally visible tokens in the selected blocks. For any query head , it applies standard scaled dot-product attention restricted to these tokens
— MiniMax Sparse Attention
Filed under model architecture › token mixer › sparse attention › block selection