general family · filed under model architecture
Sparse attention
Sparse attention factors causal attention into an indexer that selects which keys to attend to, followed by attention computed only over the selected keys.1 Selections are made per query position (or per GQA-group query head) rather than for the whole layer, so each query computes standard softmax attention restricted to its own index set.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
A sparse attention layer factors causal attention into an indexer that selects which keys to attend to and a sparse attention computation over the selected keys
— MiniMax Sparse AttentionIn multi-head attention, each query, specified by a position and a query head , can select a different key/value index set
— MiniMax Sparse Attention
Also called sparse attention layers, sparse attention throughout.