single mention · model architecture
Native Sparse Attention
Native Sparse Attention is a sparse attention method that uses top-k selection to decide which KV blocks each query attends to.1 It runs three parallel paths — compress, select, and sliding window — with a learned gate, and query heads within a GQA group share a single top-k selection.2,3
Trade-off During decode, attending only to selected KV blocks cuts memory-bandwidth pressure on the KV cache by roughly an order of magnitude.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
TopK decides which KV blocks to keep for this layer and this GQA group
— MiniMax Goes Sparse: Decoding M3's Attention from a Single DiagramNSA originally has three parallel paths (compress + select + sliding window) plus a learned gate
— MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagramquery heads within the same GQA group share a single top-k selection
— MiniMax Goes Sparse: Decoding M3's Attention from a Single DiagramDuring decode, each query interacts only with the selected KV blocks, and the memory-bandwidth pressure on the KV cache drops by roughly an order of magnitude
— MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
Also called NSA.
Filed under model architecture › token mixer › sparse attention