Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

single mention · model architecture

Native Sparse Attention

Native Sparse Attention is a sparse attention method that uses top-k selection to decide which KV blocks each query attends to.1 It runs three parallel paths — compress, select, and sliding window — with a learned gate, and query heads within a GQA group share a single top-k selection.2,3

Trade-off During decode, attending only to selected KV blocks cuts memory-bandwidth pressure on the KV cache by roughly an order of magnitude.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. TopK decides which KV blocks to keep for this layer and this GQA group — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  2. NSA originally has three parallel paths (compress + select + sliding window) plus a learned gate — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  3. query heads within the same GQA group share a single top-k selection — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  4. During decode, each query interacts only with the selected KV blocks, and the memory-bandwidth pressure on the KV cache drops by roughly an order of magnitude — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram

Also called NSA.

Filed under model architecture › token mixer › sparse attention