Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionblock selection

single mention · model architecture

Block-level selection

Block-level selection is a sparse-attention method in which each query selects a small number of KV blocks to attend to, instead of every token.1,2 Attention is then computed on the real (uncompressed) K/V, and queries in the same GQA group share one top-k selection so one set of KV blocks can be loaded in a single pass.3,4

Trade-off It keeps the full expressive power of softmax attention, but the KV cache doesn't shrink along with the attention sparsification.5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. block-level selection like CSA but attention is computed on the real K/V — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  2. During decode, each query interacts only with the selected KV blocks — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  3. block-level selection, but attention computed on the real K/V — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  4. query heads within the same GQA group share a single top-k selection — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram
  5. The cost is that the KV cache doesn't shrink along with attention sparsification — MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram

Filed under model architecture › token mixer › sparse attention › block selection