Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionblock selection

single mention · model architecture

Dynamic sparse selection

Dynamic sparse selection lets each query choose which tokens to attend to based on content, within a fixed sparse selection budget.1 Unlike position-fixed patterns such as a sliding window, the selected tokens are content-dependent, which better matches agent tasks.2

Trade-off It outperforms a FLOP-matched sliding-window baseline in perplexity at the same selection budget.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the selected tokens are fixed by position or chosen dynamically — MiniMax Sparse Attention
  2. a position-fixed sparse pattern is less suitable than content-dependent token selection — MiniMax Sparse Attention
  3. Under the same sparse selection budget, the sliding-window model has higher perplexity than MSA — MiniMax Sparse Attention

Filed under model architecture › token mixer › sparse attention › block selection