single mention · model architecture
Dynamic sparse selection
Dynamic sparse selection lets each query choose which tokens to attend to based on content, within a fixed sparse selection budget.1 Unlike position-fixed patterns such as a sliding window, the selected tokens are content-dependent, which better matches agent tasks.2
Trade-off It outperforms a FLOP-matched sliding-window baseline in perplexity at the same selection budget.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
the selected tokens are fixed by position or chosen dynamically
— MiniMax Sparse Attentiona position-fixed sparse pattern is less suitable than content-dependent token selection
— MiniMax Sparse AttentionUnder the same sparse selection budget, the sliding-window model has higher perplexity than MSA
— MiniMax Sparse Attention
Filed under model architecture › token mixer › sparse attention › block selection