single mention · model architecture
Fixed-budget sparse-attention selection
Sparse selection constrained to a fixed budget of blocks or tokens, after which attention cost stops growing with context length.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention for the methods filed beside it.
Also called fixed budget of 512 blocks or 2048 tokens.
Filed under model architecture › token mixer › sparse attention