Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

single mention · model architecture

Fixed-budget sparse-attention selection

Sparse selection constrained to a fixed budget of blocks or tokens, after which attention cost stops growing with context length.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention for the methods filed beside it.

Also called fixed budget of 512 blocks or 2048 tokens.

Filed under model architecture › token mixer › sparse attention