Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

single mention · model architecture

Progressive context-length extension for sparse attention

Sparse attention is trained at 64K sequence length and the context is later extended to 1M tokens.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention for the methods filed beside it.

Filed under model architecture › token mixer › sparse attention