single mention · model architecture
Progressive context-length extension for sparse attention
Sparse attention is trained at 64K sequence length and the context is later extended to 1M tokens.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention for the methods filed beside it.
Filed under model architecture › token mixer › sparse attention