Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

single mention · model architecture

CSA2 compressor and indexer design

The described CSA2 design removes overlap and absolute positional embeddings and projects indexer keys from main KV entries.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention indexer for the methods filed beside it.

Filed under model architecture › token mixer › sparse attention › sparse attention indexer