Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

single mention · model architecture

Joint backbone and indexer training under sparse attention

A training procedure that enables QSA and jointly trains the backbone and indexer during continued pretraining.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention for the methods filed beside it.

Also called sparse training.

Filed under model architecture › token mixer › sparse attention