single mention · model architecture
Detached indexer-input optimization
Detaches the indexer input from the computational graph so the indexer can be optimized separately.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See sparse attention indexer for the methods filed beside it.
Filed under model architecture › token mixer › sparse attention › sparse attention indexer