single mention · not yet filed
Micro-block compressed lightweight indexer
The sources on this page only mention it in passing.
Related methods
Used for
Methods the sources say this one serves — as a part, a setting, a variant or an enabler.
- Qwen Sparse Attentionpart of
QSA employs a lightweight indexer that compresses the sequence into micro-block representations, estimates their importance, and selects the most relevant context for attention computation.
Supported by
Methods the sources say are used for this one.
- Average poolingimplements it
keys are partitioned into non-overlapping blocks of tokens and compressed by average pooling
- Block-causal scoringimplements it
Block-level importance scores are then obtained through block-causal scoring.
We first distill the full-sequence attention distribution of the backbone into the indexer.
- Partial rotary position embeddingsconfigures it
For positional encoding, the indexer applies partial RoPE. Specifically, partial RoPE is applied to 64 of the 128 dimensions in each indexer head