specific method · filed under model architecture
Qwen Sparse Attention
Qwen Sparse Attention is a sparse attention method that uses a lightweight indexer to compress the sequence into micro-block representations and estimate their importance.1 It selects the most relevant context for attention computation, reducing indexing overhead on long inputs while balancing task performance and inference efficiency.2
Trade-off QSA trades a small amount of cache capacity for much lower long-context compute and memory traffic.3
How this definition was written
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
QSA employs a lightweight indexer that compresses the sequence into micro-block representations, estimates their importance, and selects the most relevant context for attention computation
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilityselects the most relevant context for attention computation. This design reduces indexing overhead on long inputs while balancing task performance and inference efficiency
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityQSA trades a small amount of cache capacity for much lower long-context compute and memory traffic
— Qwen3.8-Flash-Next: Day-0 Support in SGLang
Also called Qwen Sparse Attention (QSA).