Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

specific method · filed under model architecture

Qwen Sparse Attention

Qwen Sparse Attention is a sparse attention method that uses a lightweight indexer to compress the sequence into micro-block representations and estimate their importance.1 It selects the most relevant context for attention computation, reducing indexing overhead on long inputs while balancing task performance and inference efficiency.2

Trade-off QSA trades a small amount of cache capacity for much lower long-context compute and memory traffic.3

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. QSA employs a lightweight indexer that compresses the sequence into micro-block representations, estimates their importance, and selects the most relevant context for attention computation — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. selects the most relevant context for attention computation. This design reduces indexing overhead on long inputs while balancing task performance and inference efficiency — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  3. QSA trades a small amount of cache capacity for much lower long-context compute and memory traffic — Qwen3.8-Flash-Next: Day-0 Support in SGLang

Also called Qwen Sparse Attention (QSA).

sources
6
model
1
lab adopt it
1
strongest
core

Documented in

Alibaba (Qwen)

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
core 6

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted6 sources

Qwen Sparse Attention (QSA) uses a compressed lightweight indexer to select the important context at micro-block granularity

coremodel architecturein Qwen3.8-Flash-NextQwen Team
5 more from this model

QSA operates at the micro-block level.

coreunclearin Qwen3.8-Flash-NextQwen Team

QSA replaces the Gated Attention layers of Qwen3.5's hybrid with a sparse mechanism that "operates at the micro-block level" rather than per token

coremodel architecturein Qwen3.8-Flash-NextQwen Team

it aggregates the sequence into micro-blocks, estimates importance at the block level, and then selects the most relevant regions

coreunclearin Qwen3.8-Flash-NextQwen Team

the fourth uses Qwen Sparse Attention for precise long-range retrieval.

coremodel architecturein Qwen3.8-Flash-NextQwen Team

Qwen3.8-Flash-Next adopts Qwen Sparse Attention (QSA).

coreunclearin Qwen3.8-Flash-NextQwen Team
↑