Model techniques map
Techniquesmodel architecturetoken mixersparse attention

specific method · filed under model architecture

Qwen Sparse Attention

A sparse-attention method that aggregates tokens into micro-blocks, estimates their importance, and selects relevant regions for attention.

Also called Qwen Sparse Attention (QSA).

sources
6
model
1
lab adopt it
1
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

core 6

Documented in

Qwen3.8-Flash-Next

Evidence

6 spans quoted from the sources, strongest treatment first.

it aggregates the sequence into micro-blocks, estimates importance at the block level, and then selects the most relevant regions

coreunclearin Qwen3.8-Flash-NextQwen
Qwen3.8-Flash-Next: Day-0 Support in SGLang·LMSYS·2026-08-26

QSA replaces the Gated Attention layers of Qwen3.5's hybrid with a sparse mechanism that "operates at the micro-block level" rather than per token

coremodel architecturein Qwen3.8-Flash-NextQwen
Qwen3.8-Flash-Next — Architecture, Evals & Inference Performance·SemiAnalysis·2026-08-26

Qwen3.8-Flash-Next adopts Qwen Sparse Attention (QSA).

coreunclearin Qwen3.8-Flash-NextQwen
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability·Qwen·2026-08-31

the fourth uses Qwen Sparse Attention for precise long-range retrieval.

coremodel architecturein Qwen3.8-Flash-NextQwen
Qwen3.8-Flash-Next on vLLM — Serve command for H200, H100, GB200 NVL4, GB300 NVL4, MI355X, RTX Pro 6000 4x, DGX Station (GB300)·vLLM·2026-09-17

QSA operates at the micro-block level.

coreunclearin Qwen3.8-Flash-NextQwen
Qwen/Qwen3.8-Flash-Next · Hugging Face·Qwen·2026-08-26

Qwen Sparse Attention (QSA) uses a compressed lightweight indexer to select the important context at micro-block granularity

coremodel architecturein Qwen3.8-Flash-NextQwen
Qwen3.8-Flash-Next·Qwen

Filed alongside

Other methods under model architecture :: token mixer :: sparse attention.

DeepSeek Sparse AttentionCompressed Sparse AttentionSparse attentionHeavily Compressed AttentionGated DeepSeek Sparse AttentionCSA2 Full ModeKV-outer sparse attentionProgressive sequence-length extension for sparse attentionSparse-attention continued pre-training with joint model and indexer optimizationCross-layer KV and index reuse with statically assigned CSA2 modesFixed-budget sparse-attention selectionFrom-scratch sparse attention training without dense warmupJoint backbone and indexer training under sparse attentionNative Sparse AttentionNatively trained sparsityNoPE sparse multi-head latent attentionQSA micro-block compression at ratio 4Sequential block processingSparse retrieval over long contextsSparse softmax attentionToken-wise compressionTwo-stage introduction of sparse attentionTwo-stage sparse attention