specific method · filed under model architecture
DeepSeek Sparse Attention
A fine-grained sparse-attention method designed to reduce computation while preserving performance on long-context tasks.
Also called DSA, DSA (DeepSeek Sparse Attention), DeepSeek Sparse Attention (DSA), Dense Sparse Attention.
- sources
- 15
- models
- 6
- labs adopt it
- 3
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models paper arxiv.orgintroduces DSA (lightning indexer + top-k token selection)
- DeepSeek Sparse Attention - LLM Architecture Gallery explainer sebastianraschka.com
In the code
Checked as “Indexer-selected sparse attention (DeepSeek Sparse Attention)” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | A Glm5NextTextIndexer lightweight indexer scores pooled candidate tokens and each query attends only to the resulting top-k selection (index_topk=2048) in deepseek_sparse_attention layers. modeling_glm5_next.py#L885 |
| DeepSeek-V4.1-Flash | in its code | not stated | A DeepseekV4Indexer with dedicated heads/head_dim scores past (compressed) tokens and each query attends only to the indexer's selected top-k tokens (index_topk), matching the DeepSeek Sparse Attention / lightning indexer pattern. attention.py#L1245 |
| Hy4-preview | in its code | not stated | HYV4Indexer scores past tokens with a lightweight head and each query attends only to the index_topk selected tokens via a scattered sparse mask. modeling_hy_v4.py#L268 |
| DeepSeek-V4-Flash-0731 | in its code | not stated | DeepseekV4Indexer scores compressed KV entries and returns top index_topk=512 indices per query, used by CSA layers to build a sparse block_bias restricting attention. modeling_deepseek_v4.py#L585 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | No indexer-based top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. |
| MiMo-V2.6-Flash | not in its code | not stated | No indexer scoring or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. |
| DeepSeek-V4-Flash | in its code | core | A DeepseekV4Indexer scores compressed keys with ReLU(q·k) weighted sums and selects the top index_topk=512 entries per query for the CSA compressed-sparse-attention layers. modeling_deepseek_v4.py#L501 |
| MiMo-V2.5 | not in its code | not stated | No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase. |
| GLM-5.3 | in its code | not stated | A lightweight GlmMoeDsaIndexer scores past tokens per head and each query attends only to the index_topk selected tokens via an additive sparse mask, DeepSeek Sparse Attention style. modeling_glm_moe_dsa.py#L252 |
| Hy3 | not in its code | not stated | No indexer module or top-k token selection mechanism exists in the attention code; the config has no index_topk/index_n_heads keys |
| GLM-5.2 | in its code | core | GlmMoeDsaIndexer scores past tokens per head and each query attends only to the index_topk selected tokens via an additive sparse mask (DeepSeek Sparse Attention). modeling_glm_moe_dsa.py#L252 |
| MiniMax-M3 | in its code | evaluated | A lightning indexer scores keys per head and pools into blocks, selecting the top index_topk_blocks blocks per query for sparse attention on sparse_attention_freq==1 layers. modeling_minimax_m3_vl.py#L600 |
| DeepSeek-V3.2 | in its code | core | A DeepseekV32Indexer scores past tokens with lightweight projections and selects the top index_topk tokens per query, which are turned into a sparse attention mask. modeling_deepseek_v32.py#L253 |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | The CSA compressor uses a Lightning Indexer that scores compressed KV blocks and keeps only the top index_topk=512 per query (index_n_heads=64, index_head_dim=128), used in compressed_sparse_attention layers of this config. modeling_deepseek_v4.py#L585 |
| DeepSeek-V4-Pro | in its code | core | The CSA compressor runs a Lightning Indexer that scores compressed KV blocks and each query attends only to the top index_topk=1024 selected blocks modeling_deepseek_v4.py#L585 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | The DeepseekV4Indexer scores compressed KV entries with a lightweight scorer and each query attends only to the top index_topk entries selected, used in compressed_sparse_attention layers. modeling_deepseek_v4.py#L585 |
| Gemma 4 31B | not in its code | not stated | No indexer scoring mechanism or top-k token selection code exists in this file for the language model. |
| Inkling | not in its code | not stated | No indexer-based top-k token selection mechanism (no index_topk/index_n_heads config or code) is present; attention is dense within the (possibly windowed) causal region. |
| Kimi K3 | not in its code | not stated | No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code. |
| Laguna-S-2.1 | not in its code | not stated | No indexer-based top-k token selection mechanism appears anywhere in the attention or MoE code. |
| MiMo-V2.5-Pro | not in its code | not stated | No indexer-scoring or top-k token selection mechanism appears in the code or config (no index_topk/index_n_heads keys). |
| MiMo-V2.6-Pro | not in its code | not stated | No indexer scoring module or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | No indexer-based top-k token selection mechanism appears anywhere in the code or config keys. |
| Qwen3.5-397B-A17B | not in its code | core | No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase. |
| Qwen3.6-35B-A3B | not in its code | not stated | No indexer scoring module or top-k token selection mechanism (index_topk, index_n_heads) appears in the code or config. |
| Step-3.7-Flash | code unclear | not stated | The config carries legacy sparse-index fields (index_n_heads, index_head_dim, etc.) mapped from sparse_attention_config, and the config code references a 'minimax_m3_sparse' layer type and Lightning Indexer, but no indexer module implementation is present in the given files and this config's layer_types contains no such entries. configuration_step3p7.py#L280 |
| gpt-oss-120b | not in its code | not stated | No indexer-based top-k token selection mechanism exists in this attention implementation. |
Evidence
17 spans quoted from the sources, strongest treatment first.
We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios.
DeepSeek-V3.2: the only architectural modification of DeepSeek-V3.1-Terminus is the introduction of DeepSeek Sparse Attention (DSA) through continued training.
DeepSeek-V3.2 uses exactly the same architecture as DeepSeek-V3.2-Exp. Compared with DeepSeek-V3.1-Terminus, the last version of DeepSeek-V3.1, the only architectural modification of DeepSeek-V3.2 is the introduction of DeepSeek Sparse Attention (DSA) through continued training.
DeepSeek Sparse Attention
Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)
fine-grained sparse attention mechanism that reduces training and inference cost while preserving quality in long-context scenarios
we adopt DSA (DeepSeek Sparse Attention) [9], a novel architectural innovation that significantly reduces both training and inference costs.
GLM-5 also integrates DeepSeek Sparse Attention (DSA) to induce token-level sparsity on top of it.
CSA compresses KV caches along the sequence dimension and applies DeepSeek Sparse Attention (DSA)
GLM-5 also integrates DeepSeek Sparse Attention (DSA), largely reducing deployment cost while preserving long-context capacity.
CSA then applies DeepSeek Sparse Attention (DSA) to select top-k compressed KV entries for core attention
Using DSA, we addressed critical computation complexity without sacrificing long-context performance.
using DeepSeek Sparse Attention (DSA) as its base attention mechanism
For DSA warmup stage, the learning rate goes down from 5e-3 to 2e-4. For DSA sparse adaption stage, we use a constant learning rate of 1e-5.
Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely
DeepSeek V3.2 DSA: MLA + token-level top-k
the attention mechanism is an original CSA/HCA hybrid rather than the leaked 'MODEL1' sparse attention
Filed alongside
Other methods under model architecture :: token mixer :: sparse attention.