an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios — deepseek-ai/DeepSeek-V3.2 · Hugging Face
Also called DSA, DSA (DeepSeek Sparse Attention), DeepSeek Sparse Attention (DSA), Dense Sparse Attention.
A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict attention, DeepSeek Sparse Attention style.
if self.indexer is not None:
topk_indices = self.indexer(
hidden_states,
position_embeddings,
attention_mask[:, 0, :, :],
position_ids, # Kept for BC
past_key_values=past_key_values,
) # [B, S, topk]
A lightning indexer (MiniMaxM3VLIndexer) scores keys per block and each query attends only to its top index_topk_blocks blocks, enabled on every layer via sparse_attention_freq all set to 1.
An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and each query attends only to the index_topk selected tokens, configured by index_n_heads/index_head_dim/index_topk.
A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers.
An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query, configured with index_n_heads/index_head_dim, DeepSeek Sparse Attention style.
An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention.
Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer.
select_k = min(self.index_topk // self.index_kpool, index_scores.shape[-1])
# Selection is based on 2 steps
# 1. The actual scores (selected indices)
# 2. And removing invalid rows based on padding (selected valid)
selected = index_scores.topk(select_k, dim=-1).indices
A Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those tokens, enabled via indexer_n_heads/indexer_budget/indexer_compress_ratio in the config.
DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type.
A GlmMoeDsaIndexer scores past tokens with lightweight projections and selects the top index_topk=2048 tokens per query, which is turned into the attention mask (DeepSeek Sparse Attention).
DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk
The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer.
A lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens it selects (DeepSeek Sparse Attention).
DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention.
DeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query for CSA layers.
A DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into the attention computation (DeepSeek Sparse Attention).
No indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no index_topk/index_n_heads/index_head_dim keys.
The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active.
# `layer_types` is the canonical per-layer attention dispatch: it tells
# `DynamicCache(config=...)` which layers want the sparse cache and tells
# `Step3p7Attention` which layers build a sparse Lightning Indexer.
We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios.
DeepSeek-V3.2: the only architectural modification of DeepSeek-V3.1-Terminus is the introduction of DeepSeek Sparse Attention (DSA) through continued training.
DeepSeek-V3.2 uses exactly the same architecture as DeepSeek-V3.2-Exp. Compared with DeepSeek-V3.1-Terminus, the last version of DeepSeek-V3.1, the only architectural modification of DeepSeek-V3.2 is the introduction of DeepSeek Sparse Attention (DSA) through continued training.