Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

specific method · filed under model architecture

DeepSeek Sparse Attention

DeepSeek Sparse Attention is a sparse attention mechanism with two components: a lightning indexer and a fine-grained token selection mechanism.1 The indexer computes index scores between a query token and preceding tokens, and each query attends only to the key-value entries with the top-k index scores.2

Trade-off It substantially reduces computational complexity while preserving model performance, especially in long-context scenarios.3

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. The prototype of DSA primarily consists of two components: a lightning indexer and a fine-grained token selection mechanism — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. our fine-grained token selection mechanism retrieves only the key-value entries corresponding to the top-k index scores — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  3. an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios — deepseek-ai/DeepSeek-V3.2 · Hugging Face

Also called DSA, DSA (DeepSeek Sparse Attention), DeepSeek Sparse Attention (DSA), Dense Sparse Attention.

sources
16
models
7
labs adopt it
4
strongest
core

In practice

Checked as “Indexer-selected sparse attention (DeepSeek Sparse Attention)” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codeadopted
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict attention, DeepSeek Sparse Attention style.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_mistral_large4.py · L273–274
if self.indexer is not None:
            topk_indices = self.indexer(
                hidden_states,
                position_embeddings,
                attention_mask[:, 0, :, :],
                position_ids,  # Kept for BC
                past_key_values=past_key_values,
            )  # [B, S, topk]
modeling_mistral_large4.py · L447–454
MiniMax-M3.1-Flashin its codenot stated
use_sparse_attention=true · sparse_topk_blocks=16

A lightning indexer (MiniMaxM3VLIndexer) scores keys per block and each query attends only to its top index_topk_blocks blocks, enabled on every layer via sparse_attention_freq all set to 1.

text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_topk_blocks = 16
topk = min(self.topk_blocks, num_key_blocks)
        topk_scores, topk_indices = block_scores.topk(topk, dim=-1)  # [B, H_idx, S_q, topk]
        return topk_indices.masked_fill(topk_scores == float("-inf"), -1)
modeling_minimax_m3_vl.py · L583–585
self.indexer = (
            MiniMaxM3VLIndexer(config, layer_idx) if config.layer_types[layer_idx] == "minimax_m3_sparse" else None
        )
modeling_minimax_m3_vl.py · L416–418
DeepSeek-V4.1-Flash-1017in its codenot stated
index_topk=512 · index_n_heads=32 · index_head_dim=128

An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and each query attends only to the index_topk selected tokens, configured by index_n_heads/index_head_dim/index_topk.

text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128
get_indexer_topk(op.topk_backend)(
        logits,
        seq_lens,
        1,
        topk_rows,
        op.topk_tokens,
        plan.max_seq_len,
    )
indexer.py · L378–385
Step-5-Previewin its codenot stated
enabled=true · topk=512 · apply_to_layer_types=["full_attention"]

A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers.

text_config.sparse_config.enabled = truetext_config.sparse_config.topk = 512text_config.sparse_config.apply_to_layer_types = ["full_attention"]
topk = values["topk"]
    if not 1 <= topk <= _STEP5_SPARSE_TOPK_MAX:
        raise ValueError(f"Step5 DSA requires topk in [1, 1024], got {topk}.")
config.py · L148–150
self.use_dsa_backend = sparse_config is not None and sliding_window is None
model.py · L1008
DeepSeek-V4.1-Flashin its codenot stated
index_topk=512 · index_n_heads=32 · index_head_dim=128

An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query, configured with index_n_heads/index_head_dim, DeepSeek Sparse Attention style.

text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128
op.topk_indices_buffer[:num_tokens] = -1
    schedule = decode.schedule_metadata
indexer.py · L260–261
Hy4-previewin its codenot stated
index_topk=2048 · index_n_heads=32 · use_dsa=true

An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention.

index_topk = 2048index_n_heads = 32use_dsa = true
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_hy_v4.py · L257–258
self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else HYV4Indexer(config, layer_idx)
modeling_hy_v4.py · L369–370
GLM-5.3-Flashin its codenot stated
index_topk=2048 · index_n_heads=32 · index_head_dim=128

Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
        select_k = min(self.index_topk // self.index_kpool, index_scores.shape[-1])

        # Selection is based on 2 steps
        #   1. The actual scores (selected indices)
        #   2. And removing invalid rows based on padding (selected valid)
        selected = index_scores.topk(select_k, dim=-1).indices
modeling_glm5_next.py · L884–889
        self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else Glm5NextTextIndexer(config, layer_idx)
modeling_glm5_next.py · L1150–1151
Qwen3.8-Flash-Nextin its codenot stated
indexer_n_heads=4 · indexer_budget=2048 · indexer_compress_ratio=4

A Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those tokens, enabled via indexer_n_heads/indexer_budget/indexer_compress_ratio in the config.

text_config.indexer_n_heads = 4text_config.indexer_budget = 2048text_config.indexer_compress_ratio = 4
self.index_n_heads = config.indexer_n_heads
        self.index_kv_heads = config.indexer_kv_heads
        self.index_head_dim = config.indexer_head_dim
        self.token_budget = config.indexer_budget
        self.compress_ratio = config.indexer_compress_ratio
        self.block_topk = self.token_budget // self.compress_ratio
modeling_qwen4_exp.py · L669–674
selected_block_indices = scores.topk(min(self.block_topk, num_complete_blocks), dim=0).indices
modeling_qwen4_exp.py · L747
DeepSeek-V4-Flash-Vision-Expin its codenot stated
index_topk=512 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type.

index_topk = 512index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
modeling_deepseek_v4.py · L467–469
top_k = min(self.index_topk, compressed_len)
modeling_deepseek_v4.py · L580
GLM-5.3in its codenot stated
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A GlmMoeDsaIndexer scores past tokens with lightweight projections and selects the top index_topk=2048 tokens per query, which is turned into the attention mask (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 32index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_glm_moe_dsa.py · L241–242
DeepSeek-V4-Pro-0813in its codenot stated
index_topk=1024 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk

index_topk = 1024index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
        return (scores * weights.unsqueeze(-1)).sum(dim=2)  # [B, S, T]
modeling_deepseek_v4.py · L467–470
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L593
DeepSeek-V4-Flash-0731in its codenot stated
index_topk=512 · index_n_heads=64 · index_head_dim=128

The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer.

index_topk = 512index_n_heads = 64index_head_dim = 128
index_scores = index_scores.masked_fill(future_mask, float("-inf"))
            top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L592–593
GLM-5.2in its codeadopted
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens it selects (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 32index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_glm_moe_dsa.py · L241–242
MiniMax-M3in its codediscussed
use_sparse_attention=true · sparse_attention_freq=[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1…

A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via sparse_attention_freq/layer_types.

text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_attention_freq = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
topk = min(self.topk_blocks, num_key_blocks)
        topk_scores, topk_indices = block_scores.topk(topk, dim=-1)  # [B, H_idx, S_q, topk]
        return topk_indices.masked_fill(topk_scores == float("-inf"), -1)
modeling_minimax_m3_vl.py · L583–585
self.indexer = (
            MiniMaxM3VLIndexer(config, layer_idx) if config.layer_types[layer_idx] == "minimax_m3_sparse" else None
        )
modeling_minimax_m3_vl.py · L416–418
DeepSeek-V4-Flashin its codeadopted
index_topk=512 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention.

index_topk = 512index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L593–595
DeepSeek-V4-Proin its codeadopted
index_topk=1024 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query for CSA layers.

index_topk = 1024index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L593–595
DeepSeek-V3.2in its codeadopted
index_topk=2048 · index_n_heads=64 · index_head_dim=128

A DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into the attention computation (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 64index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_deepseek_v32.py · L240–241
LFM3-1.7Bnot in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the provided code.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
verdict and lines

No index_topk, index_n_heads, or index_head_dim keys appear anywhere in the config, and no indexer code is present in the given files.

Kolibri-1not in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the attention code; config has no index_topk/index_n_heads keys.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

No indexer scoring or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present.

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

No indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are present.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config exists in this model.

Laguna-S-2.1not in its codenot stated
verdict and lines

No indexer scoring module or index_topk/index_n_heads config keys exist; attention is dense GQA/sliding-window softmax attention only.

Kimi K3not in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code.

Inklingnot in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism (no index_topk/index_n_heads/index_head_dim) is present anywhere in the attention code.

Hy3not in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism exists in the attention code, and there are no index_topk-related config keys.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no index_topk/index_n_heads/index_head_dim keys.

Step-3.7-Flashcode unclearnot stated
verdict and lines

The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active.

# `layer_types` is the canonical per-layer attention dispatch: it tells
        # `DynamicCache(config=...)` which layers want the sparse cache and tells
        # `Step3p7Attention` which layers build a sparse Lightning Indexer.
configuration_step3p7.py · L277–279
MolmoAct 2not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config keys exist; attention is standard dense softmax attention.

MiMo-V2.5not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No indexer-scoring or top-k token selection mechanism appears in the code or config (no index_topk/index_n_heads keys).

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

No indexer scoring or top-k token selection mechanism (index_topk, index_n_heads) appears in config or code.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No indexer or top-k token selection mechanism (index_topk/index_n_heads) appears anywhere in the code or config.

Qwen3.5-397B-A17Bnot in its codeadopted
verdict and lines

No index_topk / indexer module is present in the code or config.

X-VLAnot in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism is present in any attention module.

gpt-oss-120bnot in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism is present in this attention implementation

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No indexer-related config keys (index_topk, index_n_heads, index_head_dim) or code implementing a top-k token selector are present.

Documented in

Evidence

18 quotes from 16 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 8core 7not used 2

DeepSeek-V4-Pro

DeepSeekadopted4 sources

CSA then applies DeepSeek Sparse Attention (DSA) to select top-k compressed KV entries for core attention

usedmodel architecturein DeepSeek-V4 seriesDeepSeek
3 more from this model

CSA compresses KV caches along the sequence dimension and applies DeepSeek Sparse Attention (DSA)

usedmodel architecturein DeepSeek-V4DeepSeek

the attention mechanism is an original CSA/HCA hybrid rather than the leaked 'MODEL1' sparse attention

not usedmodel architecturein DeepSeek-V4DeepSeek

Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)

coremodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V3.2

DeepSeekadopted4 sources

We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios.

coremodel architecturein DeepSeek-V3.2DeepSeek
4 more from this model

DeepSeek-V3.2: the only architectural modification of DeepSeek-V3.1-Terminus is the introduction of DeepSeek Sparse Attention (DSA) through continued training.

coremodel architecturein DeepSeek-V3.2DeepSeek

DeepSeek-V3.2 uses exactly the same architecture as DeepSeek-V3.2-Exp. Compared with DeepSeek-V3.1-Terminus, the last version of DeepSeek-V3.1, the only architectural modification of DeepSeek-V3.2 is the introduction of DeepSeek Sparse Attention (DSA) through continued training.

coremodel architecturein DeepSeek-V3.2DeepSeek

Using DSA, we addressed critical computation complexity without sacrificing long-context performance.

usedunclearin DeepSeek-V3.2DeepSeek

fine-grained sparse attention mechanism that reduces training and inference cost while preserving quality in long-context scenarios

coremodel architecturein DeepSeek-V3.2DeepSeek

GLM-5.2

Z.ai (Zhipu AI)adopted3 sources

we adopt DSA (DeepSeek Sparse Attention) [9], a novel architectural innovation that significantly reduces both training and inference costs.

coremodel architecturein GLM-5Z.ai
3 more from this model

For DSA warmup stage, the learning rate goes down from 5e-3 to 2e-4. For DSA sparse adaption stage, we use a constant learning rate of 1e-5.

usedoptimizationin GLM-5Z.ai

GLM-5 also integrates DeepSeek Sparse Attention (DSA), largely reducing deployment cost while preserving long-context capacity.

usedmodel architecturein GLM-5Z.ai

using DeepSeek Sparse Attention (DSA) as its base attention mechanism

usedmodel architecturein GLM-5.2Z.ai

DeepSeek-V4-Flash

DeepSeekadopted3 sources

CSA then applies DeepSeek Sparse Attention (DSA) to select top-k compressed KV entries for core attention

usedmodel architecturein DeepSeek-V4 seriesDeepSeek
2 more from this model

CSA compresses KV caches along the sequence dimension and applies DeepSeek Sparse Attention (DSA)

usedmodel architecturein DeepSeek-V4DeepSeek

Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)

coremodel architecturein DeepSeek-V4DeepSeek

Mistral-Large-4

Mistral AIadopted1 source

we adapted ML4 to DSA, using lightweight indexers to select the top-2,048 key tokens for each query

usedmodel architecturein Mistral Large 4Mistral AI

MiniMax-M3

MiniMaxdiscussed2 sources

Compared with approaches like DSA and MoBA, MSA can partition the KV into blocks more precisely

evaluatedevaluation onlyin MiniMax-M3MiniMax
1 more from this model

DeepSeek V3.2 DSA: MLA + token-level top-k

not usedmodel architecturein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑