Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

methods · token mixer

sparse attention

60 methods filed at this node or below it, from the sources of 13 models.

In practice

Checked as “Indexer-selected sparse attention (DeepSeek Sparse Attention)” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
Mistral-Large-4in its codeadopted
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict attention, DeepSeek Sparse Attention style.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_mistral_large4.py · L273–274
if self.indexer is not None:
            topk_indices = self.indexer(
                hidden_states,
                position_embeddings,
                attention_mask[:, 0, :, :],
                position_ids,  # Kept for BC
                past_key_values=past_key_values,
            )  # [B, S, topk]
modeling_mistral_large4.py · L447–454
MiniMax-M3.1-Flashin its codenot stated
use_sparse_attention=true · sparse_topk_blocks=16

A lightning indexer (MiniMaxM3VLIndexer) scores keys per block and each query attends only to its top index_topk_blocks blocks, enabled on every layer via sparse_attention_freq all set to 1.

text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_topk_blocks = 16
topk = min(self.topk_blocks, num_key_blocks)
        topk_scores, topk_indices = block_scores.topk(topk, dim=-1)  # [B, H_idx, S_q, topk]
        return topk_indices.masked_fill(topk_scores == float("-inf"), -1)
modeling_minimax_m3_vl.py · L583–585
self.indexer = (
            MiniMaxM3VLIndexer(config, layer_idx) if config.layer_types[layer_idx] == "minimax_m3_sparse" else None
        )
modeling_minimax_m3_vl.py · L416–418
DeepSeek-V4.1-Flash-1017in its codenot stated
index_topk=512 · index_n_heads=32 · index_head_dim=128

An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and each query attends only to the index_topk selected tokens, configured by index_n_heads/index_head_dim/index_topk.

text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128
get_indexer_topk(op.topk_backend)(
        logits,
        seq_lens,
        1,
        topk_rows,
        op.topk_tokens,
        plan.max_seq_len,
    )
indexer.py · L378–385
Step-5-Previewin its codenot stated
enabled=true · topk=512 · apply_to_layer_types=["full_attention"]

A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers.

text_config.sparse_config.enabled = truetext_config.sparse_config.topk = 512text_config.sparse_config.apply_to_layer_types = ["full_attention"]
topk = values["topk"]
    if not 1 <= topk <= _STEP5_SPARSE_TOPK_MAX:
        raise ValueError(f"Step5 DSA requires topk in [1, 1024], got {topk}.")
config.py · L148–150
self.use_dsa_backend = sparse_config is not None and sliding_window is None
model.py · L1008
DeepSeek-V4.1-Flashin its codenot stated
index_topk=512 · index_n_heads=32 · index_head_dim=128

An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query, configured with index_n_heads/index_head_dim, DeepSeek Sparse Attention style.

text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128
op.topk_indices_buffer[:num_tokens] = -1
    schedule = decode.schedule_metadata
indexer.py · L260–261
Hy4-previewin its codenot stated
index_topk=2048 · index_n_heads=32 · use_dsa=true

An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention.

index_topk = 2048index_n_heads = 32use_dsa = true
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_hy_v4.py · L257–258
self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else HYV4Indexer(config, layer_idx)
modeling_hy_v4.py · L369–370
GLM-5.3-Flashin its codenot stated
index_topk=2048 · index_n_heads=32 · index_head_dim=128

Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
        select_k = min(self.index_topk // self.index_kpool, index_scores.shape[-1])

        # Selection is based on 2 steps
        #   1. The actual scores (selected indices)
        #   2. And removing invalid rows based on padding (selected valid)
        selected = index_scores.topk(select_k, dim=-1).indices
modeling_glm5_next.py · L884–889
        self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else Glm5NextTextIndexer(config, layer_idx)
modeling_glm5_next.py · L1150–1151
Qwen3.8-Flash-Nextin its codenot stated
indexer_n_heads=4 · indexer_budget=2048 · indexer_compress_ratio=4

A Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those tokens, enabled via indexer_n_heads/indexer_budget/indexer_compress_ratio in the config.

text_config.indexer_n_heads = 4text_config.indexer_budget = 2048text_config.indexer_compress_ratio = 4
self.index_n_heads = config.indexer_n_heads
        self.index_kv_heads = config.indexer_kv_heads
        self.index_head_dim = config.indexer_head_dim
        self.token_budget = config.indexer_budget
        self.compress_ratio = config.indexer_compress_ratio
        self.block_topk = self.token_budget // self.compress_ratio
modeling_qwen4_exp.py · L669–674
selected_block_indices = scores.topk(min(self.block_topk, num_complete_blocks), dim=0).indices
modeling_qwen4_exp.py · L747
DeepSeek-V4-Flash-Vision-Expin its codenot stated
index_topk=512 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type.

index_topk = 512index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
modeling_deepseek_v4.py · L467–469
top_k = min(self.index_topk, compressed_len)
modeling_deepseek_v4.py · L580
GLM-5.3in its codenot stated
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A GlmMoeDsaIndexer scores past tokens with lightweight projections and selects the top index_topk=2048 tokens per query, which is turned into the attention mask (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 32index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_glm_moe_dsa.py · L241–242
DeepSeek-V4-Pro-0813in its codenot stated
index_topk=1024 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk

index_topk = 1024index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
        return (scores * weights.unsqueeze(-1)).sum(dim=2)  # [B, S, T]
modeling_deepseek_v4.py · L467–470
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L593
DeepSeek-V4-Flash-0731in its codenot stated
index_topk=512 · index_n_heads=64 · index_head_dim=128

The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer.

index_topk = 512index_n_heads = 64index_head_dim = 128
index_scores = index_scores.masked_fill(future_mask, float("-inf"))
            top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L592–593
GLM-5.2in its codeadopted
index_topk=2048 · index_n_heads=32 · index_head_dim=128

A lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens it selects (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 32index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_glm_moe_dsa.py · L241–242
MiniMax-M3in its codediscussed
use_sparse_attention=true · sparse_attention_freq=[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1…

A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via sparse_attention_freq/layer_types.

text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_attention_freq = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
topk = min(self.topk_blocks, num_key_blocks)
        topk_scores, topk_indices = block_scores.topk(topk, dim=-1)  # [B, H_idx, S_q, topk]
        return topk_indices.masked_fill(topk_scores == float("-inf"), -1)
modeling_minimax_m3_vl.py · L583–585
self.indexer = (
            MiniMaxM3VLIndexer(config, layer_idx) if config.layer_types[layer_idx] == "minimax_m3_sparse" else None
        )
modeling_minimax_m3_vl.py · L416–418
DeepSeek-V4-Flashin its codeadopted
index_topk=512 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention.

index_topk = 512index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L593–595
DeepSeek-V4-Proin its codeadopted
index_topk=1024 · index_n_heads=64 · index_head_dim=128

DeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query for CSA layers.

index_topk = 1024index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L593–595
DeepSeek-V3.2in its codeadopted
index_topk=2048 · index_n_heads=64 · index_head_dim=128

A DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into the attention computation (DeepSeek Sparse Attention).

index_topk = 2048index_n_heads = 64index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_deepseek_v32.py · L240–241
LFM3-1.7Bnot in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the provided code.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
verdict and lines

No index_topk, index_n_heads, or index_head_dim keys appear anywhere in the config, and no indexer code is present in the given files.

Kolibri-1not in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the attention code; config has no index_topk/index_n_heads keys.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

No indexer scoring or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

No indexer scoring module or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present.

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

No indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are present.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config exists in this model.

Laguna-S-2.1not in its codenot stated
verdict and lines

No indexer scoring module or index_topk/index_n_heads config keys exist; attention is dense GQA/sliding-window softmax attention only.

Kimi K3not in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code.

Inklingnot in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism (no index_topk/index_n_heads/index_head_dim) is present anywhere in the attention code.

Hy3not in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism exists in the attention code, and there are no index_topk-related config keys.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no index_topk/index_n_heads/index_head_dim keys.

Step-3.7-Flashcode unclearnot stated
verdict and lines

The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active.

# `layer_types` is the canonical per-layer attention dispatch: it tells
        # `DynamicCache(config=...)` which layers want the sparse cache and tells
        # `Step3p7Attention` which layers build a sparse Lightning Indexer.
configuration_step3p7.py · L277–279
MolmoAct 2not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config keys exist; attention is standard dense softmax attention.

MiMo-V2.5not in its codenot stated
verdict and lines

No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No indexer-scoring or top-k token selection mechanism appears in the code or config (no index_topk/index_n_heads keys).

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

No indexer scoring or top-k token selection mechanism (index_topk, index_n_heads) appears in config or code.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No indexer or top-k token selection mechanism (index_topk/index_n_heads) appears anywhere in the code or config.

Qwen3.5-397B-A17Bnot in its codeadopted
verdict and lines

No index_topk / indexer module is present in the code or config.

X-VLAnot in its codenot stated
verdict and lines

No indexer-based top-k token selection mechanism is present in any attention module.

gpt-oss-120bnot in its codenot stated
verdict and lines

No indexer module or top-k token selection mechanism is present in this attention implementation

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No indexer-related config keys (index_topk, index_n_heads, index_head_dim) or code implementing a top-k token selector are present.

In this branch 60

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 24

DeepSeek Sparse Attention core · 16 sources · 18 quotes
Sparse attention core · 8 sources · 8 quotes
Compressed Sparse Attention core · 6 sources · 6 quotes
Qwen Sparse Attention core · 6 sources · 6 quotes
Heavily Compressed Attention core · 5 sources · 5 quotes
Gated DeepSeek Sparse Attention core · 3 sources · 3 quotes
Compressed Sparse Attention 2 core · 2 sources · 3 quotes
KV-outer sparse attention used · 2 sources · 2 quotes
15 from one source, or only mentioned as not used
CSA2 Full, Reindex, and Reuse modes core · 1 source · 1 quote
Fixed-budget sparse-attention selection core · 1 source · 1 quote
NoPE sparse multi-head latent attention core · 1 source · 1 quote
QSA micro-block compression at ratio 4 core · 1 source · 1 quote
Token-wise compression core · 1 source · 1 quote
FlashMLA sparse attention default · 1 source · 1 quote
Natively trained sparsity used · 1 source · 1 quote
Sparse softmax attention used · 1 source · 1 quote
Two-stage sparse-attention formulation used · 1 source · 1 quote
Two-stage sparse-attention training used · 1 source · 1 quote
Native Sparse Attention not used · 1 source · 1 quote

block selection 9

MiniMax Sparse Attention core · 7 sources · 8 quotes
Top-k block selection used · 2 sources · 2 quotes
7 single mentions
Block-level selection core · 1 source · 1 quote
Dynamic sparse selection core · 1 source · 1 quote
Main Branch core · 1 source · 2 quotes
Top-512 block selection core · 1 source · 1 quote
Block Max Pooling used · 1 source · 1 quote
Top-scoring compressed-block selection used · 1 source · 1 quote
Mixture of Block Attention evaluated · 1 source · 1 quote

sparse attention indexer 24

IndexCache core · 3 sources · 3 quotes
IndexShare core · 3 sources · 3 quotes
Lightning Indexer core · 3 sources · 3 quotes
Fine-grained token selection mechanism core · 2 sources · 2 quotes
Hierarchical Sparse Indexer core · 2 sources · 3 quotes
17 from one source, or only mentioned as not used
Compressed lightweight indexer core · 1 source · 1 quote
CSA2 compressor and indexer design core · 1 source · 1 quote
DeepSeek Sparse Attention indexer core · 1 source · 1 quote
Index Branch core · 1 source · 2 quotes
IndexPool core · 1 source · 1 quote
MQA indexer core · 1 source · 1 quote
Average pooling used · 1 source · 1 quote
Block-causal scoring used · 1 source · 1 quote
Detached indexer-input optimization used · 1 source · 1 quote
Indexer warmup used · 1 source · 2 quotes
KL-distillation indexer warm-up used · 1 source · 1 quote
Single-head index key used · 1 source · 1 quote
Top-2,048 key-token selection per query used · 1 source · 1 quote
Index Branch output not used · 1 source · 1 quote
Index value head not used · 1 source · 1 quote

fixed-pattern sparse attention 3

all mentioned once or only as not used
Sparse Global Attention Anchors core · 1 source · 1 quote
Local Block used · 1 source · 1 quote
Forced Sink and Local Window not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled hereblock selectionsparse attention indexerfixed-pattern sparse attention
GLM-5.3-Flash coreNoPE sparse multi-head latent attention core——IndexPool core——
DeepSeek-V4.1-Flash coreCompressed Sparse Attention 2 coreCSA2 Full, Reindex, and Reuse modes coreProgressive context-length extension for sparse attention usedSparse attention training from scratch without dense-attention warmup used——CSA2 compressor and indexer design coreHierarchical Sparse Indexer core——
Hy4-preview coreGated DeepSeek Sparse Attention core——IndexCache core——
MiniMax-M3.1-Flash coreSparse attention coreAggressive sparsity for inference efficiency used—MiniMax Sparse Attention core———
NVIDIA-Nemotron-3-Ultra-550B-A55B core———Sparse Global Attention Anchors core—
DeepSeek-V4-Flash coreCompressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage sparse-attention training used————
DeepSeek-V3.2 coreDeepSeek Sparse Attention coreSparse-attention continued pretraining with joint model and indexer optimization used——Fine-grained token selection mechanism coreLightning Indexer coreDense attention warm-up with a frozen lightning indexer usedDetached indexer-input optimization used——
DeepSeek-V4-Pro coreCompressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage sparse-attention training used————
GLM-5.2 coreDeepSeek Sparse Attention coreSparse attention used——IndexShare coreLightning Indexer used——
MiniMax-M3 coreSparse attention coreKV-outer sparse attention usedNatively trained sparsity usedSparse softmax attention usedTwo-stage sparse-attention formulation usedDeepSeek Sparse Attention evaluatedCompressed Sparse Attention not usedNative Sparse Attention not used—Block-level selection coreDynamic sparse selection coreMain Branch coreMiniMax Sparse Attention coreBlock Max Pooling usedTop-k block selection usedMixture of Block Attention evaluated—Index Branch coreIndexer warmup usedSingle-head index key usedIndex Branch output not usedIndex value head not used—Local Block usedForced Sink and Local Window not used—
Mistral-Large-4 coreSparse attention coreFlashMLA sparse attention defaultDeepSeek Sparse Attention used——DeepSeek Sparse Attention indexer coreMulti-head latent attention with a DSA sparse indexer coreKL-distillation indexer warm-up usedShared DSA indexer across sliding-window layers usedTop-2,048 key-token selection per query used——
Qwen3.5-397B-A17B coreDeepSeek Sparse Attention core————
Qwen3.8-Flash-Next coreFixed-budget sparse-attention selection coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreJoint backbone and indexer training under sparse attention used—Top-512 block selection coreTop-scoring compressed-block selection used—Compressed lightweight indexer coreMQA indexer coreReuse QSA index selection across speculative decoding steps coreAverage pooling usedBlock-causal scoring used——
↑