methods · token mixer
sparse attention
60 methods filed at this node or below it, from the sources of 13 models.
In practice
Checked as “Indexer-selected sparse attention (DeepSeek Sparse Attention)” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| Mistral-Large-4 | in its code | adopted | index_topk=2048 · index_n_heads=32 · index_head_dim=128A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict attention, DeepSeek Sparse Attention style. text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128 | code → |
| MiniMax-M3.1-Flash | in its code | not stated | use_sparse_attention=true · sparse_topk_blocks=16A lightning indexer (MiniMaxM3VLIndexer) scores keys per block and each query attends only to its top index_topk_blocks blocks, enabled on every layer via sparse_attention_freq all set to 1. text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_topk_blocks = 16 | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | index_topk=512 · index_n_heads=32 · index_head_dim=128An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and each query attends only to the index_topk selected tokens, configured by index_n_heads/index_head_dim/index_topk. text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128 | code → |
| Step-5-Preview | in its code | not stated | enabled=true · topk=512 · apply_to_layer_types=["full_attention"]A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers. text_config.sparse_config.enabled = truetext_config.sparse_config.topk = 512text_config.sparse_config.apply_to_layer_types = ["full_attention"] | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | index_topk=512 · index_n_heads=32 · index_head_dim=128An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query, configured with index_n_heads/index_head_dim, DeepSeek Sparse Attention style. text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128 | code → |
| Hy4-preview | in its code | not stated | index_topk=2048 · index_n_heads=32 · use_dsa=trueAn HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention. index_topk = 2048index_n_heads = 32use_dsa = true | code → |
| GLM-5.3-Flash | in its code | not stated | index_topk=2048 · index_n_heads=32 · index_head_dim=128Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer. text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128 | code → |
| Qwen3.8-Flash-Next | in its code | not stated | indexer_n_heads=4 · indexer_budget=2048 · indexer_compress_ratio=4A Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those tokens, enabled via indexer_n_heads/indexer_budget/indexer_compress_ratio in the config. text_config.indexer_n_heads = 4text_config.indexer_budget = 2048text_config.indexer_compress_ratio = 4 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | index_topk=512 · index_n_heads=64 · index_head_dim=128DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type. index_topk = 512index_n_heads = 64index_head_dim = 128 | code → |
| GLM-5.3 | in its code | not stated | index_topk=2048 · index_n_heads=32 · index_head_dim=128A GlmMoeDsaIndexer scores past tokens with lightweight projections and selects the top index_topk=2048 tokens per query, which is turned into the attention mask (DeepSeek Sparse Attention). index_topk = 2048index_n_heads = 32index_head_dim = 128 | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | index_topk=1024 · index_n_heads=64 · index_head_dim=128DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk index_topk = 1024index_n_heads = 64index_head_dim = 128 | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | index_topk=512 · index_n_heads=64 · index_head_dim=128The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer. index_topk = 512index_n_heads = 64index_head_dim = 128 | code → |
| GLM-5.2 | in its code | adopted | index_topk=2048 · index_n_heads=32 · index_head_dim=128A lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens it selects (DeepSeek Sparse Attention). index_topk = 2048index_n_heads = 32index_head_dim = 128 | code → |
| MiniMax-M3 | in its code | discussed | use_sparse_attention=true · sparse_attention_freq=[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1…A lightning indexer scores past tokens in blocks and selects top-k blocks per query, enabled on most layers via sparse_attention_freq/layer_types. text_config.sparse_attention_config.use_sparse_attention = truetext_config.sparse_attention_config.sparse_attention_freq = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1… | code → |
| DeepSeek-V4-Flash | in its code | adopted | index_topk=512 · index_n_heads=64 · index_head_dim=128DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention. index_topk = 512index_n_heads = 64index_head_dim = 128 | code → |
| DeepSeek-V4-Pro | in its code | adopted | index_topk=1024 · index_n_heads=64 · index_head_dim=128DeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query for CSA layers. index_topk = 1024index_n_heads = 64index_head_dim = 128 | code → |
| DeepSeek-V3.2 | in its code | adopted | index_topk=2048 · index_n_heads=64 · index_head_dim=128A DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into the attention computation (DeepSeek Sparse Attention). index_topk = 2048index_n_heads = 64index_head_dim = 128 | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo indexer scoring module or top-k token selection mechanism exists in the provided code. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | not in its code | not stated | verdict and linesNo index_topk, index_n_heads, or index_head_dim keys appear anywhere in the config, and no indexer code is present in the given files. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesNo indexer scoring module or top-k token selection mechanism exists in the attention code; config has no index_topk/index_n_heads keys. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesNo indexer scoring or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesNo indexer scoring module or top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present. | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesNo indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are present. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config exists in this model. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesNo indexer scoring module or index_topk/index_n_heads config keys exist; attention is dense GQA/sliding-window softmax attention only. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesNo indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code. | code → |
| Inkling | not in its code | not stated | verdict and linesNo indexer-based top-k token selection mechanism (no index_topk/index_n_heads/index_head_dim) is present anywhere in the attention code. | code → |
| Hy3 | not in its code | not stated | verdict and linesNo indexer module or top-k token selection mechanism exists in the attention code, and there are no index_topk-related config keys. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesNo indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no index_topk/index_n_heads/index_head_dim keys. | code → |
| Step-3.7-Flash | code unclear | not stated | verdict and linesThe config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config keys exist; attention is standard dense softmax attention. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesNo indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo indexer-scoring or top-k token selection mechanism appears in the code or config (no index_topk/index_n_heads keys). | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesNo indexer scoring or top-k token selection mechanism (index_topk, index_n_heads) appears in config or code. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo indexer or top-k token selection mechanism (index_topk/index_n_heads) appears anywhere in the code or config. | code → |
| Qwen3.5-397B-A17B | not in its code | adopted | verdict and linesNo index_topk / indexer module is present in the code or config. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo indexer-based top-k token selection mechanism is present in any attention module. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo indexer module or top-k token selection mechanism is present in this attention implementation | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo indexer-related config keys (index_topk, index_n_heads, index_head_dim) or code implementing a top-k token selector are present. | code → |
In this branch 60
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 24
15 from one source, or only mentioned as not used
sparse attention indexer 24
17 from one source, or only mentioned as not used
fixed-pattern sparse attention 3
all mentioned once or only as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.