Model techniques map
Techniquesmodel architecturetoken mixersparse attention

specific method · filed under model architecture

Compressed Sparse Attention

A sparse-attention method that selects blocks and computes attention using the corresponding real keys and values.

Also called CSA, Compressed Sparse Attention (CSA).

sources
6
models
3
lab adopt it
1
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 3core 2not used 1

Documented in

DeepSeek-V4-FlashMiniMax-M3DeepSeek-V4-Pro

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Evidence

6 spans quoted from the sources, strongest treatment first.

we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

coremodel architecturein DeepSeek-V4DeepSeek
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence·DeepSeek·2026-04-26

CSA/HCA cuts KV cache to 10% of V3.2 levels at 1M context

coremodel architecturein DeepSeek-V4DeepSeek
DeepSeek V4 Guide: Engram Memory, Training Data Strategy & Release Status (2026)·Kili Technology·2026-03-23

Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek
deepseek V4 model card EN·DeepSeek

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4-FlashDeepSeek
deepseek-ai/DeepSeek-V4-Flash · Hugging Face·DeepSeek·2026-04-26

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4DeepSeek
deepseek-ai/DeepSeek-V4-Pro · Hugging Face·DeepSeek·2026-04-26

block-level selection like CSA but attention is computed on the real K/V.

not usedmodel architecturein MiniMax-M3MiniMax
MiniMax Goes Sparse: Decoding M3's Attention from a Single Diagram·Hugging Face community blog·2026-05-29

Filed alongside

Other methods under model architecture :: token mixer :: sparse attention.

DeepSeek Sparse AttentionQwen Sparse AttentionSparse attentionHeavily Compressed AttentionGated DeepSeek Sparse AttentionCSA2 Full ModeKV-outer sparse attentionProgressive sequence-length extension for sparse attentionSparse-attention continued pre-training with joint model and indexer optimizationCross-layer KV and index reuse with statically assigned CSA2 modesFixed-budget sparse-attention selectionFrom-scratch sparse attention training without dense warmupJoint backbone and indexer training under sparse attentionNative Sparse AttentionNatively trained sparsityNoPE sparse multi-head latent attentionQSA micro-block compression at ratio 4Sequential block processingSparse retrieval over long contextsSparse softmax attentionToken-wise compressionTwo-stage introduction of sparse attentionTwo-stage sparse attention