Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

specific method · filed under model architecture

Compressed Sparse Attention

Compressed Sparse Attention is an attention mechanism that compresses the KV cache of blocks of consecutive tokens into single weighted entries.1 It then applies DeepSeek Sparse Attention so that each query token attends only to the compressed KV entries, with uncompressed KV kept for the tail tokens of the compression branch.2,3

Trade-off Its compression makes long-context attention substantially cheaper, at the price of unstored uncompressed KV entries that must be recomputed on cache hits.4,5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. CSA integrates both compression and sparse attention strategies: it first compresses the Key-Value (KV) cache of every tokens into one entry — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. and then applies DeepSeek Sparse Attention (DSA) [dsv32] where each query token attends to only compressed KV entries — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. along with the uncompressed tail tokens from the compression branch — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  4. which substantially reduces the computational cost of attention in long-text scenarios — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  5. as uncompressed KV entries in CSA and HCA are not stored — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called CSA, Compressed Sparse Attention (CSA).

sources
6
models
3
lab adopt it
1
strongest
core

Documented in

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 3core 2not used 1

DeepSeek-V4-Pro

DeepSeekadopted4 sources

we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

coremodel architecturein DeepSeek-V4 seriesDeepSeek
3 more from this model

Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4 seriesDeepSeek

CSA/HCA cuts KV cache to 10% of V3.2 levels at 1M context

coremodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V4-Flash

DeepSeekadopted3 sources

we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

coremodel architecturein DeepSeek-V4 seriesDeepSeek
2 more from this model

Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4-FlashDeepSeek

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑