Open Models Almanac
Methodsmodel architecturetoken mixersparse attention

specific method · filed under model architecture

Heavily Compressed Attention

Heavily Compressed Attention is an attention variant that compresses the KV cache more aggressively than CSA, using a larger compression rate and no overlapped compression.1,2 Each group of KV entries is merged into one compressed entry via compression weights and learnable positional biases, shortening the sequence before attention.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. compresses the KV cache in a heavier manner, but does not employ sparse attention — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. employs a larger compression rate () and does not perform overlapped compression — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. each KV entries in will be compressed into one according to the compression weights and learnable positional biases — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called Heavily Compressed Attention (HCA).

sources
5
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

5 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
used 3core 2

DeepSeek-V4-Pro

DeepSeekadopted4 sources

we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

coremodel architecturein DeepSeek-V4 seriesDeepSeek
3 more from this model

Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4 seriesDeepSeek

CSA/HCA cuts KV cache to 10% of V3.2 levels at 1M context

coremodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V4-Flash

DeepSeekadopted3 sources

we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

coremodel architecturein DeepSeek-V4 seriesDeepSeek
2 more from this model

Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek

combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4-FlashDeepSeek

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑