specific method · filed under model architecture
Heavily Compressed Attention
Heavily Compressed Attention is an attention variant that compresses the KV cache more aggressively than CSA, using a larger compression rate and no overlapped compression.1,2 Each group of KV entries is merged into one compressed entry via compression weights and learnable positional biases, shortening the sequence before attention.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
compresses the KV cache in a heavier manner, but does not employ sparse attention
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligenceemploys a larger compression rate () and does not perform overlapped compression
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligenceeach KV entries in will be compressed into one according to the compression weights and learnable positional biases
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Also called Heavily Compressed Attention (HCA).