specific method · filed under model architecture
Compressed Sparse Attention
Compressed Sparse Attention is an attention mechanism that compresses the KV cache of blocks of consecutive tokens into single weighted entries.1 It then applies DeepSeek Sparse Attention so that each query token attends only to the compressed KV entries, with uncompressed KV kept for the tail tokens of the compression branch.2,3
Trade-off Its compression makes long-context attention substantially cheaper, at the price of unstored uncompressed KV entries that must be recomputed on cache hits.4,5
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
CSA integrates both compression and sparse attention strategies: it first compresses the Key-Value (KV) cache of every tokens into one entry
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligenceand then applies DeepSeek Sparse Attention (DSA) [dsv32] where each query token attends to only compressed KV entries
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencealong with the uncompressed tail tokens from the compression branch
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencewhich substantially reduces the computational cost of attention in long-text scenarios
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligenceas uncompressed KV entries in CSA and HCA are not stored
— DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Also called CSA, Compressed Sparse Attention (CSA).