Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

specific method · filed under model architecture

Hybrid compressed attention (CSA and HCA)

Hybrid compressed attention is an attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).1 It aims to make long-context processing more efficient by mixing the two attention forms.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA — deepseek-ai/DeepSeek-V4-Flash · Hugging Face
  2. to dramatically improve long-context efficiency — deepseek-ai/DeepSeek-V4-Flash · Hugging Face
sources
3
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

DeepSeek-V4-Flash

DeepSeekadopted2 sources

introducing several key architectural innovations: (1) Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek
1 more from this model

hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)

usedmodel architecturein DeepSeek-V4-FlashDeepSeek

DeepSeek-V4-Pro

DeepSeekadopted2 sources

Hybrid Attention Architecture: We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4 seriesDeepSeek
1 more from this model

introducing several key architectural innovations: (1) Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency.

usedmodel architecturein DeepSeek-V4DeepSeek
↑