Open Models Almanac
Methods

single mention · not yet filed

Two-stage contextual-parallel communication for compressed attention

A two-stage communication approach for context-parallel compressed attention, designed because compression needs consecutive KV entries that can straddle rank boundaries and produce varying compressed lengths.1 Each rank first sends its trailing uncompressed KV entries to its neighbor for compression, then an all-gather collects the locally compressed KV entries, which a select-and-pad operator reorganizes into the full compressed set.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the compression requires consecutive KV entries, which may straddle the boundary between two neighboring CP ranks — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. each rank sends its last uncompressed KV entries to rank . Then, rank compresses some of these received entries together with its local uncompressed KV entries — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called two-stage communication approach.