Open Models Almanac
Methodsinference & servingKV cache management

single mention · inference & serving

Decode context parallelism

Decode context parallelism keeps tensor parallelism for the weights and gives each GPU of a group every c-th token of every sequence's KV cache, merging the partial attention outputs with their log-sum-exp. It removes the copy that tensor parallelism makes of a single-head cache, but costs one more exchange per attention layer and cannot split an indexer's keys or a linear-attention state.

Written for the explainer Parallelism for serving large MoE models: what each layout copies, and what it sends.

Filed under inference & serving › KV cache management