single mention · inference & serving
Decode context parallelism
Decode context parallelism keeps tensor parallelism for the weights and gives each GPU of a group every c-th token of every sequence's KV cache, merging the partial attention outputs with their log-sum-exp. It removes the copy that tensor parallelism makes of a single-head cache, but costs one more exchange per attention layer and cannot split an indexer's keys or a linear-attention state.
Written for the explainer Parallelism for serving large MoE models: what each layout copies, and what it sends.
Filed under inference & serving › KV cache management
Explainer · 02 · 35 minServing a large reasoning model at low costA token gets cheaper as more requests share each forward pass, until the latency promise or the KV-cache budget stops the batch from growingExplainer · I3 · 36 minParallelism for serving large MoE models: what each layout copies, and what it sendsEvery layout copies something onto every GPU: tensor parallelism the single-head KV cache, data-parallel attention the non-expert weights. The batch and the context decide which copy costs more, and memory often decides before speed does