Open Models Almanac
Methodsoptimizationtraining parallelism

general family · filed under optimization

Context parallelism

Context parallelism is a method used to shard and de-replicate the KV cache across the sequence dimension.1 It is used to mitigate KV replication across TP ranks, which severely limits the system’s concurrency and KV-cache hit rate.2

Trade-off Optimized CP provides a 51% increase in decode interactivity while observing no more than 10–15% communication overhead compared to TP.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we make use of context parallelism (CP) [ 80] to shard and de-replicate the KV cache across the sequence dimension instead — technical report
  2. naive tensor parallelism (TP) gives the best latency but suffers from KV replication across TP ranks, severely limiting the system’s concurrency and KV-cache hit rate — technical report
  3. we get a 51% increase in decode interactivity (Figure 16) and observe no more than 10–15% communication overhead compared to TP — technical report

Also called context parallelism (CP).

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Mistral-Large-4

Mistral AIadopted1 source

To mitigate this, we make use of context parallelism (CP) [80] to shard and de-replicate the KV cache across the sequence dimension instead.

usedunclearin Mistral Large 4Mistral AI
↑