Mistral-Large-4
Mistral AITo mitigate this, we make use of context parallelism (CP) [80] to shard and de-replicate the KV cache across the sequence dimension instead.
general family · filed under optimization
Context parallelism is a method used to shard and de-replicate the KV cache across the sequence dimension.1 It is used to mitigate KV replication across TP ranks, which severely limits the system’s concurrency and KV-cache hit rate.2
Trade-off Optimized CP provides a 51% increase in decode interactivity while observing no more than 10–15% communication overhead compared to TP.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we make use of context parallelism (CP) [ 80] to shard and de-replicate the KV cache across the sequence dimension instead— technical report
naive tensor parallelism (TP) gives the best latency but suffers from KV replication across TP ranks, severely limiting the system’s concurrency and KV-cache hit rate— technical report
we get a 51% increase in decode interactivity (Figure 16) and observe no more than 10–15% communication overhead compared to TP— technical report
Also called context parallelism (CP).
2 quotes from 2 sources, one card per model, each opening with its strongest.
To mitigate this, we make use of context parallelism (CP) [80] to shard and de-replicate the KV cache across the sequence dimension instead.
32-way context parallelism