Open Models Almanac
Methodsinference & servingKV cache management

specific method · filed under inference & serving

Prefix caching

Prefix caching keeps the KV cache of finished requests and reuses it when a new request starts with the same tokens, so the server computes only the new part of the prompt. The new tokens still attend over the cached prefix, so a hit costs their weight matmuls plus that attention, and a single changed token early in the prompt turns the rest into a miss.

Written for the explainer Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory.

Also called prefix cache, enable-prefix-caching.

sources
10
models
7
labs adopt it
5
strongest
core

Documented in

Evidence

10 quotes from 10 sources, one card per model, each opening with its strongest.

How sources treat it
optional 5used 3default 1core 1

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑