general family · filed under inference & serving
Prefix caching
Reuses cached prefix computation to avoid prefill work for matching request prefixes.
Also called prefix cache.
- sources
- 5
- models
- 4
- labs adopt it
- 3
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
optional 2used 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023) paper arxiv.orgintroduces RadixAttention prefix caching
- Automatic Prefix Caching - vLLM Documentation docs docs.vllm.ai
Evidence
5 spans quoted from the sources, strongest treatment first.
--enable-prefix-caching
usedunclearin Qwen3.6-35B-A3BQwen
enable-prefix-caching
usedinference servingin Nemotron 3 UltraNVIDIA
Prefix Cache (extending KV storage to system RAM)
usedinference servingin GLM-5Z.ai
enable MTP-1 speculative decoding and disable prefix caching.
optionalinference servingin vLLMvLLM
--enable-prefix-caching
optionalinference servingin Nemotron 3 UltraNVIDIA
Filed alongside
Other methods under inference & serving :: KV cache management.
Chunked prefillPrompt cachingSWA Bounded ReplayLanguage-model-only serving modePeriodic cache checkpointingCompressed KV cachingKDA-aware prefix-cache managementRadixCacheZero SWA caching8-bit Mamba cache quantizationAsynchronous cache offloading and restorationAutomatic cacheBlock-based KV cacheCross-layer KV-cache reuseCustomized heterogeneous KV-cache layoutDisable prefix caching for benchmarkingDynamic allocation of fixed-size state-cache poolsEncoder SWA bounded replayExact SWA KV reconstruction via full multi-layer replayFine-grained prefix hashingHierarchical KV cachingInference-side KV-cache reset on weight synchronizationKDA with prefill cacheKey-value reuse in global attention layers