Open Models Almanac
Methodsinference & servingKV cache management

methods · inference & serving

KV cache management

53 methods filed at this node or below it, from the sources of 24 models.

In this branch 53

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 53

Prefix caching core · 10 sources · 10 quotes
Compressed KV caching core · 3 sources · 3 quotes
Cross-layer KV-cache sharing core · 2 sources · 2 quotes
Hybrid-state caching core · 2 sources · 2 quotes
SWA Bounded Replay core · 2 sources · 2 quotes
Chunked prefill default · 6 sources · 6 quotes
Prompt caching used · 4 sources · 4 quotes
Language-model-only serving mode used · 3 sources · 3 quotes
Internal prefill checkpointing used · 2 sources · 2 quotes
RadixCache used · 2 sources · 2 quotes
Periodic cache checkpointing optional · 3 sources · 3 quotes
HiCache mentioned · 2 sources · 2 quotes
40 single mentions
Block-based KV cache core · 1 source · 1 quote
Customized heterogeneous KV-cache layout core · 1 source · 1 quote
KDA-aware prefix-cache management core · 1 source · 2 quotes
Persistent per-dialogue-context KV caching core · 1 source · 1 quote
Prefix reuse core · 1 source · 1 quote
Shared-free-list cache allocation core · 1 source · 1 quote
Write-back external KV-cache policy core · 1 source · 1 quote
Automatic cache support default · 1 source · 1 quote
Asynchronous KV-cache offloading used · 1 source · 1 quote
Disabling prefix caching for benchmarking used · 1 source · 1 quote
KDA with prefill cache used · 1 source · 1 quote
KV cache sharing used · 1 source · 1 quote
KV-cache reset on weight synchronization used · 1 source · 1 quote
Overlapped host-memory prefetching used · 1 source · 1 quote
Pre-scheduled tile chunking used · 1 source · 1 quote
Preallocated static KV cache used · 1 source · 1 quote
Request-level prefix cache used · 1 source · 1 quote
Stateless in-memory prefix caching used · 1 source · 1 quote
You Only Cache Once (YOCO) used · 1 source · 1 quote
Decode context parallelism optional · 1 source · 1 quote
KV-cache management optional · 1 source · 1 quote
Zero SWA caching optional · 1 source · 1 quote
8-bit Mamba cache quantization evaluated · 1 source · 1 quote
KV-cache offloading evaluated · 1 source · 1 quote
Mamba prefix caching in align mode evaluated · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash-1017 usedCompressed KV caching used—
DeepSeek-V4.1-Flash coreCompressed KV caching coreSWA Bounded Replay core—
Step-5-Preview optionalPrefix caching optionalHiCache mentioned—
GLM-5.3 usedDisabling prefix caching for benchmarking used—
MiMo-V2.6-Flash corePersistent per-dialogue-context KV caching coreAsynchronous KV-cache offloading usedHierarchical KV caching across HBM and pinned host memory used—
NVIDIA-Nemotron-3-Ultra-550B-A55B defaultChunked prefill defaultPrefix caching used8-bit Mamba cache quantization evaluatedPeriodic cache checkpointing evaluated—
DeepSeek-V4-Flash coreCustomized heterogeneous KV-cache layout coreDynamic allocation of fixed-size state-cache pools usedKV-cache and sparse-attention-kernel co-design usedOn-disk KV-cache storage for shared-prefix reuse usedPeriodic cache checkpointing optionalZero SWA caching optional—
DeepSeek-V4-Pro coreCustomized heterogeneous KV-cache layout coreDynamic allocation of fixed-size state-cache pools usedKV-cache and sparse-attention-kernel co-design usedOn-disk KV-cache storage for shared-prefix reuse usedPeriodic cache checkpointing optionalZero SWA caching optional—
GLM-5.2 usedPrefix caching usedRadixCache used—
Gemma 4 31B usedKey-as-value reuse in global attention layers usedKV cache sharing usedPrompt caching usedStateless in-memory prefix caching used—
Inkling usedPrompt caching used—
Kimi K3 coreFine-grained prefix hashing within cache pages coreKDA-aware prefix-cache management coreProjected-input caching for speculative KDA rollback coreShared-free-list cache allocation coreSparse hash-aligned KDA recurrent-state checkpoints coreUnified paged cache layout for KDA states and MLA KV coreWrite-back external KV-cache policy coreKDA with prefill cache usedKV-cache-aware placement of one-shot option messages used—
Laguna-S-2.1 usedKV-cache reset on weight synchronization used—
MiMo-V2.5 usedChunked prefill usedRequest-level prefix cache usedRadixCache not used—
MiMo-V2.6-Pro corePersistent per-dialogue-context KV caching coreAsynchronous KV-cache offloading usedHierarchical KV caching across HBM and pinned host memory used—
MiniMax-M3 coreBlock-based KV cache coreAutomatic cache support defaultPre-scheduled tile chunking used—
Mistral-Large-4 coreCross-layer KV-cache sharing corePrefix caching defaultYou Only Cache Once (YOCO) usedDecode context parallelism optional—
MolmoAct 2 usedCached depth-token replay for unchanged cells usedPreallocated static KV cache used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedCheckpointing replay for Mamba recurrent-state cache usedHybrid-state caching usedInternal prefill checkpointing usedMamba checkpointing for speculative-decoding replay optional—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreAttention-cache and Mamba-state preservation coreHybrid-state caching corePrefix caching corePrefix reuse coreCheckpointing replay for Mamba recurrent-state cache usedInternal prefill checkpointing usedMamba checkpointing for speculative-decoding replay usedKV-cache management optionalKV-cache offloading evaluated—
Qwen3.5-397B-A17B usedChunked prefill usedLanguage-model-only serving mode usedPrompt caching usedPrefix caching optional—
Qwen3.6-35B-A3B usedChunked prefill usedPrefix caching usedPrompt caching usedLanguage-model-only serving mode optionalMamba prefix caching in align mode evaluated—
Qwen3.8-Flash-Next usedOverlapped host-memory prefetching used—
π0.5 coreRobot-state KV caching across sampling steps corePrefix key-value caching during action inference used—
↑