Open Models Almanac

Inference & serving · I4

Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory

Reading a cached prefix back from another tier usually costs far less than recomputing it; the price is in keeping it: memory held for hours, a route back to it, and for hybrid models a saved state at the right boundary

Illustration of a long ink-blue paper scroll unrolled across a writing desk, with a short fresh coral section at its end. Behind the desk stand three shelves of increasing depth, holding rolled scrolls; a small teal lantern hangs on one of them.

A coding agent works in turns. It sends its whole conversation to the model, gets back a few hundred tokens (a tool call, an edit), runs the tool, appends the result and sends everything again. SemiAnalysis’s AgentX traces of real coding-agent sessions, as vLLM describes them, have a median of 43 turns per session, a median input of 142K tokens and a median output of 444 tokens. Kimi K3’s report describes the same shape at its own scale: “a typical coding input carries a prefix of 400K tokens but requires a prefill increment of only 4K tokens”.

Take GLM-5.3 on one node of eight B200 GPUs, at a context of 142K tokens plus 4K new ones. If the node has to rebuild the KV cache of the whole context (a miss), the prefill takes about 1.05 seconds. If it still holds the cache of the first 142K tokens (a hit), it computes only the 4K new tokens, in about 32 ms. Both are computed from the model’s FLOPs per token at 14.4 PFLOP/s for the node, an assumed 40% of its dense FP8 peak. The hit is 33 times cheaper rather than 36.5 because the new tokens still attend over the cached prefix. Kimi’s API prices the difference, though not its size: $0.30 per million input tokens on a cache hit, $3.00 on a miss.

To serve agents well, every turn should be a hit. That takes a scheduler that mixes prefill and decode without stalling either, a cache that recognises a repeated prefix, memory to keep it between turns, and a router that sends the next turn to its cache. Hybrid models change each of the four.

What there is to keep

Recall from 00 and 02: prefill runs the prompt through the model and writes each token’s keys and values into the KV cache; decode then produces one token per step, reading the whole cache each time. Prefill is limited by compute, decode by memory bandwidth. Two latencies matter to a user: the time to first token (TTFT), which is mostly queueing plus prefill, and the inter-token latency (ITL), the time per decode step.

What a session leaves behind depends on the architecture, and T2 and T5 describe the kinds. A layer with full attention keeps keys and values for every token. A sliding-window layer keeps only the last WW tokens, a ring of fixed size. A recurrent layer (Mamba-2, Gated DeltaNet, Kimi Delta Attention) keeps one state per sequence, of fixed size, overwritten at every token. DeepSeek-V4 compresses groups of tokens into single entries. Five models from the map carry the article’s examples:

ModelWhat grows with the contextFixed per sequenceTotal at 142K, BF16
Hy3GQA keys and values in 80 layersnothing46.5 GB
GLM-5.3one compressed latent per token in 78 layers, plus indexer keys in 21nothing13.5 GB
Qwen3.5-397B-A17BGQA in 15 layersGated DeltaNet states in 45 layers, 192 MB4.6 GB
Kimi K3one latent per token in 24 MLA layersKDA states in 69 layers, 449 MB4.4 GB
DeepSeek-V4.1-Flashcompressed entries, written by 4 layers and reused by the others128-token windows in 40 layers, 5.2 MB0.46 GB

All five rows are computed from the configs on one basis: two bytes per cached value, except recurrent states, which are counted in the 32-bit format the code keeps them in. One session’s cache spans a factor of 101 across these models. In deployment the caches are smaller. FP8 fits 1.6 to 2.2 times as many tokens in a byte as BF16, and FP4 3.2 to 3.6 times (computed; I1 explains the formats). The ratio is below 2 where scale bytes or 32-bit states remain, and above it for DeepSeek-V4.1-Flash, whose FP8 format keeps the indexer keys in 4 bits.

The scheduler: pages, continuous batching, chunks

Early servers reserved a contiguous slab of memory per request, sized for the longest output it might produce. Kwon et al. (2023) measured that “only 20.4% - 38.2% of the KV cache memory is used to store the actual token states” in those systems. Their PagedAttention stores the cache in fixed-size blocks of tokens, allocated on demand and addressed through a per-request block table, like pages of virtual memory. Waste is then at most one partly filled block per request. vLLM’s default block today is 64 tokens (the paper’s was 16): larger blocks let the kernel read more positions at once, smaller ones waste less in the last block.

Paging makes continuous batching cheap, since a request that joins mid-batch needs only a few free blocks, not a slab sized for its longest possible output. Instead of running a batch of requests to completion, the engine reschedules at every step: finished requests leave, new ones join (Orca, Yu et al. 2022, called this iteration-level scheduling). A newcomer’s prefill then has to share steps with other users’ decodes, and a long prefill in one step stalls every decode in it. Sarathi-Serve (Agrawal et al., 2024) measured “up to 28.3×” higher ITL for such mixed batches than for decode-only ones.

Chunked prefill gives each step a token budget. The scheduler fills it with the decodes first (one token each), then with a slice of a pending prompt that fits what remains; a long prompt is spread over many steps. vLLM does this by default (“It batches all pending decode requests before scheduling any prefill operations”). The budget is --max-num-batched-tokens: for the API server 8,192 on H100- and H200-class GPUs, 16,384 on GPUs with 160 GiB or more. SGLang chunks prefills too, but by default runs each chunk as its own step (--enable-mixed-chunk is off), so the running decodes wait for it; the chunk size bounds that wait.

Choosing the budget means trading one latency for the other. A step costs roughly the larger of a fixed floor (reading weights and cache) and the compute for its tokens. Below the point where compute takes over, adding prompt tokens to a step is nearly free; above it, every extra token lengthens the step for every user decoding in it.

  • step time (inter-token latency)
  • time to first token
  • knee: compute = floor
050100150Step time, ms (time per output token)00.511.5Time to first token, s (32K prompt)5125121K1K2K2K4K4K8K8K16K16Ktoken budget per step (log scale)knee 1,87016.4 ms0.28 s (17 steps)050100150Step time, ms: what every decoding user waits per token00.511.5Time to first token of a 32K-token prompt, s5125121K1K2K2K4K4K8K8K16K16Ktoken budget per step (log scale)knee 1,87016.4 ms0.28 s (17 steps)

Budget 2,048 tokens per step = 64 decodes + a 1,984-token prompt chunk (above the knee: compute sets the step).

Step time, which every decoding user waits per token: 16.4 ms. The 32K prompt takes 17 steps: time to first token 0.28 s.

Knee at 1,870 tokens per step (floor 15 ms ÷ 8.02 µs per token).

A toy, not a measurement. Step time = the larger of a fixed floor (reading weights and cache; assumed, and held constant whatever the number of decodes) and the step's tokens × 8.0 µs, GLM-5.3's prefill FLOPs per token at 142K context on eight B200 GPUs at 40% of dense FP8 peak (computed). Every token in the step is charged that cost. No queueing, no other prompts waiting.

Figure 1.A toy step-time model for chunked prefill on one node. Below about 1,900 tokens per step the step is limited by its fixed floor, so a bigger chunk shortens the time to first token at no cost to the users decoding. Above it, a bigger budget barely helps the first token and lengthens every decode step. The floor is an assumption, every token is charged a long-context prefill token's FLOPs, and real engines add queueing.

The cost of chunking is overhead per chunk: each one re-reads the weights and the cache written so far. Sarathi measured “at most ∼25%” extra prefill time at a chunk of 512 tokens on Yi-34B and “almost negligible overhead” at 2,048.

When the cache pool fills, vLLM preempts a running request and later recomputes it rather than copying its cache to host memory and back (02), which shows up as tail latency. Before admitting a request, vLLM checks by default “whether the full input sequence length fits in the KV cache”, and it can reject requests once the queued prefill exceeds a TTFT target (max_num_queued_tokens). Labs schedule by request class. Kimi K3’s report describes “budget-based admission control”, so that “bursty long-context traffic consumes at most its own share of the capacity”.

Prefix caching: recognising a repeated prefix

Prefix caching keeps the KV blocks of finished requests and reuses them when a new request starts with the same tokens. To find them, vLLM hashes each full block from three things: the hash of the block before it, the block’s own tokens, and extra keys such as a LoRA adapter, an image hash or a tenant’s cache_salt. Because each hash includes its parent’s, a block’s hash commits to the whole prefix before it (a hash chain), so equal hashes mean equal prefixes, barring a collision of SHA-256, vLLM’s default hash. A lookup walks the request’s blocks from the start until a hash is missing. Only full blocks are cached, so a hit is a whole number of blocks. Freed blocks join an eviction queue in least-recently-used (LRU) order, each request’s last block placed to go first, since it “is less likely to be reused by other requests”.

SGLang’s RadixAttention (Zheng et al., 2024) keeps cached prefixes in a radix tree instead: a tree whose edges are labelled with runs of tokens, so each cached prefix is a path from the root and requests that share a prefix share the start of that path. Its pages are one token by default. A lookup follows the tree from the root and stops where the tokens differ, so a hit can end at any token; eviction removes least-recently-used leaves, so shared ancestors such as a system prompt survive longest. The paper also orders waiting requests by longest shared prefix, which it proves maximises hits when the whole batch is known in advance and the cache holds at least the longest request, at the risk of starving others (in SGLang today this is an option, --schedule-policy lpm; the default is first come, first served). Over a month of Chatbot Arena traffic, it reports hit rates of 52.4% and 74.1% for two models, where a hit rate is cached prompt tokens divided by prompt tokens. No source we found compares them on the same traffic.

A hit saves less than the token count suggests, because each new token still attends over the whole cached prefix. A full-attention layer charges a token at position tt about 4thdh4 t h d_h FLOPs: 2tdh2 t d_h per head for its scores against the cached keys and as much for the weighted sum of values. For Hy3 (80 such layers, hdh=8,192h d_h = 8{,}192) that is 372 GFLOP per token at 142K, against 42 GFLOP for its weight matmuls (twice its 21B active parameters). A hit at 142K + 4K therefore costs the weight matmuls of the 4K new tokens plus their attention to 142K positions:

Model (one node’s compute: 8 B200, FP8 at 40%)Miss: prefill 146KHit: prefill 4K over 142K cachedMiss / hit
GLM-5.31.05 s32 ms33×
DeepSeek-V4.1-Flash0.19 s5.4 ms35×
Qwen3.5-397B-A17B0.71 s29 ms24×
Kimi K33.2 s117 ms27×
Hy32.4 s117 ms20×

All computed, from twice the active parameters per token plus the attention term of each layer type; in a miss each token attends only to those before it. Kimi K3’s weights need two such nodes, so its times are its FLOPs at one node’s rate. The token ratio is 36.5×; the attention to the prefix is 17% of the hit’s cost for DeepSeek-V4.1-Flash and 90% for Hy3, whose 80 GQA layers read every position. vLLM measured the same effect on DeepSeek-V4.1-Flash: on two GB300 GPUs “an 8K-token prefill takes 87.9 ms without prior context, but 189.4 ms with 512K cached context”. Over a session the saving grows, since a miss re-prefills a context that grows every turn. With nn equal turns, the miss/hit ratio is (n+1)/2(n+1)/2 for the weight matmuls and (n+1)(2n+1)/6n(n+1)(2n+1)/6n for full attention, 18.5 and 12.5 at 36 turns, and each model’s session falls between the two (computed).

  • miss: prefill the whole context
  • hit: prefill only the new tokens
  • cache kept between turns
00.511.5Prefill time per turn on one node, s01020Cache the session holds after the turn, GB (FP8)116612121818242430303636turnturnturn 36: miss 1.03 s (outline)hit 32.1 ms (filled)after turn 36: 7.88 GBTurn 1: 0 cached + 4,000 new tokens. Miss 24.6 ms, hit 24.6 ms (1.00×). Cache after the turn 219 MB FP8.Turn 2: 4,000 cached + 4,000 new tokens. Miss 50.2 ms, hit 25.6 ms (1.96×). Cache after the turn 438 MB FP8.Turn 3: 8,000 cached + 4,000 new tokens. Miss 76.0 ms, hit 25.8 ms (2.95×). Cache after the turn 657 MB FP8.Turn 4: 12,000 cached + 4,000 new tokens. Miss 102 ms, hit 26.0 ms (3.93×). Cache after the turn 876 MB FP8.Turn 5: 16,000 cached + 4,000 new tokens. Miss 128 ms, hit 26.2 ms (4.90×). Cache after the turn 1.09 GB FP8.Turn 6: 20,000 cached + 4,000 new tokens. Miss 154 ms, hit 26.3 ms (5.86×). Cache after the turn 1.31 GB FP8.Turn 7: 24,000 cached + 4,000 new tokens. Miss 181 ms, hit 26.5 ms (6.82×). Cache after the turn 1.53 GB FP8.Turn 8: 28,000 cached + 4,000 new tokens. Miss 208 ms, hit 26.7 ms (7.77×). Cache after the turn 1.75 GB FP8.Turn 9: 32,000 cached + 4,000 new tokens. Miss 235 ms, hit 26.9 ms (8.72×). Cache after the turn 1.97 GB FP8.Turn 10: 36,000 cached + 4,000 new tokens. Miss 262 ms, hit 27.1 ms (9.66×). Cache after the turn 2.19 GB FP8.Turn 11: 40,000 cached + 4,000 new tokens. Miss 289 ms, hit 27.3 ms (10.6×). Cache after the turn 2.41 GB FP8.Turn 12: 44,000 cached + 4,000 new tokens. Miss 317 ms, hit 27.5 ms (11.5×). Cache after the turn 2.63 GB FP8.Turn 13: 48,000 cached + 4,000 new tokens. Miss 344 ms, hit 27.7 ms (12.4×). Cache after the turn 2.85 GB FP8.Turn 14: 52,000 cached + 4,000 new tokens. Miss 372 ms, hit 27.9 ms (13.3×). Cache after the turn 3.06 GB FP8.Turn 15: 56,000 cached + 4,000 new tokens. Miss 400 ms, hit 28.1 ms (14.3×). Cache after the turn 3.28 GB FP8.Turn 16: 60,000 cached + 4,000 new tokens. Miss 428 ms, hit 28.3 ms (15.2×). Cache after the turn 3.50 GB FP8.Turn 17: 64,000 cached + 4,000 new tokens. Miss 457 ms, hit 28.5 ms (16.1×). Cache after the turn 3.72 GB FP8.Turn 18: 68,000 cached + 4,000 new tokens. Miss 486 ms, hit 28.6 ms (17.0×). Cache after the turn 3.94 GB FP8.Turn 19: 72,000 cached + 4,000 new tokens. Miss 514 ms, hit 28.8 ms (17.8×). Cache after the turn 4.16 GB FP8.Turn 20: 76,000 cached + 4,000 new tokens. Miss 543 ms, hit 29.0 ms (18.7×). Cache after the turn 4.38 GB FP8.Turn 21: 80,000 cached + 4,000 new tokens. Miss 573 ms, hit 29.2 ms (19.6×). Cache after the turn 4.60 GB FP8.Turn 22: 84,000 cached + 4,000 new tokens. Miss 602 ms, hit 29.4 ms (20.5×). Cache after the turn 4.82 GB FP8.Turn 23: 88,000 cached + 4,000 new tokens. Miss 632 ms, hit 29.6 ms (21.3×). Cache after the turn 5.03 GB FP8.Turn 24: 92,000 cached + 4,000 new tokens. Miss 661 ms, hit 29.8 ms (22.2×). Cache after the turn 5.25 GB FP8.Turn 25: 96,000 cached + 4,000 new tokens. Miss 691 ms, hit 30.0 ms (23.1×). Cache after the turn 5.47 GB FP8.Turn 26: 100,000 cached + 4,000 new tokens. Miss 722 ms, hit 30.2 ms (23.9×). Cache after the turn 5.69 GB FP8.Turn 27: 104,000 cached + 4,000 new tokens. Miss 752 ms, hit 30.4 ms (24.8×). Cache after the turn 5.91 GB FP8.Turn 28: 108,000 cached + 4,000 new tokens. Miss 782 ms, hit 30.6 ms (25.6×). Cache after the turn 6.13 GB FP8.Turn 29: 112,000 cached + 4,000 new tokens. Miss 813 ms, hit 30.7 ms (26.5×). Cache after the turn 6.35 GB FP8.Turn 30: 116,000 cached + 4,000 new tokens. Miss 844 ms, hit 30.9 ms (27.3×). Cache after the turn 6.57 GB FP8.Turn 31: 120,000 cached + 4,000 new tokens. Miss 875 ms, hit 31.1 ms (28.1×). Cache after the turn 6.79 GB FP8.Turn 32: 124,000 cached + 4,000 new tokens. Miss 907 ms, hit 31.3 ms (28.9×). Cache after the turn 7.01 GB FP8.Turn 33: 128,000 cached + 4,000 new tokens. Miss 938 ms, hit 31.5 ms (29.8×). Cache after the turn 7.22 GB FP8.Turn 34: 132,000 cached + 4,000 new tokens. Miss 970 ms, hit 31.7 ms (30.6×). Cache after the turn 7.44 GB FP8.Turn 35: 136,000 cached + 4,000 new tokens. Miss 1.00 s, hit 31.9 ms (31.4×). Cache after the turn 7.66 GB FP8.Turn 36: 140,000 cached + 4,000 new tokens. Miss 1.03 s, hit 32.1 ms (32.2×). Cache after the turn 7.88 GB FP8.00.511.5Prefill time per turn on one node, s01020Cache the session holds after the turn, GB (FP8)116612121818242430303636turnturnturn 36: miss 1.03 s (outline)hit 32.1 ms (filled)after turn 36: 7.88 GBTurn 1: 0 cached + 4,000 new tokens. Miss 24.6 ms, hit 24.6 ms (1.00×). Cache after the turn 219 MB FP8.Turn 2: 4,000 cached + 4,000 new tokens. Miss 50.2 ms, hit 25.6 ms (1.96×). Cache after the turn 438 MB FP8.Turn 3: 8,000 cached + 4,000 new tokens. Miss 76.0 ms, hit 25.8 ms (2.95×). Cache after the turn 657 MB FP8.Turn 4: 12,000 cached + 4,000 new tokens. Miss 102 ms, hit 26.0 ms (3.93×). Cache after the turn 876 MB FP8.Turn 5: 16,000 cached + 4,000 new tokens. Miss 128 ms, hit 26.2 ms (4.90×). Cache after the turn 1.09 GB FP8.Turn 6: 20,000 cached + 4,000 new tokens. Miss 154 ms, hit 26.3 ms (5.86×). Cache after the turn 1.31 GB FP8.Turn 7: 24,000 cached + 4,000 new tokens. Miss 181 ms, hit 26.5 ms (6.82×). Cache after the turn 1.53 GB FP8.Turn 8: 28,000 cached + 4,000 new tokens. Miss 208 ms, hit 26.7 ms (7.77×). Cache after the turn 1.75 GB FP8.Turn 9: 32,000 cached + 4,000 new tokens. Miss 235 ms, hit 26.9 ms (8.72×). Cache after the turn 1.97 GB FP8.Turn 10: 36,000 cached + 4,000 new tokens. Miss 262 ms, hit 27.1 ms (9.66×). Cache after the turn 2.19 GB FP8.Turn 11: 40,000 cached + 4,000 new tokens. Miss 289 ms, hit 27.3 ms (10.6×). Cache after the turn 2.41 GB FP8.Turn 12: 44,000 cached + 4,000 new tokens. Miss 317 ms, hit 27.5 ms (11.5×). Cache after the turn 2.63 GB FP8.Turn 13: 48,000 cached + 4,000 new tokens. Miss 344 ms, hit 27.7 ms (12.4×). Cache after the turn 2.85 GB FP8.Turn 14: 52,000 cached + 4,000 new tokens. Miss 372 ms, hit 27.9 ms (13.3×). Cache after the turn 3.06 GB FP8.Turn 15: 56,000 cached + 4,000 new tokens. Miss 400 ms, hit 28.1 ms (14.3×). Cache after the turn 3.28 GB FP8.Turn 16: 60,000 cached + 4,000 new tokens. Miss 428 ms, hit 28.3 ms (15.2×). Cache after the turn 3.50 GB FP8.Turn 17: 64,000 cached + 4,000 new tokens. Miss 457 ms, hit 28.5 ms (16.1×). Cache after the turn 3.72 GB FP8.Turn 18: 68,000 cached + 4,000 new tokens. Miss 486 ms, hit 28.6 ms (17.0×). Cache after the turn 3.94 GB FP8.Turn 19: 72,000 cached + 4,000 new tokens. Miss 514 ms, hit 28.8 ms (17.8×). Cache after the turn 4.16 GB FP8.Turn 20: 76,000 cached + 4,000 new tokens. Miss 543 ms, hit 29.0 ms (18.7×). Cache after the turn 4.38 GB FP8.Turn 21: 80,000 cached + 4,000 new tokens. Miss 573 ms, hit 29.2 ms (19.6×). Cache after the turn 4.60 GB FP8.Turn 22: 84,000 cached + 4,000 new tokens. Miss 602 ms, hit 29.4 ms (20.5×). Cache after the turn 4.82 GB FP8.Turn 23: 88,000 cached + 4,000 new tokens. Miss 632 ms, hit 29.6 ms (21.3×). Cache after the turn 5.03 GB FP8.Turn 24: 92,000 cached + 4,000 new tokens. Miss 661 ms, hit 29.8 ms (22.2×). Cache after the turn 5.25 GB FP8.Turn 25: 96,000 cached + 4,000 new tokens. Miss 691 ms, hit 30.0 ms (23.1×). Cache after the turn 5.47 GB FP8.Turn 26: 100,000 cached + 4,000 new tokens. Miss 722 ms, hit 30.2 ms (23.9×). Cache after the turn 5.69 GB FP8.Turn 27: 104,000 cached + 4,000 new tokens. Miss 752 ms, hit 30.4 ms (24.8×). Cache after the turn 5.91 GB FP8.Turn 28: 108,000 cached + 4,000 new tokens. Miss 782 ms, hit 30.6 ms (25.6×). Cache after the turn 6.13 GB FP8.Turn 29: 112,000 cached + 4,000 new tokens. Miss 813 ms, hit 30.7 ms (26.5×). Cache after the turn 6.35 GB FP8.Turn 30: 116,000 cached + 4,000 new tokens. Miss 844 ms, hit 30.9 ms (27.3×). Cache after the turn 6.57 GB FP8.Turn 31: 120,000 cached + 4,000 new tokens. Miss 875 ms, hit 31.1 ms (28.1×). Cache after the turn 6.79 GB FP8.Turn 32: 124,000 cached + 4,000 new tokens. Miss 907 ms, hit 31.3 ms (28.9×). Cache after the turn 7.01 GB FP8.Turn 33: 128,000 cached + 4,000 new tokens. Miss 938 ms, hit 31.5 ms (29.8×). Cache after the turn 7.22 GB FP8.Turn 34: 132,000 cached + 4,000 new tokens. Miss 970 ms, hit 31.7 ms (30.6×). Cache after the turn 7.44 GB FP8.Turn 35: 136,000 cached + 4,000 new tokens. Miss 1.00 s, hit 31.9 ms (31.4×). Cache after the turn 7.66 GB FP8.Turn 36: 140,000 cached + 4,000 new tokens. Miss 1.03 s, hit 32.1 ms (32.2×). Cache after the turn 7.88 GB FP8.

GLM-5.3, 36 turns of 4,000 new tokens (to 144K): prefill over the whole session 18.4 s if every turn misses, 1.03 s if every turn hits (17.8× less).

Turn 36: miss 1.03 s, hit 32.1 ms. Cache the session leaves behind: 7.88 GB (FP8).

Computed, not measured: FLOPs per token from each model's config (twice the active parameters plus attention to every earlier token, sparse where the model is), on eight B200 GPUs at 40% of dense FP8 peak. The cache format changes only the lower panel; the prefill compute is FP8 in both. Hover or tap a turn for its numbers.

Figure 2.One agent session of 36 turns, 4,000 new tokens each, prefilled on one node of eight B200 GPUs (computed). Without a cache each turn rebuilds the whole context; with a hit it computes only the new tokens and their attention to the cached prefix.

A hit also needs the tokens to be identical. One changed token early in the prompt makes everything after it a miss. SGLang’s GLM-5.3 cookbook documents a case. Claude Code prepends a per-request header to the system prompt, and GLM-5.3’s chat template puts only the tool list before the system prompt, so “that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn”. The fix is on the client: CLAUDE_CODE_ATTRIBUTION_HEADER=0 drops the header. Caches are also tied to the weights: DeepSeek’s DeepInfer deployment notes list changes to “model weights, the block byte layout, or the tokenizer” as invalidating every stored entry.

Tiers: where a prefix waits between turns

Between two turns an agent runs its tool, which can take seconds or minutes, while the GPU’s memory is needed for other sessions. A server that can only cache in GPU memory (HBM) loses the prefix as soon as the batch grows. Engines can therefore move idle cache blocks down to host memory (DRAM), local SSD, or a pool shared by many servers, though vLLM and SGLang both leave this off by default (--kv-offloading-size, --enable-hierarchical-cache).

Moving a prefix back costs bytes over a link; recomputing it costs FLOPs. Loading wins when

bytes of the prefixlink bandwidth<FLOPs to recompute itachieved FLOP/s,\frac{\text{bytes of the prefix}}{\text{link bandwidth}} < \frac{\text{FLOPs to recompute it}}{\text{achieved FLOP/s}},

so the break-even bandwidth is the one that moves the prefix in the time recomputing it takes. For Hy3 in FP8, 23.3 GB in 2.25 s over eight GPUs, each loading its own eighth, needs 1.3 GB/s per GPU; DeepSeek-V4.1-Flash, with 0.21 GB, needs 0.14 GB/s (computed, at 40% of the B200’s dense FP8 peak). Every link a server has is faster than that, as long as each GPU loads only its own share.

Tensor parallelism (02) splits a cache by KV head, and a cache with fewer KV heads than GPUs is copied instead: the latent of GLM-5.3 and Kimi K3 and DeepSeek’s single key-value head sit on all eight GPUs, and Qwen3.5’s two KV heads on four each. If each GPU then loads its own copy over its own link, the break-even rises four to eight times; for GLM-5.3 it reaches 7.6 GB/s per GPU, and one older SSD per GPU then loses (1.11 s against 1.02 s, computed). Splitting the cache by sequence position (decode context parallelism) or by request (data-parallel attention, where each GPU runs attention for its own requests, I3) can store and load each byte once.

  • recompute the prefix
  • load the cached prefix over a link
  • load slower than recompute
Time to get the 142K prefix back, log scale0.1 ms1 ms10 ms0.1 s1 s10 sV4.1-Flashrecompute 182 mshost DRAMDeepSeek-V4.1-Flash from host DRAM over PCIe Gen5 x16 (64 GB/s): 209 MB in 0.41 ms; recompute 182 ms; 446×446× fasterRDMA NICDeepSeek-V4.1-Flash from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 209 MB in 0.52 ms; recompute 182 ms; 348×348× fasterNVMe Gen5DeepSeek-V4.1-Flash from local NVMe Gen5 drive (14 GB/s, assumed): 209 MB in 1.9 ms; recompute 182 ms; 97.5×98× fasterNVMe Gen4DeepSeek-V4.1-Flash from local NVMe Gen4 drive (7 GB/s, assumed): 209 MB in 3.7 ms; recompute 182 ms; 48.8×49× fasterDeepSeek-V4.1-Flash: recompute 142K tokens on the node, 182 msKimi K3recompute 3.09 shost DRAMKimi K3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 2.69 GB in 5.2 ms; recompute 3.09 s; 590×590× fasterRDMA NICKimi K3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 2.69 GB in 6.7 ms; recompute 3.09 s; 461×461× fasterNVMe Gen5Kimi K3 from local NVMe Gen5 drive (14 GB/s, assumed): 2.69 GB in 24.0 ms; recompute 3.09 s; 129×129× fasterNVMe Gen4Kimi K3 from local NVMe Gen4 drive (7 GB/s, assumed): 2.69 GB in 47.9 ms; recompute 3.09 s; 64.5×65× fasterKimi K3: recompute 142K tokens on the node, 3.09 sQwen3.5recompute 682 mshost DRAMQwen3.5-397B-A17B from host DRAM over PCIe Gen5 x16 (64 GB/s): 2.37 GB in 4.6 ms; recompute 682 ms; 147×147× fasterRDMA NICQwen3.5-397B-A17B from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 2.37 GB in 5.9 ms; recompute 682 ms; 115×115× fasterNVMe Gen5Qwen3.5-397B-A17B from local NVMe Gen5 drive (14 GB/s, assumed): 2.37 GB in 21.2 ms; recompute 682 ms; 32.2×32× fasterNVMe Gen4Qwen3.5-397B-A17B from local NVMe Gen4 drive (7 GB/s, assumed): 2.37 GB in 42.4 ms; recompute 682 ms; 16.1×16× fasterQwen3.5-397B-A17B: recompute 142K tokens on the node, 682 msGLM-5.3recompute 1.02 shost DRAMGLM-5.3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 7.77 GB in 15.2 ms; recompute 1.02 s; 67.1×67× fasterRDMA NICGLM-5.3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 7.77 GB in 19.4 ms; recompute 1.02 s; 52.4×52× fasterNVMe Gen5GLM-5.3 from local NVMe Gen5 drive (14 GB/s, assumed): 7.77 GB in 69.4 ms; recompute 1.02 s; 14.7×15× fasterNVMe Gen4GLM-5.3 from local NVMe Gen4 drive (7 GB/s, assumed): 7.77 GB in 139 ms; recompute 1.02 s; 7.33×7.3× fasterGLM-5.3: recompute 142K tokens on the node, 1.02 sHy3recompute 2.25 shost DRAMHy3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 23.3 GB in 45.4 ms; recompute 2.25 s; 49.5×50× fasterRDMA NICHy3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 23.3 GB in 58.2 ms; recompute 2.25 s; 38.7×39× fasterNVMe Gen5Hy3 from local NVMe Gen5 drive (14 GB/s, assumed): 23.3 GB in 208 ms; recompute 2.25 s; 10.8×11× fasterNVMe Gen4Hy3 from local NVMe Gen4 drive (7 GB/s, assumed): 23.3 GB in 415 ms; recompute 2.25 s; 5.41×5.4× fasterHy3: recompute 142K tokens on the node, 2.25 s142K-token sessions per TiB of host memoryone stored copy02,0004,0006,000V4.1-FlashDeepSeek-V4.1-Flash: 209 MB stored (FP8); 5,273 sessions per TiB5,273Kimi K3Kimi K3: 2.69 GB stored (FP8); 409 sessions per TiB409Qwen3.5Qwen3.5-397B-A17B: 2.37 GB stored (FP8); 463 sessions per TiB463GLM-5.3GLM-5.3: 7.77 GB stored (FP8); 141 sessions per TiB141Hy3Hy3: 23.3 GB stored (FP8); 47 sessions per TiB47Time to get the 142K prefix back, log scale0.1 ms1 ms10 ms0.1 s1 s10 sV4.1-Flashrecompute 182 mshost DRAMDeepSeek-V4.1-Flash from host DRAM over PCIe Gen5 x16 (64 GB/s): 209 MB in 0.41 ms; recompute 182 ms; 446×446× fasterRDMA NICDeepSeek-V4.1-Flash from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 209 MB in 0.52 ms; recompute 182 ms; 348×348× fasterNVMe Gen5DeepSeek-V4.1-Flash from local NVMe Gen5 drive (14 GB/s, assumed): 209 MB in 1.9 ms; recompute 182 ms; 97.5×98× fasterNVMe Gen4DeepSeek-V4.1-Flash from local NVMe Gen4 drive (7 GB/s, assumed): 209 MB in 3.7 ms; recompute 182 ms; 48.8×49× fasterDeepSeek-V4.1-Flash: recompute 142K tokens on the node, 182 msKimi K3recompute 3.09 shost DRAMKimi K3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 2.69 GB in 5.2 ms; recompute 3.09 s; 590×590× fasterRDMA NICKimi K3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 2.69 GB in 6.7 ms; recompute 3.09 s; 461×461× fasterNVMe Gen5Kimi K3 from local NVMe Gen5 drive (14 GB/s, assumed): 2.69 GB in 24.0 ms; recompute 3.09 s; 129×129× fasterNVMe Gen4Kimi K3 from local NVMe Gen4 drive (7 GB/s, assumed): 2.69 GB in 47.9 ms; recompute 3.09 s; 64.5×65× fasterKimi K3: recompute 142K tokens on the node, 3.09 sQwen3.5recompute 682 mshost DRAMQwen3.5-397B-A17B from host DRAM over PCIe Gen5 x16 (64 GB/s): 2.37 GB in 4.6 ms; recompute 682 ms; 147×147× fasterRDMA NICQwen3.5-397B-A17B from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 2.37 GB in 5.9 ms; recompute 682 ms; 115×115× fasterNVMe Gen5Qwen3.5-397B-A17B from local NVMe Gen5 drive (14 GB/s, assumed): 2.37 GB in 21.2 ms; recompute 682 ms; 32.2×32× fasterNVMe Gen4Qwen3.5-397B-A17B from local NVMe Gen4 drive (7 GB/s, assumed): 2.37 GB in 42.4 ms; recompute 682 ms; 16.1×16× fasterQwen3.5-397B-A17B: recompute 142K tokens on the node, 682 msGLM-5.3recompute 1.02 shost DRAMGLM-5.3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 7.77 GB in 15.2 ms; recompute 1.02 s; 67.1×67× fasterRDMA NICGLM-5.3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 7.77 GB in 19.4 ms; recompute 1.02 s; 52.4×52× fasterNVMe Gen5GLM-5.3 from local NVMe Gen5 drive (14 GB/s, assumed): 7.77 GB in 69.4 ms; recompute 1.02 s; 14.7×15× fasterNVMe Gen4GLM-5.3 from local NVMe Gen4 drive (7 GB/s, assumed): 7.77 GB in 139 ms; recompute 1.02 s; 7.33×7.3× fasterGLM-5.3: recompute 142K tokens on the node, 1.02 sHy3recompute 2.25 shost DRAMHy3 from host DRAM over PCIe Gen5 x16 (64 GB/s): 23.3 GB in 45.4 ms; recompute 2.25 s; 49.5×50× fasterRDMA NICHy3 from a neighbour over a 400 Gb/s RDMA NIC (50 GB/s): 23.3 GB in 58.2 ms; recompute 2.25 s; 38.7×39× fasterNVMe Gen5Hy3 from local NVMe Gen5 drive (14 GB/s, assumed): 23.3 GB in 208 ms; recompute 2.25 s; 10.8×11× fasterNVMe Gen4Hy3 from local NVMe Gen4 drive (7 GB/s, assumed): 23.3 GB in 415 ms; recompute 2.25 s; 5.41×5.4× fasterHy3: recompute 142K tokens on the node, 2.25 sSessions per TiBof host memoryone stored copy02,0004,0006,000V4.1-FlashDeepSeek-V4.1-Flash: 209 MB stored (FP8); 5,273 sessions per TiB5,273Kimi K3Kimi K3: 2.69 GB stored (FP8); 409 sessions per TiB409Qwen3.5Qwen3.5-397B-A17B: 2.37 GB stored (FP8); 463 sessions per TiB463GLM-5.3GLM-5.3: 7.77 GB stored (FP8); 141 sessions per TiB141Hy3Hy3: 23.3 GB stored (FP8); 47 sessions per TiB47

Smallest margin: Hy3 from NVMe Gen4, 415 ms against 2.25 s to recompute (5.4×). Largest: Kimi K3 from host DRAM, 590×.

Sessions of 142K tokens in 1 TiB of host memory (FP8, one stored copy): V4.1-Flash 5,273, Kimi K3 409, Qwen3.5 463, GLM-5.3 141, Hy3 47.

Computed at nominal line rates, one link (or one drive) per GPU, on eight B200 GPUs at 40% of dense FP8 peak; recompute is at that rate in both formats. The NVMe rates are assumed, and achieved rates are lower (about 50 GB/s one way over PCIe on an H100, as vLLM measured). Under tensor parallelism a latent cache (GLM-5.3, DeepSeek-V4.1-Flash, Kimi K3) sits whole on every GPU, and Qwen3.5's two KV heads on four GPUs each; the second loading option shows what that costs when each GPU fetches its own copy over its own link. Sessions per TiB count one stored copy, or one per GPU that needs it.

Figure 3.Getting a 142K-token prefix back on one node of eight B200 GPUs: recomputing it, or loading it over each link (computed, nominal bandwidths, FP8 cache, recompute at an assumed 40% of peak). With the load split over the eight GPUs' links, every tier is faster than recomputing. The second toggle shows plain tensor parallelism, where a latent cache sits on every GPU and each loads its own copy: GLM-5.3 from one older SSD per GPU then loses. The right panel shows how many sessions each tier holds: 1 TiB of host memory holds 47 of Hy3's sessions or 5,273 of DeepSeek-V4.1-Flash's.

Achieved bandwidths are lower (vLLM measured about 50 GB/s one way over PCIe on an H100), and small copies are worse: vLLM’s offloading throughput rose “by an order of magnitude” when a new layout grew its blocks to 0.5–2.5 MB. With the load split across the GPUs, loading still wins at those rates. A head-sharded cache comes close to break-even only for short prefixes, whose tokens cost little to recompute: for Hy3 it reaches 6.2 GB/s per GPU at 4K tokens (computed). Once tiers are on, the engines start the load while the request waits in the queue, and SGLang stops waiting for its shared tier after a timeout (2 s plus 0.1 s per 1,024 tokens by default) and recomputes what has not arrived.

Tiers therefore differ mostly in capacity: how many sessions each holds, and for how long. The engines differ in when they write blocks to a lower tier and which servers can read them.

  • SGLang’s HiCache calls GPU memory L1, host memory L2 and a distributed store L3 (a distributed KV store or file system, such as Mooncake or DeepSeek’s 3FS). L1 and L2 are private to one server; “A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3”. The default write policy is write-through, “Every access is immediately written back to the next level”; write-back writes “only when it is evicted from the upper level”.
  • vLLM’s offloading connector copies a prompt’s blocks to host memory “as they are produced” (write-through; decode blocks are skipped by default), so a host tier smaller than the GPU cache “just mirrors what the GPU already holds and adds no hit rate”. Files, object stores and peer hosts sit behind the host tier.
  • Labs pick by workload. Kimi K3’s RL rollouts write idle prefixes to “an external KV cache pool in CPU DRAM only when it is evicted from GPU” (write-back), so as not to copy blocks still in use, and prefetch them back before the next reuse. MiMo-V2.6’s rollouts move a trajectory’s state to host memory during tool time, the known idle gap. DeepSeek’s production system keeps only its sliding windows in host memory and stages loads of the main cache there. Compressed blocks go to a shared SSD cluster on 3FS, “roughly 100 PiB of SSD storage for the KV cache, with a TTL (time-to-live) of 3 days for agent traffic and 7 days for chatbot traffic”.

In cache-hierarchy terms, writing through keeps the lower tier inclusive, so the two hold only as many distinct blocks as the larger of them, which is why vLLM’s small host tier adds nothing; writing on eviction makes it close to exclusive, holding up to both tiers’ worth.

In every measurement, a lower tier paid off only when the GPU cache was under pressure. In vLLM’s tiered-offloading test (Qwen3.6-35B-A3B on two H100s, 12K-token prompts plus 4K per round), GPU-only, host and storage caching all did well up to about 64 conversations. From 64 to 128 “throughput drops dramatically without offloading”, and beyond 128 only the storage tier kept a high hit rate. The fall is a cliff, as LRU suggests when sessions take turns: a session’s prefix survives only if every other session’s fit beside it, so the hit rate holds until their combined cache outgrows the tier. The LMCache team’s test on Claude Code traces (MiniMax-M2.5, two MI300X) found 3.0× lower average TTFT with a host tier than with vLLM’s GPU-only prefix cache when GPU memory was deliberately squeezed (in a 20-minute run that completed 28 and 12 requests), and worse results at a light load, where everything fit in GPU memory. DeepSeek gives the one lab-reported production hit rate with both its period and its denominator, from V3 and R1 in February 2025: of all input tokens over 24 hours, “608B, of which 342B tokens (56.3%) hit the on-disk KV cache”. The DeepInfer notes add that a TTL of about 3 days “costs chatbot traffic 0.5%–1% of KV cache hit rate”.

Prefill/decode disaggregation

Chunking limits how much a prefill slows the decodes it shares a step with, but doesn’t remove it, and a node that does both has to pick one layout for two phases with opposite limits. Prefill/decode disaggregation (P/D) runs them on separate pools of GPUs. A request is prefilled in the first pool, its KV cache is sent to the second, and decoding continues there. DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2024) proposed it. SGLang’s documentation gives two reasons: prefill batches “frequently interrupt ongoing decode batches”, and with data-parallel attention, where each GPU is called a rank, all ranks meet at every expert layer’s exchange, so the decoding ranks wait for the one that is prefilling.

The first cost is the transfer: the decode pool needs the whole context’s cache unless it still holds the prefix from the previous turn.

  • whole context (146K tokens)
  • the turn's 4K new tokens
  • fixed state, sent every turn
Transfer per request, ms, one 400 Gb/s NIC0200400600V4.1-FlashDeepSeek-V4.1-Flash: whole context 214 MB, 4.3 mswhole context 214 MB: 4.3 msDeepSeek-V4.1-Flash: increment 8.42 MB (2.62 MB of it sliding-window rings), 0.17 msincrement 8.42 MB: 0.17 msKimi K3Kimi K3: whole context 2.75 GB, 55.0 mswhole context 2.75 GB: 55.0 msKimi K3: increment 512 MB (449 MB of it KDA states), 10.2 msincrement 512 MB: 10.2 ms (88% state)Qwen3.5Qwen3.5-397B-A17B: whole context 2.43 GB, 48.7 mswhole context 2.43 GB: 48.7 msQwen3.5-397B-A17B: increment 254 MB (192 MB of it Gated DeltaNet states), 5.1 msincrement 254 MB: 5.1 ms (76% state)GLM-5.3GLM-5.3: whole context 7.99 GB, 160 mswhole context 7.99 GB: 160 msGLM-5.3: increment 219 MB, 4.4 msincrement 219 MB: 4.4 msHy3Hy3: whole context 23.9 GB, 478 mswhole context 23.9 GB: 478 msHy3: increment 655 MB, 13.1 msincrement 655 MB: 13.1 msTransfer time per request, ms, over one 400 Gb/s NIC (50 GB/s)0200400600V4.1-FlashDeepSeek-V4.1-Flash: whole context 214 MB, 4.3 mswhole context 214 MB: 4.3 msDeepSeek-V4.1-Flash: increment 8.42 MB (2.62 MB of it sliding-window rings), 0.17 msincrement 8.42 MB: 0.17 msKimi K3Kimi K3: whole context 2.75 GB, 55.0 mswhole context 2.75 GB: 55.0 msKimi K3: increment 512 MB (449 MB of it KDA states), 10.2 msincrement 512 MB: 10.2 ms (88% state)Qwen3.5Qwen3.5-397B-A17B: whole context 2.43 GB, 48.7 mswhole context 2.43 GB: 48.7 msQwen3.5-397B-A17B: increment 254 MB (192 MB of it Gated DeltaNet states), 5.1 msincrement 254 MB: 5.1 ms (76% state)GLM-5.3GLM-5.3: whole context 7.99 GB, 160 mswhole context 7.99 GB: 160 msGLM-5.3: increment 219 MB, 4.4 msincrement 219 MB: 4.4 msHy3Hy3: whole context 23.9 GB, 478 mswhole context 23.9 GB: 478 msHy3: increment 655 MB, 13.1 msincrement 655 MB: 13.1 ms

Whole context over one 400 Gb/s NIC (50 GB/s) (FP8): V4.1-Flash 4.3 ms, Kimi K3 55.0 ms, Qwen3.5 48.7 ms, GLM-5.3 160 ms, Hy3 478 ms.

Only the turn's 4,000 tokens: V4.1-Flash 0.17 ms, Kimi K3 10.2 ms, Qwen3.5 5.1 ms, GLM-5.3 4.4 ms, Hy3 13.1 ms. Of Kimi K3's increment, 88% is its KDA state; of Qwen3.5's, 76% is its Gated DeltaNet state.

Computed: cache bytes from each model's config, timed over one link at its nominal rate. The axis is rescaled for each view.

Figure 4.Bytes a decode pool receives per request at 142K + 4K tokens, timed over one 400 Gb/s link (computed, nominal bandwidth, FP8 cache). A recurrent model sends its fixed state with every turn even when the decode pool keeps the prefix: for Kimi K3 and Qwen3.5 that is most of the increment. The increment assumes the prefill side already holds the previous answer's cache.

Sending each layer as soon as it is computed hides part of the transfer, but only if the decode GPUs are chosen before prefill starts, as Splitwise does (“At the end of each layer, we trigger an asynchronous transfer”); vLLM’s guide counts the whole transfer against TTFT. Standard P/D also moves the cache one way, and in a multi-turn session that wastes work: “On turn two, decode still holds KV for the answer it just generated but prefill has never computed it, so prefill recomputes it” (vLLM’s guide). The fixes send the cache back (vLLM’s bidirectional transfer), keep it on the decode side, or have decode write what it generated into the shared store, as DeepSeek and vLLM’s Mooncake store do. For a hybrid model that write matters most, since the state at the end of the output is the one the next turn resumes from. SGLang’s decode-side prefix reuse “asks prefill to send only the missing suffix”; on GLM-5.2-FP8 with eight B300s per worker and synthetic conversations growing to 160K tokens, it cut median TTFT by 60.3% and average TTFT by 25.4%, with GPU and host tiers together.

P/D also leaves two pools to size. DistServe’s example (a 13B model, 512 input and 64 output tokens) put two prefill GPUs per decode GPU in its ideal allocation. DeepSeek reports a P:D ratio of “roughly 1:1” in GPUs and a smallest efficient unit of “64 GPUs of prefill plus 64 GPUs of decode”, because its decode runs wide expert parallelism. vLLM’s AgentX post explains why there is no rule: context lengths and hit rates “vary wildly across sessions”, and the best ratio “also shifts with concurrency”. A cache hit moves work out of prefill: if a share hh of requests hit, a request’s prefill costs (1−h)(1-h) of a miss plus hh of a hit, so for GLM-5.3 at 142K a hit rate of 56% cuts the prefill pool needed 2.2-fold and 90% about eightfold (computed, ignoring queueing).

What it buys is easy to overstate. vLLM’s documentation says it “DOES NOT improve throughput”; it separates the tuning of TTFT and ITL, and its later guide calls the gain goodput, the request rate served within both latency targets. The measured throughput gains for NVIDIA’s 7 + 4 split for Nemotron 3.5 Super VL and vLLM’s 4 + 4 recipe for Mistral Large 4, hold under a minimum per-user speed on their own setups (02), the Mistral one on unique prompts with prefix caching off. NVIDIA’s split also raised median TTFT from 0.49 s to 1.12 s.

Routing: affinity or a shared store

A cache in one server’s GPU or host memory helps only the requests that reach that server. With many servers, which server a turn reaches decides whether it hits.

The simplest rule is session affinity: hash a session id to a server, so every turn lands where its prefix is. vLLM’s Mistral Large 4 recipe routes each session to the same data-parallel rank and reports that without it “the hit rate drops 14–22 points and TTFT p99 is 2–3x worse”. vLLM’s AgentX post found that balancing by queue depth, running tokens or cache use “all … underperformed simple session-aware sticky routing”, because the next turn often arrives while the prefix is still on the previous GPU. Kimi K3 pins each session by consistent hashing to a primary cluster and a secondary, spread across the fleet, that “holds none of the session’s prefix cache and must re-prefill it upon failover”.

The price of affinity is balance: a hot session can’t move, and a server that fails takes its sessions’ caches with it. Cache-aware routers trade the two explicitly. SGLang’s gateway keeps an approximate radix tree of what it has sent to each worker. It routes to the best match if more than 30% of the prompt’s characters match, and falls back to plain load balancing when the busiest and idlest workers’ loads differ by more than 64 requests and by more than 1.5 times. That tree records what the gateway sent, not what each worker still holds, so it misses evictions; NVIDIA Dynamo and llm-d instead follow events that the engines publish when blocks are stored or freed. Dynamo prices each worker by the blocks it would have to prefill, minus credits for blocks it holds in GPU, host or disk, plus its decode load, so “A cache-hot worker can lose to a colder worker when its active load makes the total cost higher”.

The other design takes most of the conflict away. DeepSeek stores the main cache on a storage cluster that every prefill server reads, which “lets every prefill machine hit any prefix, avoiding a conflict between prefix affinity and the load-balancing routing policy”, and “there is no prefix-affinity routing”: engines pull work by remaining compute quota. The price is a transfer on every hit and a store that must keep up; DeepSeek’s sizing asks for “at least 40 GB/s read and 5 GB/s write” per storage server. A vLLM test with a shared Mooncake store and deliberately round-robin routing kept the hit rate “above 95%”. A shared store turns a lost hit into a load, which still costs: in vLLM’s AgentX tests sticky routing won with a distributed pool in place too, since a moved session’s prefix must be fetched and occupies the new GPU’s cache while it arrives. DeepSeek can skip affinity partly because its V4.1 cache is small, 0.21 GB per 142K-token session in FP8.

Hybrid models: state that resumes only where it was saved

Everything so far assumed a cache with one entry per token, valid for any prefix: the cache of the first tt tokens is the first tt entries. Hybrid models break that in three ways, and the hit rule becomes the intersection of three different rules.

A sliding-window layer needs only the last WW entries before the hit point. vLLM’s hybrid manager accepts a hit if the last “sliding_window_size - 1 tokens” are still cached and frees blocks that fall out of the window (they stay cached until evicted); SGLang’s radix cache marks them as dead (“tombstones”) but keeps the tree. Per copy the ring is small: 5.2 MB per sequence for DeepSeek-V4.1-Flash in BF16, about 1% of its 0.46 GB of cache at 142K.

A recurrent layer has no per-token entries at all. Its state after token tt summarises the whole prefix and is overwritten at token t+1t+1, so a hit at tt needs a copy saved at exactly tt. SGLang’s K3 post: “A recurrent state cannot run backwards and must be replayed forward from an earlier checkpoint.” Both engines copy a saved state into a private slot before extending it (copy-on-write), so the shared copy survives. Saved states are large. One Kimi K3 state for all 69 KDA layers is 449 MB in 32 bits, as much as 16,253 tokens of its MLA cache in BF16 stored once; one Qwen3.5 state is 192 MB, 6,252 tokens’ worth (computed). In GPU memory under eight-way tensor parallelism, where every GPU holds the whole latent but only an eighth of the state, the K3 figure is about 2,000 (SGLang gives 54 MB against 27 KB per token per GPU). NVIDIA counts per layer: on its Mamba-2 model one layer’s state (a Mamba cache page) “occupies at least as many bytes as approximately 4K tokens of Attention KV cache” in vLLM’s layout.

Where a cache hit may end (schematic, toy positions)full attention: any block boundarysliding window: the 512 tokens before heldrecurrent state: only where savedcompressed groups: multiples of 128hit allowed: all four agree3,968×9,000: not a multiple of 12802K4K6K8K10Ktoken positionWhere a cache hit may end (schematic, toy positions)full attention: any block boundarysliding window: the 512 tokens before heldrecurrent state: only where savedcompressed groups: multiples of 128hit allowed: all four agree3,968×9,000: not a multiple of 12802K4K6K8K10Ktoken position

Above, a toy: a hit must end where every layer type can resume. Below, computed for one session of 142K tokens.

  • saved snapshots
  • the session's KV cache (BF16)
  • recompute after a hit
Stored per session, GB (one stored copy)0200400600a snapshot every 512 tokens (277)124 GB (28.4×)one per turn (35)15.7 GB (3.6×)the session's own cache at 142K4.38 GBRecompute on a hit, ms on one 8×B200 node050100150expected: 256 tokens since the snapshot7.4 msfor scale: prefill of a 4K-token turn117 msStored per session, GB (one stored copy)0200400600a snapshot every 512 tokens (277)124 GB (28.4×)one per turn (35)15.7 GB (3.6×)the session's own cache at 142K4.38 GBRecompute on a hit, ms on one 8×B200 node050100150expected: 256 tokens since the snapshot7.4 msfor scale: prefill of a 4K-token turn117 ms

Kimi K3 (state in 32 bits): one snapshot (KDA states + conv tails, FP32 state) is 449 MB, as much as 16,253 tokens of its BF16 cache.

Every 512 tokens: 277 snapshots = 124 GB, 28.4× the session's 4.38 GB cache. A hit at a random token recomputes 256 tokens on average: 7.4 ms, against 117 ms for a 4K-token turn.

Storage × expected recompute: 926 GB·ms, about the same at every interval: the slider only trades one for the other. Keeping the newest state only: 449 MB; one per turn (35): 15.7 GB, 3.6× the cache.

Computed, not measured: a hit at a random token resumes from the snapshot at or before it, and the tokens since are prefilled at the model's FLOPs per token at 142K, on eight B200 GPUs at 40% of dense FP8 peak. Sizes count one stored copy. Per GPU under tensor parallelism the comparison shrinks, since a state is split by head while a latent cache sits whole on every GPU: one Kimi K3 state is 16,250 tokens of cache in one copy, about 2,030 on each GPU. Axes are linear from zero and stay fixed while the slider moves.

Figure 5.Where a hit may land in a hybrid model, and what saving a recurrent state every B tokens costs against recomputing up to B tokens on a hit at a random position (computed, one session of 142K tokens, one stored copy). One Kimi K3 state weighs as much as 16,000 tokens of its attention cache, so saving it every 512 tokens takes 28 times the session's cache. An agent's next turn starts where the last one ended, so the state saved there is all it needs: one copy per session, or 35 (3.6 times the cache) if every turn's state is kept for branches.

So engines and labs save recurrent states sparsely. vLLM’s hybrid manager stores Mamba states only “on the Mamba block grid” and makes attention blocks as large as a state in bytes, so that one allocator with one page size can hold both. An attention page of that many bytes holds hundreds or thousands of tokens: 400 to 4,224 in the Nemotron deployments of vLLM’s hybrid post. Kimi K3’s report describes where that leads: the shared block size was “forced to 1024–6144 tokens”, and “At such a coarse granularity caching is nearly useless: requests shorter than one block can never be reused”. Its fix separates matching from storage. Hashes run on smaller blocks inside the large physical block (512 tokens in the report’s example), and states are saved “only at (a sparse subset of) MLA’s hash endpoints”, typically at conversation-turn boundaries, where an agent’s next request resumes. Moonshot’s engine also packs K3’s states into the same paged pool as the MLA cache, so one allocator and one eviction policy serve both. SGLang saves a state at each prefill chunk boundary and every 256 decode tokens but keeps only a few. Unless the kernel saves mid-step, as Moonshot’s does at the last hash-aligned position, the scheduler’s chunks are also the save grid.

Saving every BB tokens trades storage of ST/BS T / B bytes, for a state of SS bytes, against an expected recompute of B/2B/2 tokens on a hit at a random point, a product that stays the same as BB moves: the trade-off that sets how often a long job checkpoints against failure. It matters only when hits can land anywhere. A missed boundary costs a recompute from the last saved state: nothing for an agent whose turns end at saved states, more for requests that branch mid-turn, such as retries and several samples from one prompt.

Speculative decoding (P1) makes recurrent state costlier twice over. A verify step must keep the state after every draft token it might accept, since it cannot roll back, and NVIDIA’s Nemotron 3.5 and SGLang both size extra state buffers for this. A drafter resuming after a hit also needs the target model’s hidden state at the hit point, which vLLM’s prefix cache lacks, so vLLM replays from the last saved state; NVIDIA added an option to keep one extra checkpoint just before that boundary.

DeepSeek’s compressed cache is the third case. Its CSA and HCA layers write one entry per 4 and per 128 tokens, so a prefix that ends inside a group has entries that don’t exist yet. The V4 report stores “all of the compressed KV entries”, reuses them “until the last complete compression block”, and recomputes the tokens of the tail; a cache block must hold a multiple of lcm(4, 128) = 128 tokens, so that its boundary is a group boundary for both layer types, and V4.1’s production block is 128 tokens. The V4 report’s numbers are for V4-Flash, an older model with 43 layers and 0.98 GB of compressed cache at 142K. For the sliding windows the report weighed storing every token’s window entries, saving the window every few tokens, or storing none and recomputing. An exact rebuild reaches back one window per layer, since each layer’s window entries depend on the layer below over the previous WW tokens: at most W⋅LW \cdot L = 128 × 43 = 5,504 tokens. In production V4 kept the window at two points per turn and it still took “nearly half of the persistent KV cache capacity” (V4.1 report). One saved window set weighs as much as about 800 tokens of V4-Flash’s compressed cache (computed), so two per turn weigh as much as the entries of 1,600 new tokens, and the report singles out “multi-turn conversations with short turns”. Storing none “proved prohibitive in production deployments”. V4.1 moved the windows out of the SSD store into a pool of host memory with a TTL of minutes. When the compressed prefix hits but its windows have expired, it replays only the last 128 tokens. That replay is approximate (“not mathematically identical”), and the report finds it “barely compromises response quality”. Together with the smaller compressed cache, the persistent store fell to 1/8 of V4’s under the same workload.

The mechanisms side by side

MechanismWhat it keepsWhat it buysWhat it costsMain knob
Paged blockscache in fixed blocksmemory waste of at most one block per requestan indirection per blockblock size
Chunked prefillnothing newdecodes never wait for a whole promptper-chunk overhead; budget trades TTFT for ITLtoken budget per step
Prefix cache in HBMfinished requests’ blocksa turn costs its new tokens plus their attentionHBM taken from live sessions; evicted under loadeviction policy
Lower tiersidle blocks in DRAM, SSD or a poolcapacity for hours; load beats recompute when each GPU loads its sharememory and storage held per session; transfers; write trafficwrite policy, TTL
P/Dnothing newno prefill in decode steps; one layout per phasea transfer per request; two pools to sizeP:D ratio
Affinity routingnothing newhits from server-local cachesload imbalance; failover re-prefillbalance thresholds
Shared storeblocks every server can readhits under any routinga transfer on every hit, still cheapest where the prefix already is; store bandwidthstore size, TTL
Hybrid snapshotsrecurrent states and windows at chosen pointshits for models without per-token statea state is worth thousands of tokens; hits only at saved pointssnapshot points

Where the models in the map sit

The configs give what each model must keep. For a few labs, the reports also say how they keep it. Of the 35 language models with a code-check row (16 labs), 19 models from 9 labs have sliding-window attention somewhere, and 8 models from 4 labs (Qwen, Moonshot, Z.ai, NVIDIA) interleave recurrent layers with attention (the code checks, cleaned as in T5). Of these 13 labs, 3 describe how they cache that state in serving: DeepSeek, Moonshot and NVIDIA (Xiaomi does for RL rollouts).

ModelGrows with context (layers)Fixed per sequenceAt 142K, BF16What the lab says about reuse
Hy3GQA (80)none46.5 GBno lab document in the corpus
GLM-5.3latent + indexer keys (78)none13.5 GBRL: P/D, rollout-to-rank hashing (GLM-5 report)
Mistral Large 4latent (63 own, 9 shared)none11.9 GBP/D “to eliminate prefill interrupts”; FP8 cache and offloading
Gemma 4 31Bfull attention (10)50 windows of 1,024, 839 MB12.5 GBnothing
MiMo-V2.6-Profull attention (10)60 windows of 128, 39 MB7.3 GBRL: host offload during tool time
Qwen3.5-397B-A17BGQA (15)Gated DeltaNet (45), 192 MB4.6 GBnothing
Kimi K3MLA (24)KDA (69), 449 MB4.4 GBKDA state in the MLA page pool; 512-token hash blocks; cluster affinity; admission budgets
Nemotron 3.5 Super VLattention (8)Mamba-2 (40), 170 MB1.3 GBcoarse Mamba pages; extra checkpoint for speculative decoding; 7P + 4D
DeepSeek-V4.1-Flashcompressed, shared (4)40 windows of 128, 5.2 MB0.46 GB128-token blocks; SSD store; windows in host memory; bounded replay (V4: compressed entries whole, tail recomputed)

Sizes are computed from the configs (two bytes per value, recurrent states in 32 bits, MTP layers excluded; 02‘s 7.8 GB for GLM-5.3 in FP8 includes its MTP layer). “Nothing” means the lab’s own documents, searched, say nothing about cache reuse.

Easy to get wrong

  • A hit still costs attention: the new tokens attend over the cached prefix, 17% to 90% of a hit’s cost at 142K for these models, and more at longer contexts.
  • “Offloading is slower than recomputing” is false for these models at agent lengths whenever each GPU loads its own share of the cache. A lower tier fails when it is too small or not shared, or when every GPU must load its own copy of a latent cache from a slow drive.
  • Routing alone moves the hit rate: vLLM’s Mistral recipe loses 14–22 points when sessions stop landing on their cache.
  • P/D’s measured gains are goodput under a per-user speed floor, each on its own setup, and the split adds a transfer to the first token.
  • Many lab statements on caches describe RL rollouts: Kimi K3’s DRAM write-back pool, GLM-5’s P/D and rank affinity, MiMo’s tool-time offload. Public serving may differ.

Check yourself

A model’s cache is 0.5 MB per token in FP8, and recomputing a 100K-token prefix on its node takes 4 s. Its servers hold idle prefixes on local SSDs that read 5 GB/s per GPU, eight GPUs per node, each GPU holding an eighth of the cache. Is loading worth it? What break-even bandwidth per GPU does this case imply?

The prefix is 100,000 × 0.5 MB = 50 GB. Over eight drives at 5 GB/s it loads in 50 / 40 = 1.25 s, against 4 s of recompute: loading wins by 3.2×. Break-even is where loading also takes 4 s: 50 GB / 4 s = 12.5 GB/s per node, about 1.6 GB/s per GPU. If the cache were copied on every GPU instead, each would read all 50 GB at 5 GB/s, 10 s, and recomputing would win.

Why can’t a prefix cache for Kimi K3 hit at an arbitrary token, the way it can for GLM-5.3?

GLM-5.3 keeps one cache entry per token in every layer, so the first tt entries are the cache of any tt-token prefix. Kimi K3’s KDA layers keep one state per sequence that is overwritten at every token, so a hit needs a state saved at the hit point. A state is 449 MB, as much as 16,000 tokens of its MLA cache, so states are saved only at a few points, typically turn boundaries, and a hit can land only there (or recompute from the last one).

DeepSeek routes prefills with no session affinity at all. Why doesn’t its hit rate collapse?

Its main cache sits on a storage cluster every prefill server reads, so any server can load any prefix. Affinity only matters when a cache is private to one server. The price is a load from the store on every hit and a store sized to serve all of them; DeepSeek-V4.1-Flash’s cache is small enough (0.21 GB per 142K-token session in FP8) that the load is cheap.

In the map

Methods

Taxonomy nodes

Further reading

  • Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (2023): paged blocks, sharing and the swap-or-recompute measurements.
  • Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (2024): chunked prefill, stall-free schedules and how to pick the token budget.
  • Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (2024): RadixAttention and cache-aware scheduling, with the proof that longest-prefix-first order maximises hits.
  • Zhong et al., “DistServe” (2024) and Patel et al., “Splitwise” (2024): the case for disaggregation, goodput, and how to place the two pools.
  • Qin et al., “Mooncake” (2024): Kimi’s KV-cache-centric serving, with a scheduler that weighs prefix length against load.
  • DeepSeek-V4 report §3.5 and DeepSeek-V4.1 report §3.2, plus DeepSeek’s DeepInfer deployment Q&A: the most detailed account of a production cache for a hybrid model.
  • Kimi K3 technical report §5.3–5.4: hybrid prefix caching, admission budgets and cluster affinity.
  • NVIDIA Nemotron 3.5 Super and Lightning report §5.3–5.6: Mamba checkpoints with speculative decoding, and a measured P/D split.
↑