A coding agent works in turns. It sends its whole conversation to the model, gets back a few hundred tokens (a tool call, an edit), runs the tool, appends the result and sends everything again. SemiAnalysis’s AgentX traces of real coding-agent sessions, as vLLM describes them, have a median of 43 turns per session, a median input of 142K tokens and a median output of 444 tokens. Kimi K3’s report describes the same shape at its own scale: “a typical coding input carries a prefix of 400K tokens but requires a prefill increment of only 4K tokens”.
Take GLM-5.3 on one node of eight B200 GPUs, at a context of 142K tokens plus 4K new ones. If the node has to rebuild the KV cache of the whole context (a miss), the prefill takes about 1.05 seconds. If it still holds the cache of the first 142K tokens (a hit), it computes only the 4K new tokens, in about 32 ms. Both are computed from the model’s FLOPs per token at 14.4 PFLOP/s for the node, an assumed 40% of its dense FP8 peak. The hit is 33 times cheaper rather than 36.5 because the new tokens still attend over the cached prefix. Kimi’s API prices the difference, though not its size: $0.30 per million input tokens on a cache hit, $3.00 on a miss.
To serve agents well, every turn should be a hit. That takes a scheduler that mixes prefill and decode without stalling either, a cache that recognises a repeated prefix, memory to keep it between turns, and a router that sends the next turn to its cache. Hybrid models change each of the four.
What there is to keep
Recall from 00 and 02: prefill runs the prompt through the model and writes each token’s keys and values into the KV cache; decode then produces one token per step, reading the whole cache each time. Prefill is limited by compute, decode by memory bandwidth. Two latencies matter to a user: the time to first token (TTFT), which is mostly queueing plus prefill, and the inter-token latency (ITL), the time per decode step.
What a session leaves behind depends on the architecture, and T2 and T5 describe the kinds. A layer with full attention keeps keys and values for every token. A sliding-window layer keeps only the last tokens, a ring of fixed size. A recurrent layer (Mamba-2, Gated DeltaNet, Kimi Delta Attention) keeps one state per sequence, of fixed size, overwritten at every token. DeepSeek-V4 compresses groups of tokens into single entries. Five models from the map carry the article’s examples:
| Model | What grows with the context | Fixed per sequence | Total at 142K, BF16 |
|---|---|---|---|
| Hy3 | GQA keys and values in 80 layers | nothing | 46.5 GB |
| GLM-5.3 | one compressed latent per token in 78 layers, plus indexer keys in 21 | nothing | 13.5 GB |
| Qwen3.5-397B-A17B | GQA in 15 layers | Gated DeltaNet states in 45 layers, 192 MB | 4.6 GB |
| Kimi K3 | one latent per token in 24 MLA layers | KDA states in 69 layers, 449 MB | 4.4 GB |
| DeepSeek-V4.1-Flash | compressed entries, written by 4 layers and reused by the others | 128-token windows in 40 layers, 5.2 MB | 0.46 GB |
All five rows are computed from the configs on one basis: two bytes per cached value, except recurrent states, which are counted in the 32-bit format the code keeps them in. One session’s cache spans a factor of 101 across these models. In deployment the caches are smaller. FP8 fits 1.6 to 2.2 times as many tokens in a byte as BF16, and FP4 3.2 to 3.6 times (computed; I1 explains the formats). The ratio is below 2 where scale bytes or 32-bit states remain, and above it for DeepSeek-V4.1-Flash, whose FP8 format keeps the indexer keys in 4 bits.
The scheduler: pages, continuous batching, chunks
Early servers reserved a contiguous slab of memory per request, sized for the longest output it might produce. Kwon et al. (2023) measured that “only 20.4% - 38.2% of the KV cache memory is used to store the actual token states” in those systems. Their PagedAttention stores the cache in fixed-size blocks of tokens, allocated on demand and addressed through a per-request block table, like pages of virtual memory. Waste is then at most one partly filled block per request. vLLM’s default block today is 64 tokens (the paper’s was 16): larger blocks let the kernel read more positions at once, smaller ones waste less in the last block.
Paging makes continuous batching cheap, since a request that joins mid-batch needs only a few free blocks, not a slab sized for its longest possible output. Instead of running a batch of requests to completion, the engine reschedules at every step: finished requests leave, new ones join (Orca, Yu et al. 2022, called this iteration-level scheduling). A newcomer’s prefill then has to share steps with other users’ decodes, and a long prefill in one step stalls every decode in it. Sarathi-Serve (Agrawal et al., 2024) measured “up to 28.3×” higher ITL for such mixed batches than for decode-only ones.
Chunked prefill gives each step a token budget. The scheduler fills it with the decodes first (one token each), then with a slice of a pending prompt that fits what remains; a long
prompt is spread over many steps. vLLM does this by default (“It batches all pending decode requests before scheduling any prefill
operations”). The budget is --max-num-batched-tokens: for the API server 8,192 on H100- and H200-class GPUs, 16,384
on GPUs with 160 GiB or more. SGLang chunks prefills too, but by default runs each chunk as its own step
(--enable-mixed-chunk is off), so the running decodes wait for it; the chunk size bounds that wait.
Choosing the budget means trading one latency for the other. A step costs roughly the larger of a fixed floor (reading weights and cache) and the compute for its tokens. Below the point where compute takes over, adding prompt tokens to a step is nearly free; above it, every extra token lengthens the step for every user decoding in it.
- step time (inter-token latency)
- time to first token
- knee: compute = floor
Budget 2,048 tokens per step = 64 decodes + a 1,984-token prompt chunk (above the knee: compute sets the step).
Step time, which every decoding user waits per token: 16.4 ms. The 32K prompt takes 17 steps: time to first token 0.28 s.
Knee at 1,870 tokens per step (floor 15 ms ÷ 8.02 µs per token).
A toy, not a measurement. Step time = the larger of a fixed floor (reading weights and cache; assumed, and held constant whatever the number of decodes) and the step's tokens × 8.0 µs, GLM-5.3's prefill FLOPs per token at 142K context on eight B200 GPUs at 40% of dense FP8 peak (computed). Every token in the step is charged that cost. No queueing, no other prompts waiting.
The cost of chunking is overhead per chunk: each one re-reads the weights and the cache written so far. Sarathi measured “at most ∼25%” extra prefill time at a chunk of 512 tokens on Yi-34B and “almost negligible overhead” at 2,048.
When the cache pool fills, vLLM preempts a running request and later recomputes it rather than copying its cache to
host memory and back (02), which shows up as tail latency. Before admitting a request, vLLM checks by
default “whether the full input sequence length fits in the KV cache”, and it can reject requests once the queued
prefill exceeds a TTFT target (max_num_queued_tokens). Labs schedule by request class. Kimi K3’s report describes
“budget-based admission control”, so that “bursty long-context traffic consumes at most its own share of the
capacity”.
Prefix caching: recognising a repeated prefix
Prefix caching keeps the KV blocks of finished requests and reuses them
when a new request starts with the same tokens. To find them, vLLM hashes each full block from three things: the hash of the block before it, the block’s own tokens, and extra keys such as a LoRA adapter, an image
hash or a tenant’s cache_salt. Because each hash includes its parent’s,
a block’s hash commits to the whole prefix before it (a hash chain), so equal hashes mean equal prefixes, barring a
collision of SHA-256, vLLM’s default hash. A lookup walks the request’s blocks from the start until a hash is missing. Only full blocks are cached, so a
hit is a whole number of blocks. Freed blocks join an eviction queue in least-recently-used (LRU) order, each request’s last
block placed to go first, since it “is less likely to be reused by other requests”.
SGLang’s RadixAttention (Zheng et al., 2024) keeps cached prefixes in a radix tree
instead: a tree whose edges are labelled with runs of tokens, so each cached prefix is a path from the root and requests
that share a prefix share the start of that path. Its pages are one token by default. A lookup follows the tree from the root and stops where the tokens differ,
so a hit can end at any token; eviction removes least-recently-used leaves, so shared ancestors such as a system prompt
survive longest. The paper also orders waiting requests by longest shared prefix, which it proves maximises hits when
the whole batch is known in advance and the cache holds at least the longest request, at the risk of starving others
(in SGLang today this is an option, --schedule-policy lpm; the default is first come, first served). Over a month of Chatbot Arena traffic, it reports hit rates of 52.4% and 74.1%
for two models, where a hit rate is cached prompt tokens divided by prompt tokens. No source we found compares them on the same traffic.
A hit saves less than the token count suggests, because each new token still attends over the whole cached prefix. A full-attention layer charges a token at position about FLOPs: per head for its scores against the cached keys and as much for the weighted sum of values. For Hy3 (80 such layers, ) that is 372 GFLOP per token at 142K, against 42 GFLOP for its weight matmuls (twice its 21B active parameters). A hit at 142K + 4K therefore costs the weight matmuls of the 4K new tokens plus their attention to 142K positions:
| Model (one node’s compute: 8 B200, FP8 at 40%) | Miss: prefill 146K | Hit: prefill 4K over 142K cached | Miss / hit |
|---|---|---|---|
| GLM-5.3 | 1.05 s | 32 ms | 33× |
| DeepSeek-V4.1-Flash | 0.19 s | 5.4 ms | 35× |
| Qwen3.5-397B-A17B | 0.71 s | 29 ms | 24× |
| Kimi K3 | 3.2 s | 117 ms | 27× |
| Hy3 | 2.4 s | 117 ms | 20× |
All computed, from twice the active parameters per token plus the attention term of each layer type; in a miss each token attends only to those before it. Kimi K3’s weights need two such nodes, so its times are its FLOPs at one node’s rate. The token ratio is 36.5×; the attention to the prefix is 17% of the hit’s cost for DeepSeek-V4.1-Flash and 90% for Hy3, whose 80 GQA layers read every position. vLLM measured the same effect on DeepSeek-V4.1-Flash: on two GB300 GPUs “an 8K-token prefill takes 87.9 ms without prior context, but 189.4 ms with 512K cached context”. Over a session the saving grows, since a miss re-prefills a context that grows every turn. With equal turns, the miss/hit ratio is for the weight matmuls and for full attention, 18.5 and 12.5 at 36 turns, and each model’s session falls between the two (computed).
- miss: prefill the whole context
- hit: prefill only the new tokens
- cache kept between turns
GLM-5.3, 36 turns of 4,000 new tokens (to 144K): prefill over the whole session 18.4 s if every turn misses, 1.03 s if every turn hits (17.8× less).
Turn 36: miss 1.03 s, hit 32.1 ms. Cache the session leaves behind: 7.88 GB (FP8).
Computed, not measured: FLOPs per token from each model's config (twice the active parameters plus attention to every earlier token, sparse where the model is), on eight B200 GPUs at 40% of dense FP8 peak. The cache format changes only the lower panel; the prefill compute is FP8 in both. Hover or tap a turn for its numbers.
A hit also needs the tokens to be identical. One changed token early in the prompt makes everything after it a miss.
SGLang’s GLM-5.3 cookbook documents a case. Claude Code prepends a per-request header to the system prompt, and
GLM-5.3’s chat template puts only the tool list before the system prompt, so “that per-request hash is the first token to diverge
between turns and the radix prefix cache re-prefills the whole system + history every turn”. The fix is on the client:
CLAUDE_CODE_ATTRIBUTION_HEADER=0 drops the header. Caches are also tied to the weights: DeepSeek’s DeepInfer
deployment notes list changes to “model weights, the block byte layout, or the tokenizer” as invalidating
every stored entry.
Tiers: where a prefix waits between turns
Between two turns an agent runs its tool, which can take seconds or minutes, while the GPU’s memory is needed for other
sessions. A server that can only cache in GPU memory (HBM) loses the prefix as soon as the batch grows. Engines
can therefore move idle cache blocks down to host memory (DRAM), local SSD, or a pool shared by many servers, though
vLLM and SGLang both leave this off by default (--kv-offloading-size, --enable-hierarchical-cache).
Moving a prefix back costs bytes over a link; recomputing it costs FLOPs. Loading wins when
so the break-even bandwidth is the one that moves the prefix in the time recomputing it takes. For Hy3 in FP8, 23.3 GB in 2.25 s over eight GPUs, each loading its own eighth, needs 1.3 GB/s per GPU; DeepSeek-V4.1-Flash, with 0.21 GB, needs 0.14 GB/s (computed, at 40% of the B200’s dense FP8 peak). Every link a server has is faster than that, as long as each GPU loads only its own share.
Tensor parallelism (02) splits a cache by KV head, and a cache with fewer KV heads than GPUs is copied instead: the latent of GLM-5.3 and Kimi K3 and DeepSeek’s single key-value head sit on all eight GPUs, and Qwen3.5’s two KV heads on four each. If each GPU then loads its own copy over its own link, the break-even rises four to eight times; for GLM-5.3 it reaches 7.6 GB/s per GPU, and one older SSD per GPU then loses (1.11 s against 1.02 s, computed). Splitting the cache by sequence position (decode context parallelism) or by request (data-parallel attention, where each GPU runs attention for its own requests, I3) can store and load each byte once.
- recompute the prefix
- load the cached prefix over a link
- load slower than recompute
Smallest margin: Hy3 from NVMe Gen4, 415 ms against 2.25 s to recompute (5.4×). Largest: Kimi K3 from host DRAM, 590×.
Sessions of 142K tokens in 1 TiB of host memory (FP8, one stored copy): V4.1-Flash 5,273, Kimi K3 409, Qwen3.5 463, GLM-5.3 141, Hy3 47.
Computed at nominal line rates, one link (or one drive) per GPU, on eight B200 GPUs at 40% of dense FP8 peak; recompute is at that rate in both formats. The NVMe rates are assumed, and achieved rates are lower (about 50 GB/s one way over PCIe on an H100, as vLLM measured). Under tensor parallelism a latent cache (GLM-5.3, DeepSeek-V4.1-Flash, Kimi K3) sits whole on every GPU, and Qwen3.5's two KV heads on four GPUs each; the second loading option shows what that costs when each GPU fetches its own copy over its own link. Sessions per TiB count one stored copy, or one per GPU that needs it.
Achieved bandwidths are lower (vLLM measured about 50 GB/s one way over PCIe on an H100), and small copies are worse: vLLM’s offloading throughput rose “by an order of magnitude” when a new layout grew its blocks to 0.5–2.5 MB. With the load split across the GPUs, loading still wins at those rates. A head-sharded cache comes close to break-even only for short prefixes, whose tokens cost little to recompute: for Hy3 it reaches 6.2 GB/s per GPU at 4K tokens (computed). Once tiers are on, the engines start the load while the request waits in the queue, and SGLang stops waiting for its shared tier after a timeout (2 s plus 0.1 s per 1,024 tokens by default) and recomputes what has not arrived.
Tiers therefore differ mostly in capacity: how many sessions each holds, and for how long. The engines differ in when they write blocks to a lower tier and which servers can read them.
- SGLang’s HiCache calls GPU memory L1, host memory L2 and a distributed store L3 (a distributed KV store or file system, such as Mooncake or DeepSeek’s 3FS). L1 and L2 are private to one server; “A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3”. The default write policy is write-through, “Every access is immediately written back to the next level”; write-back writes “only when it is evicted from the upper level”.
- vLLM’s offloading connector copies a prompt’s blocks to host memory “as they are produced” (write-through; decode blocks are skipped by default), so a host tier smaller than the GPU cache “just mirrors what the GPU already holds and adds no hit rate”. Files, object stores and peer hosts sit behind the host tier.
- Labs pick by workload. Kimi K3’s RL rollouts write idle prefixes to “an external KV cache pool in CPU DRAM only when it is evicted from GPU” (write-back), so as not to copy blocks still in use, and prefetch them back before the next reuse. MiMo-V2.6’s rollouts move a trajectory’s state to host memory during tool time, the known idle gap. DeepSeek’s production system keeps only its sliding windows in host memory and stages loads of the main cache there. Compressed blocks go to a shared SSD cluster on 3FS, “roughly 100 PiB of SSD storage for the KV cache, with a TTL (time-to-live) of 3 days for agent traffic and 7 days for chatbot traffic”.
In cache-hierarchy terms, writing through keeps the lower tier inclusive, so the two hold only as many distinct blocks as the larger of them, which is why vLLM’s small host tier adds nothing; writing on eviction makes it close to exclusive, holding up to both tiers’ worth.
In every measurement, a lower tier paid off only when the GPU cache was under pressure. In vLLM’s tiered-offloading test (Qwen3.6-35B-A3B on two H100s, 12K-token prompts plus 4K per round), GPU-only, host and storage caching all did well up to about 64 conversations. From 64 to 128 “throughput drops dramatically without offloading”, and beyond 128 only the storage tier kept a high hit rate. The fall is a cliff, as LRU suggests when sessions take turns: a session’s prefix survives only if every other session’s fit beside it, so the hit rate holds until their combined cache outgrows the tier. The LMCache team’s test on Claude Code traces (MiniMax-M2.5, two MI300X) found 3.0× lower average TTFT with a host tier than with vLLM’s GPU-only prefix cache when GPU memory was deliberately squeezed (in a 20-minute run that completed 28 and 12 requests), and worse results at a light load, where everything fit in GPU memory. DeepSeek gives the one lab-reported production hit rate with both its period and its denominator, from V3 and R1 in February 2025: of all input tokens over 24 hours, “608B, of which 342B tokens (56.3%) hit the on-disk KV cache”. The DeepInfer notes add that a TTL of about 3 days “costs chatbot traffic 0.5%–1% of KV cache hit rate”.
Prefill/decode disaggregation
Chunking limits how much a prefill slows the decodes it shares a step with, but doesn’t remove it, and a node that does both has to pick one layout for two phases with opposite limits. Prefill/decode disaggregation (P/D) runs them on separate pools of GPUs. A request is prefilled in the first pool, its KV cache is sent to the second, and decoding continues there. DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2024) proposed it. SGLang’s documentation gives two reasons: prefill batches “frequently interrupt ongoing decode batches”, and with data-parallel attention, where each GPU is called a rank, all ranks meet at every expert layer’s exchange, so the decoding ranks wait for the one that is prefilling.
The first cost is the transfer: the decode pool needs the whole context’s cache unless it still holds the prefix from the previous turn.
- whole context (146K tokens)
- the turn's 4K new tokens
- fixed state, sent every turn
Whole context over one 400 Gb/s NIC (50 GB/s) (FP8): V4.1-Flash 4.3 ms, Kimi K3 55.0 ms, Qwen3.5 48.7 ms, GLM-5.3 160 ms, Hy3 478 ms.
Only the turn's 4,000 tokens: V4.1-Flash 0.17 ms, Kimi K3 10.2 ms, Qwen3.5 5.1 ms, GLM-5.3 4.4 ms, Hy3 13.1 ms. Of Kimi K3's increment, 88% is its KDA state; of Qwen3.5's, 76% is its Gated DeltaNet state.
Computed: cache bytes from each model's config, timed over one link at its nominal rate. The axis is rescaled for each view.
Sending each layer as soon as it is computed hides part of the transfer, but only if the decode GPUs are chosen before prefill starts, as Splitwise does (“At the end of each layer, we trigger an asynchronous transfer”); vLLM’s guide counts the whole transfer against TTFT. Standard P/D also moves the cache one way, and in a multi-turn session that wastes work: “On turn two, decode still holds KV for the answer it just generated but prefill has never computed it, so prefill recomputes it” (vLLM’s guide). The fixes send the cache back (vLLM’s bidirectional transfer), keep it on the decode side, or have decode write what it generated into the shared store, as DeepSeek and vLLM’s Mooncake store do. For a hybrid model that write matters most, since the state at the end of the output is the one the next turn resumes from. SGLang’s decode-side prefix reuse “asks prefill to send only the missing suffix”; on GLM-5.2-FP8 with eight B300s per worker and synthetic conversations growing to 160K tokens, it cut median TTFT by 60.3% and average TTFT by 25.4%, with GPU and host tiers together.
P/D also leaves two pools to size. DistServe’s example (a 13B model, 512 input and 64 output tokens) put two prefill GPUs per decode GPU in its ideal allocation. DeepSeek reports a P:D ratio of “roughly 1:1” in GPUs and a smallest efficient unit of “64 GPUs of prefill plus 64 GPUs of decode”, because its decode runs wide expert parallelism. vLLM’s AgentX post explains why there is no rule: context lengths and hit rates “vary wildly across sessions”, and the best ratio “also shifts with concurrency”. A cache hit moves work out of prefill: if a share of requests hit, a request’s prefill costs of a miss plus of a hit, so for GLM-5.3 at 142K a hit rate of 56% cuts the prefill pool needed 2.2-fold and 90% about eightfold (computed, ignoring queueing).
What it buys is easy to overstate. vLLM’s documentation says it “DOES NOT improve throughput”; it separates the tuning of TTFT and ITL, and its later guide calls the gain goodput, the request rate served within both latency targets. The measured throughput gains for NVIDIA’s 7 + 4 split for Nemotron 3.5 Super VL and vLLM’s 4 + 4 recipe for Mistral Large 4, hold under a minimum per-user speed on their own setups (02), the Mistral one on unique prompts with prefix caching off. NVIDIA’s split also raised median TTFT from 0.49 s to 1.12 s.
Routing: affinity or a shared store
A cache in one server’s GPU or host memory helps only the requests that reach that server. With many servers, which server a turn reaches decides whether it hits.
The simplest rule is session affinity: hash a session id to a server, so every turn lands where its prefix is. vLLM’s Mistral Large 4 recipe routes each session to the same data-parallel rank and reports that without it “the hit rate drops 14–22 points and TTFT p99 is 2–3x worse”. vLLM’s AgentX post found that balancing by queue depth, running tokens or cache use “all … underperformed simple session-aware sticky routing”, because the next turn often arrives while the prefix is still on the previous GPU. Kimi K3 pins each session by consistent hashing to a primary cluster and a secondary, spread across the fleet, that “holds none of the session’s prefix cache and must re-prefill it upon failover”.
The price of affinity is balance: a hot session can’t move, and a server that fails takes its sessions’ caches with it. Cache-aware routers trade the two explicitly. SGLang’s gateway keeps an approximate radix tree of what it has sent to each worker. It routes to the best match if more than 30% of the prompt’s characters match, and falls back to plain load balancing when the busiest and idlest workers’ loads differ by more than 64 requests and by more than 1.5 times. That tree records what the gateway sent, not what each worker still holds, so it misses evictions; NVIDIA Dynamo and llm-d instead follow events that the engines publish when blocks are stored or freed. Dynamo prices each worker by the blocks it would have to prefill, minus credits for blocks it holds in GPU, host or disk, plus its decode load, so “A cache-hot worker can lose to a colder worker when its active load makes the total cost higher”.
The other design takes most of the conflict away. DeepSeek stores the main cache on a storage cluster that every prefill server reads, which “lets every prefill machine hit any prefix, avoiding a conflict between prefix affinity and the load-balancing routing policy”, and “there is no prefix-affinity routing”: engines pull work by remaining compute quota. The price is a transfer on every hit and a store that must keep up; DeepSeek’s sizing asks for “at least 40 GB/s read and 5 GB/s write” per storage server. A vLLM test with a shared Mooncake store and deliberately round-robin routing kept the hit rate “above 95%”. A shared store turns a lost hit into a load, which still costs: in vLLM’s AgentX tests sticky routing won with a distributed pool in place too, since a moved session’s prefix must be fetched and occupies the new GPU’s cache while it arrives. DeepSeek can skip affinity partly because its V4.1 cache is small, 0.21 GB per 142K-token session in FP8.
Hybrid models: state that resumes only where it was saved
Everything so far assumed a cache with one entry per token, valid for any prefix: the cache of the first tokens is the first entries. Hybrid models break that in three ways, and the hit rule becomes the intersection of three different rules.
A sliding-window layer needs only the last entries before the hit point. vLLM’s hybrid manager accepts a hit if the
last “sliding_window_size - 1 tokens” are still cached and frees blocks that fall out of the window (they stay cached
until evicted); SGLang’s radix cache marks them as dead (“tombstones”) but keeps the tree. Per copy the ring is small:
5.2 MB per sequence for DeepSeek-V4.1-Flash in BF16, about 1% of its 0.46 GB of cache at 142K.
A recurrent layer has no per-token entries at all. Its state after token summarises the whole prefix and is overwritten at token , so a hit at needs a copy saved at exactly . SGLang’s K3 post: “A recurrent state cannot run backwards and must be replayed forward from an earlier checkpoint.” Both engines copy a saved state into a private slot before extending it (copy-on-write), so the shared copy survives. Saved states are large. One Kimi K3 state for all 69 KDA layers is 449 MB in 32 bits, as much as 16,253 tokens of its MLA cache in BF16 stored once; one Qwen3.5 state is 192 MB, 6,252 tokens’ worth (computed). In GPU memory under eight-way tensor parallelism, where every GPU holds the whole latent but only an eighth of the state, the K3 figure is about 2,000 (SGLang gives 54 MB against 27 KB per token per GPU). NVIDIA counts per layer: on its Mamba-2 model one layer’s state (a Mamba cache page) “occupies at least as many bytes as approximately 4K tokens of Attention KV cache” in vLLM’s layout.
Above, a toy: a hit must end where every layer type can resume. Below, computed for one session of 142K tokens.
- saved snapshots
- the session's KV cache (BF16)
- recompute after a hit
Kimi K3 (state in 32 bits): one snapshot (KDA states + conv tails, FP32 state) is 449 MB, as much as 16,253 tokens of its BF16 cache.
Every 512 tokens: 277 snapshots = 124 GB, 28.4× the session's 4.38 GB cache. A hit at a random token recomputes 256 tokens on average: 7.4 ms, against 117 ms for a 4K-token turn.
Storage × expected recompute: 926 GB·ms, about the same at every interval: the slider only trades one for the other. Keeping the newest state only: 449 MB; one per turn (35): 15.7 GB, 3.6× the cache.
Computed, not measured: a hit at a random token resumes from the snapshot at or before it, and the tokens since are prefilled at the model's FLOPs per token at 142K, on eight B200 GPUs at 40% of dense FP8 peak. Sizes count one stored copy. Per GPU under tensor parallelism the comparison shrinks, since a state is split by head while a latent cache sits whole on every GPU: one Kimi K3 state is 16,250 tokens of cache in one copy, about 2,030 on each GPU. Axes are linear from zero and stay fixed while the slider moves.
So engines and labs save recurrent states sparsely. vLLM’s hybrid manager stores Mamba states only “on the Mamba block grid” and makes attention blocks as large as a state in bytes, so that one allocator with one page size can hold both. An attention page of that many bytes holds hundreds or thousands of tokens: 400 to 4,224 in the Nemotron deployments of vLLM’s hybrid post. Kimi K3’s report describes where that leads: the shared block size was “forced to 1024–6144 tokens”, and “At such a coarse granularity caching is nearly useless: requests shorter than one block can never be reused”. Its fix separates matching from storage. Hashes run on smaller blocks inside the large physical block (512 tokens in the report’s example), and states are saved “only at (a sparse subset of) MLA’s hash endpoints”, typically at conversation-turn boundaries, where an agent’s next request resumes. Moonshot’s engine also packs K3’s states into the same paged pool as the MLA cache, so one allocator and one eviction policy serve both. SGLang saves a state at each prefill chunk boundary and every 256 decode tokens but keeps only a few. Unless the kernel saves mid-step, as Moonshot’s does at the last hash-aligned position, the scheduler’s chunks are also the save grid.
Saving every tokens trades storage of bytes, for a state of bytes, against an expected recompute of tokens on a hit at a random point, a product that stays the same as moves: the trade-off that sets how often a long job checkpoints against failure. It matters only when hits can land anywhere. A missed boundary costs a recompute from the last saved state: nothing for an agent whose turns end at saved states, more for requests that branch mid-turn, such as retries and several samples from one prompt.
Speculative decoding (P1) makes recurrent state costlier twice over. A verify step must keep the state after every draft token it might accept, since it cannot roll back, and NVIDIA’s Nemotron 3.5 and SGLang both size extra state buffers for this. A drafter resuming after a hit also needs the target model’s hidden state at the hit point, which vLLM’s prefix cache lacks, so vLLM replays from the last saved state; NVIDIA added an option to keep one extra checkpoint just before that boundary.
DeepSeek’s compressed cache is the third case. Its CSA and HCA layers write one entry per 4 and per 128 tokens, so a prefix that ends inside a group has entries that don’t exist yet. The V4 report stores “all of the compressed KV entries”, reuses them “until the last complete compression block”, and recomputes the tokens of the tail; a cache block must hold a multiple of lcm(4, 128) = 128 tokens, so that its boundary is a group boundary for both layer types, and V4.1’s production block is 128 tokens. The V4 report’s numbers are for V4-Flash, an older model with 43 layers and 0.98 GB of compressed cache at 142K. For the sliding windows the report weighed storing every token’s window entries, saving the window every few tokens, or storing none and recomputing. An exact rebuild reaches back one window per layer, since each layer’s window entries depend on the layer below over the previous tokens: at most = 128 × 43 = 5,504 tokens. In production V4 kept the window at two points per turn and it still took “nearly half of the persistent KV cache capacity” (V4.1 report). One saved window set weighs as much as about 800 tokens of V4-Flash’s compressed cache (computed), so two per turn weigh as much as the entries of 1,600 new tokens, and the report singles out “multi-turn conversations with short turns”. Storing none “proved prohibitive in production deployments”. V4.1 moved the windows out of the SSD store into a pool of host memory with a TTL of minutes. When the compressed prefix hits but its windows have expired, it replays only the last 128 tokens. That replay is approximate (“not mathematically identical”), and the report finds it “barely compromises response quality”. Together with the smaller compressed cache, the persistent store fell to 1/8 of V4’s under the same workload.
The mechanisms side by side
| Mechanism | What it keeps | What it buys | What it costs | Main knob |
|---|---|---|---|---|
| Paged blocks | cache in fixed blocks | memory waste of at most one block per request | an indirection per block | block size |
| Chunked prefill | nothing new | decodes never wait for a whole prompt | per-chunk overhead; budget trades TTFT for ITL | token budget per step |
| Prefix cache in HBM | finished requests’ blocks | a turn costs its new tokens plus their attention | HBM taken from live sessions; evicted under load | eviction policy |
| Lower tiers | idle blocks in DRAM, SSD or a pool | capacity for hours; load beats recompute when each GPU loads its share | memory and storage held per session; transfers; write traffic | write policy, TTL |
| P/D | nothing new | no prefill in decode steps; one layout per phase | a transfer per request; two pools to size | P:D ratio |
| Affinity routing | nothing new | hits from server-local caches | load imbalance; failover re-prefill | balance thresholds |
| Shared store | blocks every server can read | hits under any routing | a transfer on every hit, still cheapest where the prefix already is; store bandwidth | store size, TTL |
| Hybrid snapshots | recurrent states and windows at chosen points | hits for models without per-token state | a state is worth thousands of tokens; hits only at saved points | snapshot points |
Where the models in the map sit
The configs give what each model must keep. For a few labs, the reports also say how they keep it. Of the 35 language models with a code-check row (16 labs), 19 models from 9 labs have sliding-window attention somewhere, and 8 models from 4 labs (Qwen, Moonshot, Z.ai, NVIDIA) interleave recurrent layers with attention (the code checks, cleaned as in T5). Of these 13 labs, 3 describe how they cache that state in serving: DeepSeek, Moonshot and NVIDIA (Xiaomi does for RL rollouts).
| Model | Grows with context (layers) | Fixed per sequence | At 142K, BF16 | What the lab says about reuse |
|---|---|---|---|---|
| Hy3 | GQA (80) | none | 46.5 GB | no lab document in the corpus |
| GLM-5.3 | latent + indexer keys (78) | none | 13.5 GB | RL: P/D, rollout-to-rank hashing (GLM-5 report) |
| Mistral Large 4 | latent (63 own, 9 shared) | none | 11.9 GB | P/D “to eliminate prefill interrupts”; FP8 cache and offloading |
| Gemma 4 31B | full attention (10) | 50 windows of 1,024, 839 MB | 12.5 GB | nothing |
| MiMo-V2.6-Pro | full attention (10) | 60 windows of 128, 39 MB | 7.3 GB | RL: host offload during tool time |
| Qwen3.5-397B-A17B | GQA (15) | Gated DeltaNet (45), 192 MB | 4.6 GB | nothing |
| Kimi K3 | MLA (24) | KDA (69), 449 MB | 4.4 GB | KDA state in the MLA page pool; 512-token hash blocks; cluster affinity; admission budgets |
| Nemotron 3.5 Super VL | attention (8) | Mamba-2 (40), 170 MB | 1.3 GB | coarse Mamba pages; extra checkpoint for speculative decoding; 7P + 4D |
| DeepSeek-V4.1-Flash | compressed, shared (4) | 40 windows of 128, 5.2 MB | 0.46 GB | 128-token blocks; SSD store; windows in host memory; bounded replay (V4: compressed entries whole, tail recomputed) |
Sizes are computed from the configs (two bytes per value, recurrent states in 32 bits, MTP layers excluded; 02‘s 7.8 GB for GLM-5.3 in FP8 includes its MTP layer). “Nothing” means the lab’s own documents, searched, say nothing about cache reuse.
Easy to get wrong
- A hit still costs attention: the new tokens attend over the cached prefix, 17% to 90% of a hit’s cost at 142K for these models, and more at longer contexts.
- “Offloading is slower than recomputing” is false for these models at agent lengths whenever each GPU loads its own share of the cache. A lower tier fails when it is too small or not shared, or when every GPU must load its own copy of a latent cache from a slow drive.
- Routing alone moves the hit rate: vLLM’s Mistral recipe loses 14–22 points when sessions stop landing on their cache.
- P/D’s measured gains are goodput under a per-user speed floor, each on its own setup, and the split adds a transfer to the first token.
- Many lab statements on caches describe RL rollouts: Kimi K3’s DRAM write-back pool, GLM-5’s P/D and rank affinity, MiMo’s tool-time offload. Public serving may differ.
Check yourself
A model’s cache is 0.5 MB per token in FP8, and recomputing a 100K-token prefix on its node takes 4 s. Its servers hold idle prefixes on local SSDs that read 5 GB/s per GPU, eight GPUs per node, each GPU holding an eighth of the cache. Is loading worth it? What break-even bandwidth per GPU does this case imply?
The prefix is 100,000 × 0.5 MB = 50 GB. Over eight drives at 5 GB/s it loads in 50 / 40 = 1.25 s, against 4 s of recompute: loading wins by 3.2×. Break-even is where loading also takes 4 s: 50 GB / 4 s = 12.5 GB/s per node, about 1.6 GB/s per GPU. If the cache were copied on every GPU instead, each would read all 50 GB at 5 GB/s, 10 s, and recomputing would win.
Why can’t a prefix cache for Kimi K3 hit at an arbitrary token, the way it can for GLM-5.3?
GLM-5.3 keeps one cache entry per token in every layer, so the first entries are the cache of any -token prefix. Kimi K3’s KDA layers keep one state per sequence that is overwritten at every token, so a hit needs a state saved at the hit point. A state is 449 MB, as much as 16,000 tokens of its MLA cache, so states are saved only at a few points, typically turn boundaries, and a hit can land only there (or recompute from the last one).
DeepSeek routes prefills with no session affinity at all. Why doesn’t its hit rate collapse?
Its main cache sits on a storage cluster every prefill server reads, so any server can load any prefix. Affinity only matters when a cache is private to one server. The price is a load from the store on every hit and a store sized to serve all of them; DeepSeek-V4.1-Flash’s cache is small enough (0.21 GB per 142K-token session in FP8) that the load is cheap.
In the map
Methods
- Continuous batching 1 source
- Chunked prefill 6 sources
- Prefill-decode disaggregation 4 sources
- Disaggregated serving 2 sources
- Disaggregated serving across 4+4 GPUs 1 source
- Mooncake 1 source
- Prefix caching 10 sources
- RadixCache 2 sources
- HiCache 2 sources
- KV-cache offloading 1 source
- Asynchronous KV-cache offloading 1 source
- Native FP8 KV cache with offloading 1 source
- Prefix-cache-aware session affinity scheduling 1 source
- KDA-aware prefix-cache management 1 source
- Unified paged cache layout for KDA states and MLA KV 1 source
- Mamba prefix caching in align mode 1 source
- Round-robin routing for prefill-decode disaggregation 1 source
Taxonomy nodes
- KV cache management 53 methods
- inference scheduling 35 methods
Further reading
- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (2023): paged blocks, sharing and the swap-or-recompute measurements.
- Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (2024): chunked prefill, stall-free schedules and how to pick the token budget.
- Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” (2024): RadixAttention and cache-aware scheduling, with the proof that longest-prefix-first order maximises hits.
- Zhong et al., “DistServe” (2024) and Patel et al., “Splitwise” (2024): the case for disaggregation, goodput, and how to place the two pools.
- Qin et al., “Mooncake” (2024): Kimi’s KV-cache-centric serving, with a scheduler that weighs prefix length against load.
- DeepSeek-V4 report §3.5 and DeepSeek-V4.1 report §3.2, plus DeepSeek’s DeepInfer deployment Q&A: the most detailed account of a production cache for a hybrid model.
- Kimi K3 technical report §5.3–5.4: hybrid prefix caching, admission budgets and cluster affinity.
- NVIDIA Nemotron 3.5 Super and Lightning report §5.3–5.6: Mamba checkpoints with speculative decoding, and a measured P/D split.
