Most layers can be cheap if a few global layers handle long-range retrieval: the growing cache shrinks with the ratio of cheap to global layers, and sharing caches across layers shrinks it further, with quality losses one lab reported
Gemma 4 31B has 60 layers. Holding a 128K-token prompt in BF16, its key-value (KV) cache, the
keys and values every layer keeps for every past token, takes 10.78 GiB. If all 60 layers had
the shape of its ten global layers and attended over the whole context, it would take 60 GiB
(both computed from config.json below). The difference comes from one line of the config: 50 of
the 60 layers are sliding_attention with sliding_window: 1024, so they keep only the last
1,024 tokens, and only the other ten keep everything.
None of the token mixers in T1 to T3 is both cheap and a full retriever. Softmax attention over
the whole context can read any past token on demand, and its cache grows with every token. A
sliding window, a recurrent state or a sparse selection bounds the cost and gives up some of that
access. So most of the 35 open-weight models this site tracks (the map) stack layer types: many
cheap layers and a few expensive global ones. A second, newer approach works across layers: one
layer produces keys, values or sparse indices, and others reuse them instead of producing their
own.
Background
Generating text has two phases. Prefill runs the whole prompt through the model at once and
writes the KV cache; it is limited by arithmetic (FLOPs). Decode then generates one token at a
time, and each step reads the cache of every layer; with one query per sequence that step is
limited by the bytes it reads (T1). The cache also sets how many sequences fit on a GPU at once.
A global softmax layer with g KV heads of width dh caches 2gdh numbers per token, at
b bytes each, so 2gdhbT bytes at context length T (T1). The cheap layer types from T2
and T3 cache less:
A sliding-window attention (SWA) layer attends to the last W tokens only. Its cache is a ring
buffer of W entries that overwrites the oldest one, so it stops growing at T=W. (Window
sizes here are the configs’ sliding_window, which counts the current token: Kolibri’s 513 is
its report’s “512 preceding tokens”.)
A linear-attention or state-space layer (Gated DeltaNet, Kimi Delta Attention, Mamba-2; T2)
folds the prefix into a fixed-size state matrix, after a short causal convolution over time
whose last few inputs it also keeps. Its cache does not grow with T.
A compressed or sparse layer (T3) keeps one entry per m tokens, or attends to a few entries
that a small scoring network, the indexer, picks per query; its cache still grows with T.
Write nG for the number of global layers, nS for the window layers, nR for the recurrent
ones, eG and eS for the numbers cached per token by one global and one window layer, and s
for the numbers in one recurrent state, convolution buffer included. Recurrent states and keys are
often stored at different widths, so write bKV and bR. The cache of a stack with
L=nG+nS+nR layers is
M(T)=bKV[nGeGT+nSeSmin(T,W)]+bRnRs.
Only the first term grows. Compare it with the same stack built from L global layers of the
global shape. With r=(nS+nR)/nG cheap layers per global layer and T≥W,
T∗ is the context at which the fixed caches hold as much as the growing one. The cut tends to
r+1 and is half of it at T=T∗. A 5:1 stack approaches a sixth of the all-global cache, a
3:1 stack a quarter, and going from 1:1 to 5:1 shrinks the growing part threefold, from half the
layers to a sixth. For a window stack T∗=rWeS/eG, so raising the ratio raises the bound
and pushes T∗ out in the same proportion.
Interleaving local and global attention
The oldest hybrid is local and global softmax attention in alternation. Longformer (2020) put a
sliding window and a few global tokens inside one layer. Gemma 2 (2024) made the split by layer: “We alternate between a local
sliding window attention … and global attention … in every other layer”, with a 4,096-token
window and an 8,192-token context, so half the layers were global. Gemma 3 (2025) moved to five
local layers per global layer and a window of 1,024, so that “only the global layers attend to long
context”, because of “the memory explosion of the KV cache”. Gemma 4 kept 5:1 (4:1 in its smallest
model).
Twelve of the map’s 35 models, from eight labs, interleave window and global layers, as their
configs and code show. Ratios run from 1:1 (gpt-oss-120b, window 128: “Following GPT-3, attention
blocks alternate between banded window and fully dense patterns”) to 6:1 (MiMo-V2.5-Pro, window
128). LFM3-1.7B (Liquid) mixes three kinds: 13 gated short-convolution layers, 5 window layers and 6
global ones. The table near the end lists every model.
Hover or tap a cell for its layer index and kind. Triangles mark the first and last layer.
Figure 1.Layer kinds in twelve models of the map, layer 0 on the left. Ratios run from 1:1 (gpt-oss) to 6:1 (MiMo Pro); Laguna, LFM3, GLM-5.3-Flash and Nemotron end on a cheap layer; DeepSeek-V4-Flash has no plain global layer at all. Read from each model's config.json.
Cache size by ratio
For Gemma 4 31B, a global layer caches 2⋅4⋅512=4,096 numbers per token
(num_global_key_value_heads: 4, global_head_dim: 512), 8 KiB at b=2, so ten of them hold
10⋅8KiB⋅131,072=10 GiB at 128K. A window layer has four times the KV
heads at half the width (num_key_value_heads: 16, head_dim: 256), so it caches 8,192 numbers
per token, twice what a global layer does, but only for 1,024 tokens: 16 MiB per layer, 800 MiB for
all fifty. The total is 10.78 GiB against 60 GiB for 60 global-shape layers, a cut of 5.57 (computed).
Two choices hide in that 60 GiB. If the window layers kept their own, wider shape and lost their
window, the all-global model would hold 110 GiB, a cut of 10.2. And Gemma 4’s global layers compute
values from the key projection (attention_k_eq_v: true), but the reference code normalizes and
rotates the keys and only normalizes the values, so both are cached, as we count them. Google’s report
says this reuse, with rotation on a quarter of the dimensions, “effectively reduc[es] the global KV
cache by 37.5%”; our reading is an engine that stores the values plus the rotated quarter of the keys,
which would bring the 128K cache to 7.03 GiB (computed).
The cut stays below six because the window caches are a fixed cost: T∗=5⋅1,024⋅2=10,240, so at 8K the cut is only 2.7, it reaches 5 at 5T∗=51,200 tokens, and below 2,048
tokens the hybrid holds more than 60 global-shape layers. Kolibri-1 has 50 layers, 10 global, and four KV heads of width 128
in both kinds, so T∗=4⋅513=2,052 and its cut at 128K is 4.92 against a bound of 5.
MiMo-V2-Flash’s report states “nearly a 6× reduction in KV-cache storage and attention computation
for long contexts”, from a 5:1 ratio. That counts one block of six layers. The MiMo-V2.5 and
V2.6-Flash configs have 48 layers, eight blocks of six whose very first layer is also made global (nine global layers: 0, 5, 11, …, 47), so a bound of 48/9=5.33
and, with global layers caching 1,280 numbers per token and window layers 2,560, a cut of 5.29 at 128K
(computed). The V2.5-Pro card’s “nearly 7x” holds: 10 global layers of 70 give 6.96.
The cache cut is a capacity figure. A decode step also reads the weights, which dominate at one
sequence per GPU, so the cut shows up as more sequences per GPU, and as fewer bytes per step once
many sequences share the weights. Kolibri’s report measures the long end: at 256K, “Most of this
long-context traffic comes from the 10 full-attention layers”. In prefill, attention FLOPs fall about
as much as the cache (5.77 times for Gemma 4 at 128K, computed), but projections and feed-forward
blocks do not, so the whole prefill falls by less.
At T = 128K (131,072 tokens): 10.78 GiB, all-global 60.00 GiB, cut 5.57 against a bound of L/n_G = 60/10 = 6.00. Fixed caches equal the growing one at T* = 10,240 tokens. Cheap-layer share of the cache: 7.2%.
global layers' KV
window ring buffers
recurrent states
all layers global
T*: fixed caches = growing one
y axis rescales to the marked T
Figure 2.KV cache against context length for one sequence. A hybrid's growing part is 1/(r+1) of an all-global stack's, but window buffers and recurrent states are a fixed cost that dominates below T*: for Gemma 4 31B, T* is 10,240 tokens. The toggles show deployed widths: with FP8 keys and an FP32 state, Nemotron-3-Ultra's T* moves from 16,960 to 67,840 tokens. Computed from each model's config.
Quality cost, and what reduces it
A window layer cannot read a position more than W−1 back. Information from further back can
still arrive, as in a stack of convolutions: through k window layers it can travel up to
k(W−1) positions, but only if each intermediate position copied it forward before any query
asked for it. A global layer is the only place where a query fetches a far token by content. The
Qwen3.8-Next report puts it as “information outside the window can only propagate indirectly
through depth”. T3’s error formula applies unchanged, since a window is a selection fixed in
advance: if a global layer put attention mass ε beyond the window, restricting it to the
window changes its output by ε(oˉout−oS), the gap between the far and
the near values weighted by that mass.
Labs pair aggressive ratios with a few changes to the layers. A learnable attention sink is a
per-head logit in the softmax denominator that matches no token, so a head can put little or no
weight on any token. Without it the weights must sum to one, so in a short window with nothing
relevant a head has to spread its weight over irrelevant tokens; this is our reading of why it
matters more for windows. gpt-oss and DeepSeek-V4 put a sink in every layer, the MiMo configs in the
window layers only (add_swa_attention_sink_bias: true). An output gate scales each head’s output by
a learned function of the input (a sigmoid in LFM3 and Qwen, a softplus in Laguna), so a head can
switch itself off. And the two kinds often get different position codes. Rotary position embedding
(RoPE) rotates pairs of dimensions by an angle that grows with position; partial RoPE rotates only
some, and no positional encoding (NoPE) none. Kolibri’s global layers use NoPE, and Gemma 4’s rotate a
quarter of their dimensions (T4).
Three labs printed ablations of these choices, at a fixed ratio and on test models, not on the
shipped ones. Their scores are single runs.
On a 32B dense test model, Xiaomi found that a window-128 hybrid without a sink scored lower than
the all-global model (MMLU 54.9 against 57.3), and with one matched or beat it. After continued
training at 32K, it scored 89.4 on RULER (synthetic long-context retrieval, out of 100), as the
baseline did, and a window-512 hybrid 84.7. The report’s explanation, from its own settings, is “a
clearer division of labor”: a tight window leaves long-range work to the global layers.
Aleph Alpha swept Kolibri’s window from 128 to 4,096 on a 7.8B proxy, trading window length
against KV heads and expert shapes to keep FLOPs, KV bytes and parameters about equal. Scores were
“generally similar”; on a larger proxy, 512 beat 128 by 6.7 points on long-context tasks at 256K.
Poolside added changes one at a time to a 16.6B Laguna test model at 3:1. Windows of 1,024 lowered
the short-context average from 0.5389 (all-global) to 0.5292, against a noise level of about 0.005;
gating, partial RoPE on the global layers and a 512 window brought it to 0.5449, and reallocated
query heads to 0.5455.
One lab reports the opposite. Mistral Large 4 was first trained with one global layer in four and
windows of 512, like Laguna. After midtraining (curated, often longer data between pre-training and
fine-tuning) and supervised fine-tuning (SFT), its checkpoints scored lower than expected on long-context agentic tasks, and ablation runs at the same token budget isolated the cause: SWE-bench Verified scores, where a model edits a
real repository over many steps and so reads a long, growing context, “decreased as we moved from
full attention to SWA”. Mistral then removed the windows.
The two kinds also get different shapes: Gemma 4’s window layers have four times the KV heads of its
global ones and MiMo-V2.5’s twice, which costs little memory with the cache capped at W.
Recurrent layers instead of windows
A window layer has no access to tokens older than W. A recurrent layer folds the whole prefix
into its state. Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) update a state matrix per head,
St=αt(I−βtktkt⊤)St−1+βtktvt⊤, with key and value widths
dk and dv (128 each in Qwen3.6): the β term overwrites what is stored under the key
kt, the delta rule of Widrow and Hoff, and the decay αt lets old content fade (one scalar per head in GDN, one value per key channel in KDA). Mamba-2
updates its state through a similar input-dependent decay (T2). The Qwen3.8-Next report gives the
split of work: “GDN compresses the prefix into a fixed-size recurrent state … while the
interleaved global-attention layers retain direct token-level retrieval that is difficult for any
finite-state recurrent memory to reproduce exactly.”
There is a precise version of that limit. Reading the state is linear, o=S⊤q, so a head can
return n stored values exactly for arbitrary values only if the n queries that read them are linearly independent,
which needs n≤dk: at most 128 associations per head in Qwen3.6, against 131,072 positions at
128K. Past that count, recall degrades through interference.
Eight models from four labs mix recurrent and softmax layers. Qwen3.5 and 3.6 repeat “3 * (Gated
DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE)” (Qwen3.5 card); Qwen3.8 keeps the pattern with
sparse global layers. Kimi K3 (Moonshot) runs blocks of three KDA layers and one gated multi-head
latent attention (MLA) layer, plus “an additional Gated MLA layer … at the end of the backbone”:
69 KDA and 24 MLA layers in all. MLA caches one latent vector of width dc per token instead of
per-head keys and values (T1). GLM-5.3-Flash (Z.ai) has 34 KDA and 11 sparse MLA layers, a ratio
read from its config. Nemotron 3 Ultra and 3.5 (NVIDIA) put a few attention layers, dense, with two
KV heads and no position code, among Mamba-2 layers at irregular spacing, 3.8:1 to 5:1 counting
token-mixing layers only. NVIDIA calls them “sparse global Attention anchors”, sparse in the sense
of few. Before these, Jamba (2024) found “virtually no performance difference” between 3:1 and 7:1
on a 1.3B model.
In Qwen3.6-35B-A3B a global layer caches 2⋅2⋅256=1,024 numbers per token. A GDN
layer keeps 32 value heads of 128×128 plus a convolution buffer of 8,192 channels by 4:
557,056 numbers, 1.06 MiB in BF16. So T∗=3⋅557,056/1,024=1,632, and at 128K the
model holds 2.53 GiB against 10 GiB all-global, a cut of 3.95 (computed); at the configured FP32
state, T∗ is 3,264. In Qwen’s ablation, on a 25B-A3B test model with one global layer in four, the
GDN hybrid averaged 53.81 over nine short-context benchmarks, a window-128 hybrid 51.15 and the
all-global model 49.87; the results “do not by themselves isolate which architectural component
causes each improvement”.
Recurrent layers have other costs. The state is large even for a short prompt: Nemotron 3 Ultra’s
“32-bit Mamba cache is larger than the FP8 KV cache at sequence lengths up to 64K”, which is T∗
with bR=4 and bKV=1, 67,840 tokens (computed), and NVIDIA moved to 16-bit states.
Speculative decoding, where a small drafter proposes a few tokens and the model checks them in one
pass, must undo rejected tokens: a KV cache drops its last entries, but a state that absorbed them
cannot be un-updated, so engines keep copies (below).
Where the global layers go
Two labs state a rule for the end of the stack. Kimi K3 adds its extra MLA layer so that “the final
layer always performs global attention”, and Gemma 4’s card says the interleave is arranged
“ensuring the final layer is always global”. Neither gives a reason, and six models in the map end
on a cheap layer: Laguna-S-2.1 on a window, LFM3 on a convolution, GLM-5.3-Flash on KDA, and all
three Nemotron models on Mamba-2. Longformer found wider windows near the top better than near the
bottom on a small character model, the same direction.
At the start, MiMo puts a global layer first “to stabilize early representation learning”, as do
Laguna and Step-3.7; Gemma 4, Kolibri and gpt-oss start with a window layer, and DeepSeek-V4-Flash and
V4.1-Flash with two window-only layers.
Only one lab report measures where global layers go, and it argues against hybrids. In the GLM-5
report, Z.ai converts a 40-layer GLM-9B model, trained with full attention to 128K, so that half its
layers become windows of 4,096 tokens. Converting every other layer, without retraining, drops RULER
at 128K from 75.28 to 6.51. A beam search over which layers to convert, scoring each candidate on
RULER at 16K, keeps 53.95. After 190B tokens of further training at 64K, the regular interleave
recovers to 44.93, a drop of 30.35 points, and the searched pattern to 69.59, a drop of 5.69. No
gradient passes through which layers are windows; the search chose the pattern, scored on the
benchmark the table reports, and the error formula above suggests it found layers that put little
attention mass beyond 4,096 tokens. The report concludes that
even the best method keeps “an inherent accuracy gap on fine-grained retrieval tasks”, and GLM-5
uses sparse attention in every layer instead. This is a retrofit of a model trained as all-global,
and Z.ai later shipped GLM-5.3-Flash as a hybrid.
full attention (baseline)
regular interleave
searched pattern
GDN
SimpleGDN
No retrainingAfter 190B tokens of training at 64K
GLM-9B, 40 layers, half converted to 4,096-token windows. RULER score, 0 to 100; at 128K after training the regular interleave scores 30.35 points below full attention.
Figure 3.GLM-5's retrofit study on a 9B model, converting half the layers to 4,096-token windows. A regular interleave collapses past the window without retraining (6.51 on RULER at 128K) and stays 30 points down after 190B tokens; layers picked by a search, which was scored on RULER at 16K, keep most of the score. Scores from the GLM-5 report; a retrofit, not a hybrid trained from scratch.
Two mixers in one layer
A hybrid can also combine branches inside a layer. DeepSeek-V4 compresses its cache in two ways
(T3): Compressed Sparse Attention (CSA) merges every four tokens into one entry and lets an indexer
pick 512 (Flash) or 1,024 (Pro) of them per query, and Heavily Compressed Attention (HCA) merges
every 128 tokens. V4 alternates the two, a hybrid of two global mechanisms. Neither can see the
query’s own, still uncompressed block, and “recent tokens usually possess greater relevance”, so each
layer also keeps keys and values for the last 128 tokens, and the code puts window and compressed
entries through one softmax with a sink. One softmax over two sets is a mixture weighted by each
set’s share of the normalizer; Native Sparse Attention (T3) learns that gate from the query instead.
Since no V4 layer is global without a window, we do not count V4 among the window interleaves.
Sharing across layers
Cross-layer sharing removes some layers’ caches instead of shrinking them. Cross-Layer Attention
(CLA, Brandon et al., 2024) computes key and value projections in only some layers; the others
attend with their own queries to a lower layer’s keys and values. It ties activations, not weights: layer ℓ computes softmax(qℓKj⊤)Vj over the keys and
values of layer j<ℓ, so it keeps its own choice of which past tokens to weight, but sees every
past token as it was at depth j. It is MQA’s sharing across heads (T1) applied across layers. Sharing over pairs (CLA2) halves the
cache: on 1B and 3B models trained on up to 100B tokens it cost “at worst a very modest (less than 1%
change) degradation in perplexity”. The paper also notes that “even shared KV cache layers must be
separately re-read from main memory in each attention layer”, so CLA cuts the memory a sequence
holds, not the bytes a decode step reads.
You Only Cache Once (YOCO, Sun et al., 2024) shares more. The bottom half of the network uses cheap
layers, and the hidden state at layer L/2 is projected once into keys and values that every upper
layer attends to. The upper layers write no keys or values of their own, and their outputs at prompt
positions are discarded except at the last one, so prompt tokens can stop at the midpoint: “at least
half prefilling latency reduction”, with the same result.
DeepSeek-V4.1-Flash’s Causal Encoder-Decoder (CED) builds on YOCO. Its 40 layers, numbered from 0,
are a 20-layer encoder and a 20-layer decoder whose global keys and values are “projected directly
from the hidden state of the (L/2)-th layer”. Like a classical decoder, CED’s equations give each
decoder layer its own projection, but in the shipped configuration only layer 20 makes one and the
other 19 read it: one global cache, as in YOCO. Every layer also keeps its own 128-token window,
computed from its own hidden states, and that keeps prompt tokens from skipping the decoder. Exact
window entries for the last 128 positions would depend, through the windows below, on the last
128+19⋅127=2,541; DeepSeek replays only 128, truncating each window to them, which
“approximately reconstructs the required SWA KV states”. Prompt tokens thus run the 20 encoder layers
and layer 20’s key and value projection in full (from the vLLM code). The report gives the saving as
“from O(NL) to O(NL/2 + n_win × L/2)” (N prompt length, nwin window) and as “16B parameters
per token during decode but only 8B parameters during prefill”. It calls the replay’s quality cost
“negligible”, simulates the same truncation during post-training, and prints no numbers, nor a comparison for its
“performance comparable to the baseline”.
computed
skipped
approximate replay (last window)
Plain stackYOCODeepSeek-V4.1 CED
Schematic: 24 prompt positions by 8 layers, layer 0 at the bottom; not any model's sizes.
Figure 4.Which positions and layers prefill computes. YOCO's upper layers read one set of keys and values made at the midpoint, so prompt tokens stop there and the result is exact. CED's upper layers also keep their own windows, so the last window is replayed through them; DeepSeek replays only 128 tokens with truncated windows, an approximation. Schematic, not to scale.
Inside that stack, V4.1’s Compressed Sparse Attention 2 (CSA2) also shares the sparse selection.
Each layer has one of three fixed modes: Full “generates global KV and performs indexing”, Reindex
reuses a lower layer’s keys, values and indexer keys and picks fresh top-k indices with its own
indexer query, and Reuse takes keys, values and indices from below; “each layer retains its own global
Q and SWA KV”. The report assigns 4 Full, 4 Reindex and 30 Reuse layers after two window-only ones,
so only the Full layers store global entries: three encoder layers at one entry per two tokens and
layer 20 at one per token, 1,600 numbers per token with the indexer keys. At 128K that is 0.40 GiB in
BF16, against 4.54 GiB if every compressed layer kept its own (computed); at FP4, half a byte each, it
is 800 bytes, and with the formats’ scales (one byte per 16 key-value numbers, per 32 indexer numbers) 890, the report’s “890 bytes per token” (computed). The
assignment is set by hand, and the report prints no ablation of it.
window only; left stripe: each layer's own window
writes global keys/values (Full)
reads them, own indexer query (Reindex)
reads keys/values and indices (Reuse)
stores indexer keys
own query over another layer's indexer keys
Hover or tap a layer for its mode and where its keys, values and indices come from.
Figure 5.Cross-layer sharing in DeepSeek-V4.1-Flash: four layers write global keys and values and indexer keys, four more compute their own indexer queries over them, and every layer keeps its own 128-token window. In prefill, prompt tokens stop after layer 20's key and value projection. Mistral Large 4 (right) keeps one source, layer 35, for nine upper layers. From the configs; which source each reusing layer reads is inferred from field names.
Quality losses from sharing
Mistral Large 4’s report is the corpus’s clearest account of sharing that went wrong. Its first
design stacked one global layer in four, CLA between pairs of global layers, and YOCO for the second
half, which “reduced the KV-cache size by a factor of 16 compared to a full-attention baseline”
(4 × 2 × 2 by the rule above, our reading), with YOCO’s “two-stage prefilling … 1.7× faster at 1M
context”. Early ablations put the cost at “negligible or small enough”. After fine-tuning, SWE-bench scores fell with the windows
(see “Quality cost, and what reduces it”) and “regressed further when applying CLA”. Mistral concluded that “models that
perform equally well on language-modeling loss can nevertheless perform very differently on
downstream agentic tasks”, and saw no such gap on knowledge benchmarks.
Mistral then changed an existing checkpoint: the window layers became full-context layers, and the
CLA consumers got key and value weights copied from the last layer that had them, which changes what
they compute and had to be trained back. YOCO stayed: at a ten times larger test scale “keeping YOCO
yielded better performance”. In the shipped model all 72 layers are global sparse-attention layers, 44 of them with their own indexer, and
nine of them (39, 43, …, 71) still read layer 35’s cache, close to CSA2’s Reindex mode. The former
window layers above 35 compute their own keys and values, so prompt tokens cannot stop at the
midpoint: the YOCO sharing that remains reduces memory but not prefill compute. The report’s figures print no values.
Sharing sparse indices costs little memory and saves decode reads. IndexCache (Bai et al., 2026;
Z.ai ships it as IndexShare) lets most sparse layers reuse the top-k indices of the nearest layer
with its own indexer (T3); GLM-5.2, GLM-5.3 and Hy4-preview run 21 indexers in 78 layers. With
latents of 576 and indexer keys of 128 numbers, the cache shrinks at most 1+128/576=1.22 times:
1.15 for GLM-5.2, and 1.08 for Mistral Large 4’s 44 indexers in 72 layers (1.21 with its nine
KV-reusing layers). But the indexer is the only part of a sparse layer that scans every past token
at each step, so GLM-5.2 at 128K reads about 3.2 times fewer attention bytes per step with 21 indexers
than with 78 (computed with indexer keys and latents at one width; 2.8 times with FP8 indexer keys beside BF16 latents). The paper picks the layers that keep an indexer by greedy search on
language-modelling loss, or trains each kept indexer against the averaged attention of the layers it
serves. In a recurrent hybrid the global layers are four apart, and in a 35B-A3B test Qwen found that
sharing one index between two of them “remains below the baseline” on RULER up to 1M.
One model, many caches
A hybrid model holds several kinds of cache at once, each with its own size and rule for when
entries are dropped: growing keys and values, window ring buffers, fixed states, compressed entries
waiting for m tokens, and indexer keys. Most serving engines use PagedAttention, which stores
keys and values in fixed-size blocks of tokens, allocated as a sequence grows. DeepSeek’s V4 report
says its mix “violates fundamental assumptions behind PagedAttention”, and treats each layer’s window, with the tokens waiting for compression, “as
a state-space model”: a fixed-size state per sequence from a preallocated pool.
The clearest case is prefix caching, which keeps a prompt’s cache so that a later request with the
same beginning can skip its prefill. For global layers it is free: keep the entries up to the shared
prefix. A window layer keeps only its last W entries, so a prefix can be resumed only where the
engine kept every window layer’s entries at that boundary. A recurrent state exists only at the
position it was last updated, so the engine must save copies at chosen positions, and they are
large: NVIDIA’s Nemotron 3.5 report says that in vLLM “one Mamba cache page occupies at least as many
bytes as approximately 4K tokens of Attention KV cache”. A prefix is reused only up to the last whole
block it matches, so blocks that large miss most shared prefixes. Kimi K3’s report found that a page
size shared with the state forced blocks of 1,024 to 6,144 tokens, at which “caching is nearly
useless”. Moonshot hashed prefixes in 512-token blocks instead and saved KDA checkpoints at some of
those block ends, mostly at conversation-turn boundaries. Nemotron 3 Ultra handles rejected
drafts the same way, “snapshotting the SSM state at every draft step”. DeepSeek-V4’s window entries,
uncompressed and in every layer, made up “nearly half” of its persistent prefix cache, kept on SSD or
in host memory (V4.1 report); V4.1 stores none and replays the last window approximately.
Parallelism adds constraints. Context parallelism splits a long prompt across GPUs, which is easier
for cheap layers: window layers “exchange only the KV their queries can reach” (MiMo-V2.6) and
linear layers pass on “a fixed-size recurrent state” (Kimi K3). Pipeline parallelism puts groups of
layers on different GPUs, so layers that share a cache must sit in one group or pass it along; V4.1’s
prefill shortcut runs only if the group holding layer 20 holds every layer above it, and only with a
drafter that reads no hidden state outside the last window.
Comparison
Lever
What it caches
Saves
Costs
In the map
Window and global layers
W per window layer, T per global layer
growing cache to 1/(r+1) past T∗
no read past W by content; agentic regression at 3:1 (Mistral)
12 models, 8 labs; 1:1 to 6:1
Recurrent and global layers
fixed state, T per global layer
the same
at most dk exact associations per head; large state below T∗; harder prefix caching and rollback
8 models, 4 labs; about 3:1 to 5:1
Placement
nothing extra
nothing
wrong layers lose retrieval (GLM-5 retrofit)
rules differ by lab
Branches in one layer
window plus compressed entries
local detail without a global layer
a window cache in every layer
DeepSeek-V4, V4.1
KV sharing (CLA, YOCO, CED, CSA2)
one source for several layers
memory; with YOCO or CED, prefill
past tokens seen at one depth; same decode reads; CLA regression (Mistral); CED replay approximate
DeepSeek-V4.1, Mistral Large 4
Index sharing (IndexCache)
indexer keys in fewer layers
decode reads, indexer compute
below baseline across distant global layers (Qwen test)
GLM-5.2, GLM-5.3, Hy4, Mistral Large 4
Where the models in the map sit
The table lists the 35 language models whose code was checked, with caches computed from each
config.json for one sequence at 128K tokens, every kind in BF16, and before tensor parallelism
(which copies layers with few KV heads, and MLA latents, onto every GPU). “All global” gives every
layer the model’s global layer shape. MLA layers count at their latent width, as serving engines
store them (Kimi K3’s reference code caches expanded keys and values, 53 times more). DeepSeek-V4
and V4.1 are not counted as window interleaves, since every layer has a window branch. Reading the
code changed six of the site’s stored code-check answers: five DeepSeek variants marked as
interleaves or unclear, and one Nemotron model marked unclear.
Model
Cheap layers
Cheap : global
Window
Cache at 128K
All global
Cut
gpt-oss-120b
window
18 : 18
128
4.50 GiB
9.00 GiB
2.00
Laguna-S-2.1
window
36 : 12
512
6.07
24.00
3.95
Step-3.7-Flash
window
33 : 12
512
6.06
22.50
3.71
Step-5-Preview
window (global layers sparse)
69 : 23
512
8.73
34.50
3.95
LFM3-1.7B
conv + window
18 : 6
256
1.50
6.00
3.99
Kolibri-1
window
40 : 10
513
2.54
12.50
4.92
MiMo-V2.5, V2.6-Flash
window
39 : 9
128
2.84
15.00
5.29
Gemma 4 31B
window
50 : 10
1,024
10.78
60.00
5.57
Inkling
window
55 : 11
512
5.72
33.00
5.77
MiMo-V2.5-Pro, V2.6-Pro
window
60 : 10
128
6.29
43.75
6.96
Qwen3.5-397B-A17B
GDN
45 : 15
3.84
15.00
3.90
Qwen3.6-35B-A3B
GDN
30 : 10
2.53
10.00
3.95
Qwen3.8-Flash
GDN (global layers sparse)
36 : 12
3.15
12.38
3.93
Kimi K3
KDA
69 : 24
3.60
13.08
3.64
GLM-5.3-Flash
KDA
34 : 11
1.53
5.98
3.90
Nemotron-3-Ultra
Mamba-2
48 : 12
1.69
7.50
4.43
Nemotron-3.5-Lightning
Mamba-2
23 : 6
0.77
3.62
4.69
Nemotron-3.5-Super-VL
Mamba-2
40 : 8
1.08
6.00
5.55
Compression and sharing, against a dense layer with one 512-wide key that is also the value:
DeepSeek-V4-Flash (V4-Flash, -0731, -Vision-Exp)
window branch in every layer
CSA 21, HCA 20, 2 window-only
128
0.85
5.38
6.36
DeepSeek-V4-Pro (V4-Pro, -0813)
window branch in every layer
CSA 30, HCA 31
128
1.21
7.62
6.30
DeepSeek-V4.1-Flash (and -1017)
window branch, KV and index reuse
4 KV sources in 40
128
0.40
5.00
12.64
Sharing in all-global stacks:
Mistral Large 4
KV reuse in 9 layers, index reuse
72 global
10.23
12.38
1.21
GLM-5.2, GLM-5.3, Hy4-preview
index reuse
78 global
11.62
13.41
1.15
DeepSeek-V3.2, MiniMax-M3, M3.1-Flash, Hy3
none
all global
10.48 to 40.00
same (MiniMax-M3: 16.88)
1.00 (MiniMax-M3: 1.01)
context T =
window cheap layers
recurrent cheap layers
convolution + window (LFM3)
y = 1 / (1 + x)
Hover or tap a dot for its bound, cut and T*.
Figure 6.How close each hybrid gets to its bound r+1, one sequence in BF16. Every dot lies on the curve 1/(1 + T*/T). At 128K the farthest from the bound is Nemotron-3-Ultra, whose large Mamba-2 states put T* at about 17,000 tokens, then Nemotron-3.5-Super-VL (Mamba-2 states) and Gemma 4 (wide window layers). Computed from the configs.
Easy to get wrong
A 5:1 ratio does not cut the cache fivefold. Against an all-global stack the cut approaches six,
and only at contexts well past T∗. MiMo-V2-Flash’s “nearly 6×” counts the layers of one block;
over the shipped 48-layer configs the computed cut is 5.29.
Only Kimi K3 and Gemma 4 state that the final layer is global. Six models in the map end on a cheap
layer.
Sharing keys and values across layers cuts the memory a sequence holds; each layer still reads the
shared cache at every decode step. Sharing indices does the reverse in sparse stacks: little memory,
about three times fewer attention bytes per step for GLM-5.2 at 128K.
Language-modelling loss can miss a hybrid’s cost. Mistral’s window and CLA changes looked
harmless in early pre-training ablations and lowered SWE-bench scores after fine-tuning.
The ablations behind these choices ran on test models at fixed ratios. No report of a model in the
map sweeps the ratio of its own architecture; the nearest are Gemma 3’s plots (minimal perplexity
change up to 7:1 on 2B and 9B models, perplexity only) and Jamba’s 3:1 against 7:1.
Check yourself
Laguna-S-2.1 has 48 layers, 12 of them global, eight KV heads of width 128 in both kinds, and a
window of 512. What is its T∗, how large is its cache for one sequence at 1M tokens in BF16, and
how close is the cut to its bound?
AnswerEach layer caches 2⋅8⋅128=2,048 numbers, 4 KiB,
per token, and r=3, so T∗=3⋅512=1,536. Global: 12⋅4KiB⋅1,048,576=48 GiB; window: 36⋅4KiB⋅512=72 MiB; total about 48.07 GiB.
All global: 192 GiB, so a cut of 4/(1+1,536/1,048,576)≈3.99.
GLM-5.2 runs an indexer in 21 of its 78 sparse layers. Why does that shrink its cache by only 1.15
times, while a decode step at 128K reads about three times fewer attention bytes?
AnswerEvery layer still stores its 576-wide latent; only the 128-wide
indexer keys go from 78 layers to 21, at most a 1+128/576=1.22 cut. At each step an indexer
reads its keys for all 131,072 past tokens, while attention reads only 2,048 selected latents, so 21
scans instead of 78 cut the reads about 3.2 times at one width (2.8 with FP8 indexer keys).
Why can prompt tokens skip the upper half in YOCO but not in DeepSeek-V4.1’s CED, and why is CED’s
shortcut approximate?
AnswerYOCO’s upper layers write no keys or values of their own, so
nothing needs their outputs at prompt positions except the last. CED’s upper layers keep their own
128-token windows, computed from their own hidden states, so the last window must run through them.
Exact window entries there would depend on about 2,500 earlier positions through the windows below;
DeepSeek runs only the last 128, with each window cut off at the start of that segment.