Open Models Almanac

Token mixer · T5

Hybrid layer stacking and cross-layer sharing

Most layers can be cheap if a few global layers handle long-range retrieval: the growing cache shrinks with the ratio of cheap to global layers, and sharing caches across layers shrinks it further, with quality losses one lab reported

Illustration of a long workshop seen from the side. Most of the benches in a row are small and hold only a few tools within arm's reach; every sixth bench is larger and has a long rail that reaches back to tall shelves of stored parts at the far left. A few thin threads run from one large bench to the next.

Gemma 4 31B has 60 layers. Holding a 128K-token prompt in BF16, its key-value (KV) cache, the keys and values every layer keeps for every past token, takes 10.78 GiB. If all 60 layers had the shape of its ten global layers and attended over the whole context, it would take 60 GiB (both computed from config.json below). The difference comes from one line of the config: 50 of the 60 layers are sliding_attention with sliding_window: 1024, so they keep only the last 1,024 tokens, and only the other ten keep everything.

None of the token mixers in T1 to T3 is both cheap and a full retriever. Softmax attention over the whole context can read any past token on demand, and its cache grows with every token. A sliding window, a recurrent state or a sparse selection bounds the cost and gives up some of that access. So most of the 35 open-weight models this site tracks (the map) stack layer types: many cheap layers and a few expensive global ones. A second, newer approach works across layers: one layer produces keys, values or sparse indices, and others reuse them instead of producing their own.

Background

Generating text has two phases. Prefill runs the whole prompt through the model at once and writes the KV cache; it is limited by arithmetic (FLOPs). Decode then generates one token at a time, and each step reads the cache of every layer; with one query per sequence that step is limited by the bytes it reads (T1). The cache also sets how many sequences fit on a GPU at once.

A global softmax layer with gg KV heads of width dhd_h caches 2gdh2 g d_h numbers per token, at bb bytes each, so 2gdhb T2 g d_h b\,T bytes at context length TT (T1). The cheap layer types from T2 and T3 cache less:

  • A sliding-window attention (SWA) layer attends to the last WW tokens only. Its cache is a ring buffer of WW entries that overwrites the oldest one, so it stops growing at T=WT = W. (Window sizes here are the configs’ sliding_window, which counts the current token: Kolibri’s 513 is its report’s “512 preceding tokens”.)
  • A linear-attention or state-space layer (Gated DeltaNet, Kimi Delta Attention, Mamba-2; T2) folds the prefix into a fixed-size state matrix, after a short causal convolution over time whose last few inputs it also keeps. Its cache does not grow with TT.
  • A compressed or sparse layer (T3) keeps one entry per mm tokens, or attends to a few entries that a small scoring network, the indexer, picks per query; its cache still grows with TT.

Write nGn_G for the number of global layers, nSn_S for the window layers, nRn_R for the recurrent ones, eGe_G and eSe_S for the numbers cached per token by one global and one window layer, and ss for the numbers in one recurrent state, convolution buffer included. Recurrent states and keys are often stored at different widths, so write bKVb_{KV} and bRb_R. The cache of a stack with L=nG+nS+nRL = n_G + n_S + n_R layers is

M(T)=bKV[ nG eG T+nS eSmin⁡(T,W) ]+bR nR s.M(T) = b_{KV}\big[\,n_G\, e_G\, T + n_S\, e_S \min(T, W)\,\big] + b_R\, n_R\, s.

Only the first term grows. Compare it with the same stack built from LL global layers of the global shape. With r=(nS+nR)/nGr = (n_S + n_R)/n_G cheap layers per global layer and T≥WT \ge W,

cut(T)=L eG TM(T)/bKV=r+11+T∗/T,T∗=nS eS W bKV+nR s bRnG eG bKV.\text{cut}(T) = \frac{L\, e_G\, T}{M(T)/b_{KV}} = \frac{r+1}{1 + T^*/T}, \qquad T^* = \frac{n_S\, e_S\, W\, b_{KV} + n_R\, s\, b_R}{n_G\, e_G\, b_{KV}}.

T∗T^* is the context at which the fixed caches hold as much as the growing one. The cut tends to r+1r + 1 and is half of it at T=T∗T = T^*. A 5:1 stack approaches a sixth of the all-global cache, a 3:1 stack a quarter, and going from 1:1 to 5:1 shrinks the growing part threefold, from half the layers to a sixth. For a window stack T∗=rWeS/eGT^* = r W e_S/e_G, so raising the ratio raises the bound and pushes T∗T^* out in the same proportion.

Interleaving local and global attention

The oldest hybrid is local and global softmax attention in alternation. Longformer (2020) put a sliding window and a few global tokens inside one layer. Gemma 2 (2024) made the split by layer: “We alternate between a local sliding window attention … and global attention … in every other layer”, with a 4,096-token window and an 8,192-token context, so half the layers were global. Gemma 3 (2025) moved to five local layers per global layer and a window of 1,024, so that “only the global layers attend to long context”, because of “the memory explosion of the KV cache”. Gemma 4 kept 5:1 (4:1 in its smallest model).

Twelve of the map’s 35 models, from eight labs, interleave window and global layers, as their configs and code show. Ratios run from 1:1 (gpt-oss-120b, window 128: “Following GPT-3, attention blocks alternate between banded window and fully dense patterns”) to 6:1 (MiMo-V2.5-Pro, window 128). LFM3-1.7B (Liquid) mixes three kinds: 13 gated short-convolution layers, 5 window layers and 6 global ones. The table near the end lists every model.

  • global softmax
  • sliding window
  • linear / SSM recurrent
  • short convolution
  • CSA
  • HCA (top stripe: in-layer window)
  1. gpt-oss-120bgpt-oss-120b, layer 0: sliding window, W = 128gpt-oss-120b, layer 1: global softmaxgpt-oss-120b, layer 2: sliding window, W = 128gpt-oss-120b, layer 3: global softmaxgpt-oss-120b, layer 4: sliding window, W = 128gpt-oss-120b, layer 5: global softmaxgpt-oss-120b, layer 6: sliding window, W = 128gpt-oss-120b, layer 7: global softmaxgpt-oss-120b, layer 8: sliding window, W = 128gpt-oss-120b, layer 9: global softmaxgpt-oss-120b, layer 10: sliding window, W = 128gpt-oss-120b, layer 11: global softmaxgpt-oss-120b, layer 12: sliding window, W = 128gpt-oss-120b, layer 13: global softmaxgpt-oss-120b, layer 14: sliding window, W = 128gpt-oss-120b, layer 15: global softmaxgpt-oss-120b, layer 16: sliding window, W = 128gpt-oss-120b, layer 17: global softmaxgpt-oss-120b, layer 18: sliding window, W = 128gpt-oss-120b, layer 19: global softmaxgpt-oss-120b, layer 20: sliding window, W = 128gpt-oss-120b, layer 21: global softmaxgpt-oss-120b, layer 22: sliding window, W = 128gpt-oss-120b, layer 23: global softmaxgpt-oss-120b, layer 24: sliding window, W = 128gpt-oss-120b, layer 25: global softmaxgpt-oss-120b, layer 26: sliding window, W = 128gpt-oss-120b, layer 27: global softmaxgpt-oss-120b, layer 28: sliding window, W = 128gpt-oss-120b, layer 29: global softmaxgpt-oss-120b, layer 30: sliding window, W = 128gpt-oss-120b, layer 31: global softmaxgpt-oss-120b, layer 32: sliding window, W = 128gpt-oss-120b, layer 33: global softmaxgpt-oss-120b, layer 34: sliding window, W = 128gpt-oss-120b, layer 35: global softmax18:18 · W 128 36 layers
  2. Laguna-S-2.1Laguna-S-2.1, layer 0: global softmaxLaguna-S-2.1, layer 1: sliding window, W = 512Laguna-S-2.1, layer 2: sliding window, W = 512Laguna-S-2.1, layer 3: sliding window, W = 512Laguna-S-2.1, layer 4: global softmaxLaguna-S-2.1, layer 5: sliding window, W = 512Laguna-S-2.1, layer 6: sliding window, W = 512Laguna-S-2.1, layer 7: sliding window, W = 512Laguna-S-2.1, layer 8: global softmaxLaguna-S-2.1, layer 9: sliding window, W = 512Laguna-S-2.1, layer 10: sliding window, W = 512Laguna-S-2.1, layer 11: sliding window, W = 512Laguna-S-2.1, layer 12: global softmaxLaguna-S-2.1, layer 13: sliding window, W = 512Laguna-S-2.1, layer 14: sliding window, W = 512Laguna-S-2.1, layer 15: sliding window, W = 512Laguna-S-2.1, layer 16: global softmaxLaguna-S-2.1, layer 17: sliding window, W = 512Laguna-S-2.1, layer 18: sliding window, W = 512Laguna-S-2.1, layer 19: sliding window, W = 512Laguna-S-2.1, layer 20: global softmaxLaguna-S-2.1, layer 21: sliding window, W = 512Laguna-S-2.1, layer 22: sliding window, W = 512Laguna-S-2.1, layer 23: sliding window, W = 512Laguna-S-2.1, layer 24: global softmaxLaguna-S-2.1, layer 25: sliding window, W = 512Laguna-S-2.1, layer 26: sliding window, W = 512Laguna-S-2.1, layer 27: sliding window, W = 512Laguna-S-2.1, layer 28: global softmaxLaguna-S-2.1, layer 29: sliding window, W = 512Laguna-S-2.1, layer 30: sliding window, W = 512Laguna-S-2.1, layer 31: sliding window, W = 512Laguna-S-2.1, layer 32: global softmaxLaguna-S-2.1, layer 33: sliding window, W = 512Laguna-S-2.1, layer 34: sliding window, W = 512Laguna-S-2.1, layer 35: sliding window, W = 512Laguna-S-2.1, layer 36: global softmaxLaguna-S-2.1, layer 37: sliding window, W = 512Laguna-S-2.1, layer 38: sliding window, W = 512Laguna-S-2.1, layer 39: sliding window, W = 512Laguna-S-2.1, layer 40: global softmaxLaguna-S-2.1, layer 41: sliding window, W = 512Laguna-S-2.1, layer 42: sliding window, W = 512Laguna-S-2.1, layer 43: sliding window, W = 512Laguna-S-2.1, layer 44: global softmaxLaguna-S-2.1, layer 45: sliding window, W = 512Laguna-S-2.1, layer 46: sliding window, W = 512Laguna-S-2.1, layer 47: sliding window, W = 51236:12 · W 512 48 layers
  3. Kolibri-1Kolibri-1, layer 0: sliding window, W = 513Kolibri-1, layer 1: sliding window, W = 513Kolibri-1, layer 2: sliding window, W = 513Kolibri-1, layer 3: sliding window, W = 513Kolibri-1, layer 4: global softmaxKolibri-1, layer 5: sliding window, W = 513Kolibri-1, layer 6: sliding window, W = 513Kolibri-1, layer 7: sliding window, W = 513Kolibri-1, layer 8: sliding window, W = 513Kolibri-1, layer 9: global softmaxKolibri-1, layer 10: sliding window, W = 513Kolibri-1, layer 11: sliding window, W = 513Kolibri-1, layer 12: sliding window, W = 513Kolibri-1, layer 13: sliding window, W = 513Kolibri-1, layer 14: global softmaxKolibri-1, layer 15: sliding window, W = 513Kolibri-1, layer 16: sliding window, W = 513Kolibri-1, layer 17: sliding window, W = 513Kolibri-1, layer 18: sliding window, W = 513Kolibri-1, layer 19: global softmaxKolibri-1, layer 20: sliding window, W = 513Kolibri-1, layer 21: sliding window, W = 513Kolibri-1, layer 22: sliding window, W = 513Kolibri-1, layer 23: sliding window, W = 513Kolibri-1, layer 24: global softmaxKolibri-1, layer 25: sliding window, W = 513Kolibri-1, layer 26: sliding window, W = 513Kolibri-1, layer 27: sliding window, W = 513Kolibri-1, layer 28: sliding window, W = 513Kolibri-1, layer 29: global softmaxKolibri-1, layer 30: sliding window, W = 513Kolibri-1, layer 31: sliding window, W = 513Kolibri-1, layer 32: sliding window, W = 513Kolibri-1, layer 33: sliding window, W = 513Kolibri-1, layer 34: global softmaxKolibri-1, layer 35: sliding window, W = 513Kolibri-1, layer 36: sliding window, W = 513Kolibri-1, layer 37: sliding window, W = 513Kolibri-1, layer 38: sliding window, W = 513Kolibri-1, layer 39: global softmaxKolibri-1, layer 40: sliding window, W = 513Kolibri-1, layer 41: sliding window, W = 513Kolibri-1, layer 42: sliding window, W = 513Kolibri-1, layer 43: sliding window, W = 513Kolibri-1, layer 44: global softmaxKolibri-1, layer 45: sliding window, W = 513Kolibri-1, layer 46: sliding window, W = 513Kolibri-1, layer 47: sliding window, W = 513Kolibri-1, layer 48: sliding window, W = 513Kolibri-1, layer 49: global softmax40:10 · W 513 50 layers
  4. Gemma 4 31BGemma 4 31B, layer 0: sliding window, W = 1,024Gemma 4 31B, layer 1: sliding window, W = 1,024Gemma 4 31B, layer 2: sliding window, W = 1,024Gemma 4 31B, layer 3: sliding window, W = 1,024Gemma 4 31B, layer 4: sliding window, W = 1,024Gemma 4 31B, layer 5: global softmaxGemma 4 31B, layer 6: sliding window, W = 1,024Gemma 4 31B, layer 7: sliding window, W = 1,024Gemma 4 31B, layer 8: sliding window, W = 1,024Gemma 4 31B, layer 9: sliding window, W = 1,024Gemma 4 31B, layer 10: sliding window, W = 1,024Gemma 4 31B, layer 11: global softmaxGemma 4 31B, layer 12: sliding window, W = 1,024Gemma 4 31B, layer 13: sliding window, W = 1,024Gemma 4 31B, layer 14: sliding window, W = 1,024Gemma 4 31B, layer 15: sliding window, W = 1,024Gemma 4 31B, layer 16: sliding window, W = 1,024Gemma 4 31B, layer 17: global softmaxGemma 4 31B, layer 18: sliding window, W = 1,024Gemma 4 31B, layer 19: sliding window, W = 1,024Gemma 4 31B, layer 20: sliding window, W = 1,024Gemma 4 31B, layer 21: sliding window, W = 1,024Gemma 4 31B, layer 22: sliding window, W = 1,024Gemma 4 31B, layer 23: global softmaxGemma 4 31B, layer 24: sliding window, W = 1,024Gemma 4 31B, layer 25: sliding window, W = 1,024Gemma 4 31B, layer 26: sliding window, W = 1,024Gemma 4 31B, layer 27: sliding window, W = 1,024Gemma 4 31B, layer 28: sliding window, W = 1,024Gemma 4 31B, layer 29: global softmaxGemma 4 31B, layer 30: sliding window, W = 1,024Gemma 4 31B, layer 31: sliding window, W = 1,024Gemma 4 31B, layer 32: sliding window, W = 1,024Gemma 4 31B, layer 33: sliding window, W = 1,024Gemma 4 31B, layer 34: sliding window, W = 1,024Gemma 4 31B, layer 35: global softmaxGemma 4 31B, layer 36: sliding window, W = 1,024Gemma 4 31B, layer 37: sliding window, W = 1,024Gemma 4 31B, layer 38: sliding window, W = 1,024Gemma 4 31B, layer 39: sliding window, W = 1,024Gemma 4 31B, layer 40: sliding window, W = 1,024Gemma 4 31B, layer 41: global softmaxGemma 4 31B, layer 42: sliding window, W = 1,024Gemma 4 31B, layer 43: sliding window, W = 1,024Gemma 4 31B, layer 44: sliding window, W = 1,024Gemma 4 31B, layer 45: sliding window, W = 1,024Gemma 4 31B, layer 46: sliding window, W = 1,024Gemma 4 31B, layer 47: global softmaxGemma 4 31B, layer 48: sliding window, W = 1,024Gemma 4 31B, layer 49: sliding window, W = 1,024Gemma 4 31B, layer 50: sliding window, W = 1,024Gemma 4 31B, layer 51: sliding window, W = 1,024Gemma 4 31B, layer 52: sliding window, W = 1,024Gemma 4 31B, layer 53: global softmaxGemma 4 31B, layer 54: sliding window, W = 1,024Gemma 4 31B, layer 55: sliding window, W = 1,024Gemma 4 31B, layer 56: sliding window, W = 1,024Gemma 4 31B, layer 57: sliding window, W = 1,024Gemma 4 31B, layer 58: sliding window, W = 1,024Gemma 4 31B, layer 59: global softmax50:10 · W 1,024 60 layers
  5. MiMo-V2.5MiMo-V2.5, layer 0: global softmaxMiMo-V2.5, layer 1: sliding window, W = 128MiMo-V2.5, layer 2: sliding window, W = 128MiMo-V2.5, layer 3: sliding window, W = 128MiMo-V2.5, layer 4: sliding window, W = 128MiMo-V2.5, layer 5: global softmaxMiMo-V2.5, layer 6: sliding window, W = 128MiMo-V2.5, layer 7: sliding window, W = 128MiMo-V2.5, layer 8: sliding window, W = 128MiMo-V2.5, layer 9: sliding window, W = 128MiMo-V2.5, layer 10: sliding window, W = 128MiMo-V2.5, layer 11: global softmaxMiMo-V2.5, layer 12: sliding window, W = 128MiMo-V2.5, layer 13: sliding window, W = 128MiMo-V2.5, layer 14: sliding window, W = 128MiMo-V2.5, layer 15: sliding window, W = 128MiMo-V2.5, layer 16: sliding window, W = 128MiMo-V2.5, layer 17: global softmaxMiMo-V2.5, layer 18: sliding window, W = 128MiMo-V2.5, layer 19: sliding window, W = 128MiMo-V2.5, layer 20: sliding window, W = 128MiMo-V2.5, layer 21: sliding window, W = 128MiMo-V2.5, layer 22: sliding window, W = 128MiMo-V2.5, layer 23: global softmaxMiMo-V2.5, layer 24: sliding window, W = 128MiMo-V2.5, layer 25: sliding window, W = 128MiMo-V2.5, layer 26: sliding window, W = 128MiMo-V2.5, layer 27: sliding window, W = 128MiMo-V2.5, layer 28: sliding window, W = 128MiMo-V2.5, layer 29: global softmaxMiMo-V2.5, layer 30: sliding window, W = 128MiMo-V2.5, layer 31: sliding window, W = 128MiMo-V2.5, layer 32: sliding window, W = 128MiMo-V2.5, layer 33: sliding window, W = 128MiMo-V2.5, layer 34: sliding window, W = 128MiMo-V2.5, layer 35: global softmaxMiMo-V2.5, layer 36: sliding window, W = 128MiMo-V2.5, layer 37: sliding window, W = 128MiMo-V2.5, layer 38: sliding window, W = 128MiMo-V2.5, layer 39: sliding window, W = 128MiMo-V2.5, layer 40: sliding window, W = 128MiMo-V2.5, layer 41: global softmaxMiMo-V2.5, layer 42: sliding window, W = 128MiMo-V2.5, layer 43: sliding window, W = 128MiMo-V2.5, layer 44: sliding window, W = 128MiMo-V2.5, layer 45: sliding window, W = 128MiMo-V2.5, layer 46: sliding window, W = 128MiMo-V2.5, layer 47: global softmax39:9 · W 128 48 layers
  6. MiMo-V2.5-ProMiMo-V2.5-Pro, layer 0: global softmaxMiMo-V2.5-Pro, layer 1: sliding window, W = 128MiMo-V2.5-Pro, layer 2: sliding window, W = 128MiMo-V2.5-Pro, layer 3: sliding window, W = 128MiMo-V2.5-Pro, layer 4: sliding window, W = 128MiMo-V2.5-Pro, layer 5: sliding window, W = 128MiMo-V2.5-Pro, layer 6: sliding window, W = 128MiMo-V2.5-Pro, layer 7: global softmaxMiMo-V2.5-Pro, layer 8: sliding window, W = 128MiMo-V2.5-Pro, layer 9: sliding window, W = 128MiMo-V2.5-Pro, layer 10: sliding window, W = 128MiMo-V2.5-Pro, layer 11: sliding window, W = 128MiMo-V2.5-Pro, layer 12: sliding window, W = 128MiMo-V2.5-Pro, layer 13: sliding window, W = 128MiMo-V2.5-Pro, layer 14: sliding window, W = 128MiMo-V2.5-Pro, layer 15: global softmaxMiMo-V2.5-Pro, layer 16: sliding window, W = 128MiMo-V2.5-Pro, layer 17: sliding window, W = 128MiMo-V2.5-Pro, layer 18: sliding window, W = 128MiMo-V2.5-Pro, layer 19: sliding window, W = 128MiMo-V2.5-Pro, layer 20: sliding window, W = 128MiMo-V2.5-Pro, layer 21: sliding window, W = 128MiMo-V2.5-Pro, layer 22: sliding window, W = 128MiMo-V2.5-Pro, layer 23: global softmaxMiMo-V2.5-Pro, layer 24: sliding window, W = 128MiMo-V2.5-Pro, layer 25: sliding window, W = 128MiMo-V2.5-Pro, layer 26: sliding window, W = 128MiMo-V2.5-Pro, layer 27: sliding window, W = 128MiMo-V2.5-Pro, layer 28: sliding window, W = 128MiMo-V2.5-Pro, layer 29: sliding window, W = 128MiMo-V2.5-Pro, layer 30: sliding window, W = 128MiMo-V2.5-Pro, layer 31: global softmaxMiMo-V2.5-Pro, layer 32: sliding window, W = 128MiMo-V2.5-Pro, layer 33: sliding window, W = 128MiMo-V2.5-Pro, layer 34: sliding window, W = 128MiMo-V2.5-Pro, layer 35: sliding window, W = 128MiMo-V2.5-Pro, layer 36: sliding window, W = 128MiMo-V2.5-Pro, layer 37: sliding window, W = 128MiMo-V2.5-Pro, layer 38: sliding window, W = 128MiMo-V2.5-Pro, layer 39: global softmaxMiMo-V2.5-Pro, layer 40: sliding window, W = 128MiMo-V2.5-Pro, layer 41: sliding window, W = 128MiMo-V2.5-Pro, layer 42: sliding window, W = 128MiMo-V2.5-Pro, layer 43: sliding window, W = 128MiMo-V2.5-Pro, layer 44: sliding window, W = 128MiMo-V2.5-Pro, layer 45: sliding window, W = 128MiMo-V2.5-Pro, layer 46: sliding window, W = 128MiMo-V2.5-Pro, layer 47: global softmaxMiMo-V2.5-Pro, layer 48: sliding window, W = 128MiMo-V2.5-Pro, layer 49: sliding window, W = 128MiMo-V2.5-Pro, layer 50: sliding window, W = 128MiMo-V2.5-Pro, layer 51: sliding window, W = 128MiMo-V2.5-Pro, layer 52: sliding window, W = 128MiMo-V2.5-Pro, layer 53: sliding window, W = 128MiMo-V2.5-Pro, layer 54: sliding window, W = 128MiMo-V2.5-Pro, layer 55: global softmaxMiMo-V2.5-Pro, layer 56: sliding window, W = 128MiMo-V2.5-Pro, layer 57: sliding window, W = 128MiMo-V2.5-Pro, layer 58: sliding window, W = 128MiMo-V2.5-Pro, layer 59: sliding window, W = 128MiMo-V2.5-Pro, layer 60: sliding window, W = 128MiMo-V2.5-Pro, layer 61: sliding window, W = 128MiMo-V2.5-Pro, layer 62: global softmaxMiMo-V2.5-Pro, layer 63: sliding window, W = 128MiMo-V2.5-Pro, layer 64: sliding window, W = 128MiMo-V2.5-Pro, layer 65: sliding window, W = 128MiMo-V2.5-Pro, layer 66: sliding window, W = 128MiMo-V2.5-Pro, layer 67: sliding window, W = 128MiMo-V2.5-Pro, layer 68: sliding window, W = 128MiMo-V2.5-Pro, layer 69: global softmax60:10 · W 128 70 layers
  7. LFM3-1.7BLFM3-1.7B, layer 0: short convolutionLFM3-1.7B, layer 1: short convolutionLFM3-1.7B, layer 2: global softmaxLFM3-1.7B, layer 3: short convolutionLFM3-1.7B, layer 4: sliding window, W = 256LFM3-1.7B, layer 5: short convolutionLFM3-1.7B, layer 6: global softmaxLFM3-1.7B, layer 7: short convolutionLFM3-1.7B, layer 8: sliding window, W = 256LFM3-1.7B, layer 9: short convolutionLFM3-1.7B, layer 10: global softmaxLFM3-1.7B, layer 11: short convolutionLFM3-1.7B, layer 12: sliding window, W = 256LFM3-1.7B, layer 13: short convolutionLFM3-1.7B, layer 14: global softmaxLFM3-1.7B, layer 15: short convolutionLFM3-1.7B, layer 16: sliding window, W = 256LFM3-1.7B, layer 17: short convolutionLFM3-1.7B, layer 18: global softmaxLFM3-1.7B, layer 19: short convolutionLFM3-1.7B, layer 20: sliding window, W = 256LFM3-1.7B, layer 21: short convolutionLFM3-1.7B, layer 22: global softmaxLFM3-1.7B, layer 23: short convolution18:6 · W 256 24 layers
  8. Qwen3.6-35B-A3BQwen3.6-35B-A3B, layer 0: linear / SSM recurrentQwen3.6-35B-A3B, layer 1: linear / SSM recurrentQwen3.6-35B-A3B, layer 2: linear / SSM recurrentQwen3.6-35B-A3B, layer 3: global softmaxQwen3.6-35B-A3B, layer 4: linear / SSM recurrentQwen3.6-35B-A3B, layer 5: linear / SSM recurrentQwen3.6-35B-A3B, layer 6: linear / SSM recurrentQwen3.6-35B-A3B, layer 7: global softmaxQwen3.6-35B-A3B, layer 8: linear / SSM recurrentQwen3.6-35B-A3B, layer 9: linear / SSM recurrentQwen3.6-35B-A3B, layer 10: linear / SSM recurrentQwen3.6-35B-A3B, layer 11: global softmaxQwen3.6-35B-A3B, layer 12: linear / SSM recurrentQwen3.6-35B-A3B, layer 13: linear / SSM recurrentQwen3.6-35B-A3B, layer 14: linear / SSM recurrentQwen3.6-35B-A3B, layer 15: global softmaxQwen3.6-35B-A3B, layer 16: linear / SSM recurrentQwen3.6-35B-A3B, layer 17: linear / SSM recurrentQwen3.6-35B-A3B, layer 18: linear / SSM recurrentQwen3.6-35B-A3B, layer 19: global softmaxQwen3.6-35B-A3B, layer 20: linear / SSM recurrentQwen3.6-35B-A3B, layer 21: linear / SSM recurrentQwen3.6-35B-A3B, layer 22: linear / SSM recurrentQwen3.6-35B-A3B, layer 23: global softmaxQwen3.6-35B-A3B, layer 24: linear / SSM recurrentQwen3.6-35B-A3B, layer 25: linear / SSM recurrentQwen3.6-35B-A3B, layer 26: linear / SSM recurrentQwen3.6-35B-A3B, layer 27: global softmaxQwen3.6-35B-A3B, layer 28: linear / SSM recurrentQwen3.6-35B-A3B, layer 29: linear / SSM recurrentQwen3.6-35B-A3B, layer 30: linear / SSM recurrentQwen3.6-35B-A3B, layer 31: global softmaxQwen3.6-35B-A3B, layer 32: linear / SSM recurrentQwen3.6-35B-A3B, layer 33: linear / SSM recurrentQwen3.6-35B-A3B, layer 34: linear / SSM recurrentQwen3.6-35B-A3B, layer 35: global softmaxQwen3.6-35B-A3B, layer 36: linear / SSM recurrentQwen3.6-35B-A3B, layer 37: linear / SSM recurrentQwen3.6-35B-A3B, layer 38: linear / SSM recurrentQwen3.6-35B-A3B, layer 39: global softmax30:10 40 layers
  9. Kimi K3Kimi K3, layer 0: linear / SSM recurrentKimi K3, layer 1: linear / SSM recurrentKimi K3, layer 2: linear / SSM recurrentKimi K3, layer 3: global softmaxKimi K3, layer 4: linear / SSM recurrentKimi K3, layer 5: linear / SSM recurrentKimi K3, layer 6: linear / SSM recurrentKimi K3, layer 7: global softmaxKimi K3, layer 8: linear / SSM recurrentKimi K3, layer 9: linear / SSM recurrentKimi K3, layer 10: linear / SSM recurrentKimi K3, layer 11: global softmaxKimi K3, layer 12: linear / SSM recurrentKimi K3, layer 13: linear / SSM recurrentKimi K3, layer 14: linear / SSM recurrentKimi K3, layer 15: global softmaxKimi K3, layer 16: linear / SSM recurrentKimi K3, layer 17: linear / SSM recurrentKimi K3, layer 18: linear / SSM recurrentKimi K3, layer 19: global softmaxKimi K3, layer 20: linear / SSM recurrentKimi K3, layer 21: linear / SSM recurrentKimi K3, layer 22: linear / SSM recurrentKimi K3, layer 23: global softmaxKimi K3, layer 24: linear / SSM recurrentKimi K3, layer 25: linear / SSM recurrentKimi K3, layer 26: linear / SSM recurrentKimi K3, layer 27: global softmaxKimi K3, layer 28: linear / SSM recurrentKimi K3, layer 29: linear / SSM recurrentKimi K3, layer 30: linear / SSM recurrentKimi K3, layer 31: global softmaxKimi K3, layer 32: linear / SSM recurrentKimi K3, layer 33: linear / SSM recurrentKimi K3, layer 34: linear / SSM recurrentKimi K3, layer 35: global softmaxKimi K3, layer 36: linear / SSM recurrentKimi K3, layer 37: linear / SSM recurrentKimi K3, layer 38: linear / SSM recurrentKimi K3, layer 39: global softmaxKimi K3, layer 40: linear / SSM recurrentKimi K3, layer 41: linear / SSM recurrentKimi K3, layer 42: linear / SSM recurrentKimi K3, layer 43: global softmaxKimi K3, layer 44: linear / SSM recurrentKimi K3, layer 45: linear / SSM recurrentKimi K3, layer 46: linear / SSM recurrentKimi K3, layer 47: global softmaxKimi K3, layer 48: linear / SSM recurrentKimi K3, layer 49: linear / SSM recurrentKimi K3, layer 50: linear / SSM recurrentKimi K3, layer 51: global softmaxKimi K3, layer 52: linear / SSM recurrentKimi K3, layer 53: linear / SSM recurrentKimi K3, layer 54: linear / SSM recurrentKimi K3, layer 55: global softmaxKimi K3, layer 56: linear / SSM recurrentKimi K3, layer 57: linear / SSM recurrentKimi K3, layer 58: linear / SSM recurrentKimi K3, layer 59: global softmaxKimi K3, layer 60: linear / SSM recurrentKimi K3, layer 61: linear / SSM recurrentKimi K3, layer 62: linear / SSM recurrentKimi K3, layer 63: global softmaxKimi K3, layer 64: linear / SSM recurrentKimi K3, layer 65: linear / SSM recurrentKimi K3, layer 66: linear / SSM recurrentKimi K3, layer 67: global softmaxKimi K3, layer 68: linear / SSM recurrentKimi K3, layer 69: linear / SSM recurrentKimi K3, layer 70: linear / SSM recurrentKimi K3, layer 71: global softmaxKimi K3, layer 72: linear / SSM recurrentKimi K3, layer 73: linear / SSM recurrentKimi K3, layer 74: linear / SSM recurrentKimi K3, layer 75: global softmaxKimi K3, layer 76: linear / SSM recurrentKimi K3, layer 77: linear / SSM recurrentKimi K3, layer 78: linear / SSM recurrentKimi K3, layer 79: global softmaxKimi K3, layer 80: linear / SSM recurrentKimi K3, layer 81: linear / SSM recurrentKimi K3, layer 82: linear / SSM recurrentKimi K3, layer 83: global softmaxKimi K3, layer 84: linear / SSM recurrentKimi K3, layer 85: linear / SSM recurrentKimi K3, layer 86: linear / SSM recurrentKimi K3, layer 87: global softmaxKimi K3, layer 88: linear / SSM recurrentKimi K3, layer 89: linear / SSM recurrentKimi K3, layer 90: linear / SSM recurrentKimi K3, layer 91: global softmaxKimi K3, layer 92: global softmax69:24 93 layers
  10. GLM-5.3-FlashGLM-5.3-Flash, layer 0: linear / SSM recurrentGLM-5.3-Flash, layer 1: linear / SSM recurrentGLM-5.3-Flash, layer 2: linear / SSM recurrentGLM-5.3-Flash, layer 3: global softmaxGLM-5.3-Flash, layer 4: linear / SSM recurrentGLM-5.3-Flash, layer 5: linear / SSM recurrentGLM-5.3-Flash, layer 6: linear / SSM recurrentGLM-5.3-Flash, layer 7: global softmaxGLM-5.3-Flash, layer 8: linear / SSM recurrentGLM-5.3-Flash, layer 9: linear / SSM recurrentGLM-5.3-Flash, layer 10: linear / SSM recurrentGLM-5.3-Flash, layer 11: global softmaxGLM-5.3-Flash, layer 12: linear / SSM recurrentGLM-5.3-Flash, layer 13: linear / SSM recurrentGLM-5.3-Flash, layer 14: linear / SSM recurrentGLM-5.3-Flash, layer 15: global softmaxGLM-5.3-Flash, layer 16: linear / SSM recurrentGLM-5.3-Flash, layer 17: linear / SSM recurrentGLM-5.3-Flash, layer 18: linear / SSM recurrentGLM-5.3-Flash, layer 19: global softmaxGLM-5.3-Flash, layer 20: linear / SSM recurrentGLM-5.3-Flash, layer 21: linear / SSM recurrentGLM-5.3-Flash, layer 22: linear / SSM recurrentGLM-5.3-Flash, layer 23: global softmaxGLM-5.3-Flash, layer 24: linear / SSM recurrentGLM-5.3-Flash, layer 25: linear / SSM recurrentGLM-5.3-Flash, layer 26: linear / SSM recurrentGLM-5.3-Flash, layer 27: global softmaxGLM-5.3-Flash, layer 28: linear / SSM recurrentGLM-5.3-Flash, layer 29: linear / SSM recurrentGLM-5.3-Flash, layer 30: linear / SSM recurrentGLM-5.3-Flash, layer 31: global softmaxGLM-5.3-Flash, layer 32: linear / SSM recurrentGLM-5.3-Flash, layer 33: linear / SSM recurrentGLM-5.3-Flash, layer 34: linear / SSM recurrentGLM-5.3-Flash, layer 35: global softmaxGLM-5.3-Flash, layer 36: linear / SSM recurrentGLM-5.3-Flash, layer 37: linear / SSM recurrentGLM-5.3-Flash, layer 38: linear / SSM recurrentGLM-5.3-Flash, layer 39: global softmaxGLM-5.3-Flash, layer 40: linear / SSM recurrentGLM-5.3-Flash, layer 41: linear / SSM recurrentGLM-5.3-Flash, layer 42: linear / SSM recurrentGLM-5.3-Flash, layer 43: global softmaxGLM-5.3-Flash, layer 44: linear / SSM recurrent34:11 45 layers
  11. Nemotron-3.5-LightningNemotron-3.5-Lightning, layer 0: linear / SSM recurrentNemotron-3.5-Lightning, layer 1: linear / SSM recurrentNemotron-3.5-Lightning, layer 2: linear / SSM recurrentNemotron-3.5-Lightning, layer 3: global softmaxNemotron-3.5-Lightning, layer 4: linear / SSM recurrentNemotron-3.5-Lightning, layer 5: linear / SSM recurrentNemotron-3.5-Lightning, layer 6: linear / SSM recurrentNemotron-3.5-Lightning, layer 7: global softmaxNemotron-3.5-Lightning, layer 8: linear / SSM recurrentNemotron-3.5-Lightning, layer 9: linear / SSM recurrentNemotron-3.5-Lightning, layer 10: linear / SSM recurrentNemotron-3.5-Lightning, layer 11: global softmaxNemotron-3.5-Lightning, layer 12: linear / SSM recurrentNemotron-3.5-Lightning, layer 13: linear / SSM recurrentNemotron-3.5-Lightning, layer 14: linear / SSM recurrentNemotron-3.5-Lightning, layer 15: global softmaxNemotron-3.5-Lightning, layer 16: linear / SSM recurrentNemotron-3.5-Lightning, layer 17: linear / SSM recurrentNemotron-3.5-Lightning, layer 18: linear / SSM recurrentNemotron-3.5-Lightning, layer 19: global softmaxNemotron-3.5-Lightning, layer 20: linear / SSM recurrentNemotron-3.5-Lightning, layer 21: linear / SSM recurrentNemotron-3.5-Lightning, layer 22: linear / SSM recurrentNemotron-3.5-Lightning, layer 23: linear / SSM recurrentNemotron-3.5-Lightning, layer 24: global softmaxNemotron-3.5-Lightning, layer 25: linear / SSM recurrentNemotron-3.5-Lightning, layer 26: linear / SSM recurrentNemotron-3.5-Lightning, layer 27: linear / SSM recurrentNemotron-3.5-Lightning, layer 28: linear / SSM recurrent23:6 29 layers
  12. DeepSeek-V4-FlashDeepSeek-V4-Flash, layer 0: sliding window, W = 128DeepSeek-V4-Flash, layer 1: sliding window, W = 128DeepSeek-V4-Flash, layer 2: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 3: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 4: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 5: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 6: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 7: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 8: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 9: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 10: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 11: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 12: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 13: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 14: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 15: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 16: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 17: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 18: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 19: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 20: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 21: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 22: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 23: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 24: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 25: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 26: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 27: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 28: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 29: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 30: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 31: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 32: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 33: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 34: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 35: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 36: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 37: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 38: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 39: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 40: compressed sparse (CSA) + 128-token window branchDeepSeek-V4-Flash, layer 41: heavily compressed (HCA) + 128-token window branchDeepSeek-V4-Flash, layer 42: compressed sparse (CSA) + 128-token window branchno plain global: 2 window, 21 CSA, 20 HCA · W 128 43 layers

Hover or tap a cell for its layer index and kind. Triangles mark the first and last layer.

Figure 1.Layer kinds in twelve models of the map, layer 0 on the left. Ratios run from 1:1 (gpt-oss) to 6:1 (MiMo Pro); Laguna, LFM3, GLM-5.3-Flash and Nemotron end on a cheap layer; DeepSeek-V4-Flash has no plain global layer at all. Read from each model's config.json.

Cache size by ratio

For Gemma 4 31B, a global layer caches 2⋅4⋅512=4,0962 \cdot 4 \cdot 512 = 4{,}096 numbers per token (num_global_key_value_heads: 4, global_head_dim: 512), 8 KiB at b=2b = 2, so ten of them hold 10⋅8 KiB⋅131,072=1010 \cdot 8\,\text{KiB} \cdot 131{,}072 = 10 GiB at 128K. A window layer has four times the KV heads at half the width (num_key_value_heads: 16, head_dim: 256), so it caches 8,192 numbers per token, twice what a global layer does, but only for 1,024 tokens: 16 MiB per layer, 800 MiB for all fifty. The total is 10.78 GiB against 60 GiB for 60 global-shape layers, a cut of 5.57 (computed).

Two choices hide in that 60 GiB. If the window layers kept their own, wider shape and lost their window, the all-global model would hold 110 GiB, a cut of 10.2. And Gemma 4’s global layers compute values from the key projection (attention_k_eq_v: true), but the reference code normalizes and rotates the keys and only normalizes the values, so both are cached, as we count them. Google’s report says this reuse, with rotation on a quarter of the dimensions, “effectively reduc[es] the global KV cache by 37.5%”; our reading is an engine that stores the values plus the rotated quarter of the keys, which would bring the 128K cache to 7.03 GiB (computed).

The cut stays below six because the window caches are a fixed cost: T∗=5⋅1,024⋅2=10,240T^* = 5 \cdot 1{,}024 \cdot 2 = 10{,}240, so at 8K the cut is only 2.7, it reaches 5 at 5T∗=51,2005 T^* = 51{,}200 tokens, and below 2,048 tokens the hybrid holds more than 60 global-shape layers. Kolibri-1 has 50 layers, 10 global, and four KV heads of width 128 in both kinds, so T∗=4⋅513=2,052T^* = 4 \cdot 513 = 2{,}052 and its cut at 128K is 4.92 against a bound of 5.

MiMo-V2-Flash’s report states “nearly a 6× reduction in KV-cache storage and attention computation for long contexts”, from a 5:1 ratio. That counts one block of six layers. The MiMo-V2.5 and V2.6-Flash configs have 48 layers, eight blocks of six whose very first layer is also made global (nine global layers: 0, 5, 11, …, 47), so a bound of 48/9=5.3348/9 = 5.33 and, with global layers caching 1,280 numbers per token and window layers 2,560, a cut of 5.29 at 128K (computed). The V2.5-Pro card’s “nearly 7x” holds: 10 global layers of 70 give 6.96.

The cache cut is a capacity figure. A decode step also reads the weights, which dominate at one sequence per GPU, so the cut shows up as more sequences per GPU, and as fewer bytes per step once many sequences share the weights. Kolibri’s report measures the long end: at 256K, “Most of this long-context traffic comes from the 10 full-attention layers”. In prefill, attention FLOPs fall about as much as the cache (5.77 times for Gemma 4 at 128K, computed), but projections and feed-forward blocks do not, so the whole prefill falls by less.

At T = 128K (131,072 tokens): 10.78 GiB, all-global 60.00 GiB, cut 5.57 against a bound of L/n_G = 60/10 = 6.00. Fixed caches equal the growing one at T* = 10,240 tokens. Cheap-layer share of the cache: 7.2%.

  • global layers' KV
  • window ring buffers
  • recurrent states
  • all layers global
  • T*: fixed caches = growing one
  • y axis rescales to the marked T
0153045601K4K8K32K64K128K256K512K1MT*GiB per sequencecontext length T (log scale)
Figure 2.KV cache against context length for one sequence. A hybrid's growing part is 1/(r+1) of an all-global stack's, but window buffers and recurrent states are a fixed cost that dominates below T*: for Gemma 4 31B, T* is 10,240 tokens. The toggles show deployed widths: with FP8 keys and an FP32 state, Nemotron-3-Ultra's T* moves from 16,960 to 67,840 tokens. Computed from each model's config.

Quality cost, and what reduces it

A window layer cannot read a position more than W−1W - 1 back. Information from further back can still arrive, as in a stack of convolutions: through kk window layers it can travel up to k(W−1)k(W-1) positions, but only if each intermediate position copied it forward before any query asked for it. A global layer is the only place where a query fetches a far token by content. The Qwen3.8-Next report puts it as “information outside the window can only propagate indirectly through depth”. T3’s error formula applies unchanged, since a window is a selection fixed in advance: if a global layer put attention mass ε\varepsilon beyond the window, restricting it to the window changes its output by ε(oˉout−oS)\varepsilon(\bar o_{\text{out}} - o^S), the gap between the far and the near values weighted by that mass.

Labs pair aggressive ratios with a few changes to the layers. A learnable attention sink is a per-head logit in the softmax denominator that matches no token, so a head can put little or no weight on any token. Without it the weights must sum to one, so in a short window with nothing relevant a head has to spread its weight over irrelevant tokens; this is our reading of why it matters more for windows. gpt-oss and DeepSeek-V4 put a sink in every layer, the MiMo configs in the window layers only (add_swa_attention_sink_bias: true). An output gate scales each head’s output by a learned function of the input (a sigmoid in LFM3 and Qwen, a softplus in Laguna), so a head can switch itself off. And the two kinds often get different position codes. Rotary position embedding (RoPE) rotates pairs of dimensions by an angle that grows with position; partial RoPE rotates only some, and no positional encoding (NoPE) none. Kolibri’s global layers use NoPE, and Gemma 4’s rotate a quarter of their dimensions (T4).

Three labs printed ablations of these choices, at a fixed ratio and on test models, not on the shipped ones. Their scores are single runs.

  • On a 32B dense test model, Xiaomi found that a window-128 hybrid without a sink scored lower than the all-global model (MMLU 54.9 against 57.3), and with one matched or beat it. After continued training at 32K, it scored 89.4 on RULER (synthetic long-context retrieval, out of 100), as the baseline did, and a window-512 hybrid 84.7. The report’s explanation, from its own settings, is “a clearer division of labor”: a tight window leaves long-range work to the global layers.
  • Aleph Alpha swept Kolibri’s window from 128 to 4,096 on a 7.8B proxy, trading window length against KV heads and expert shapes to keep FLOPs, KV bytes and parameters about equal. Scores were “generally similar”; on a larger proxy, 512 beat 128 by 6.7 points on long-context tasks at 256K.
  • Poolside added changes one at a time to a 16.6B Laguna test model at 3:1. Windows of 1,024 lowered the short-context average from 0.5389 (all-global) to 0.5292, against a noise level of about 0.005; gating, partial RoPE on the global layers and a 512 window brought it to 0.5449, and reallocated query heads to 0.5455.

One lab reports the opposite. Mistral Large 4 was first trained with one global layer in four and windows of 512, like Laguna. After midtraining (curated, often longer data between pre-training and fine-tuning) and supervised fine-tuning (SFT), its checkpoints scored lower than expected on long-context agentic tasks, and ablation runs at the same token budget isolated the cause: SWE-bench Verified scores, where a model edits a real repository over many steps and so reads a long, growing context, “decreased as we moved from full attention to SWA”. Mistral then removed the windows.

The two kinds also get different shapes: Gemma 4’s window layers have four times the KV heads of its global ones and MiMo-V2.5’s twice, which costs little memory with the cache capped at WW.

Recurrent layers instead of windows

A window layer has no access to tokens older than WW. A recurrent layer folds the whole prefix into its state. Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) update a state matrix per head, St=αt(I−βtktkt⊤)St−1+βtktvt⊤S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top, with key and value widths dkd_k and dvd_v (128 each in Qwen3.6): the β\beta term overwrites what is stored under the key ktk_t, the delta rule of Widrow and Hoff, and the decay αt\alpha_t lets old content fade (one scalar per head in GDN, one value per key channel in KDA). Mamba-2 updates its state through a similar input-dependent decay (T2). The Qwen3.8-Next report gives the split of work: “GDN compresses the prefix into a fixed-size recurrent state … while the interleaved global-attention layers retain direct token-level retrieval that is difficult for any finite-state recurrent memory to reproduce exactly.”

There is a precise version of that limit. Reading the state is linear, o=S⊤qo = S^\top q, so a head can return nn stored values exactly for arbitrary values only if the nn queries that read them are linearly independent, which needs n≤dkn \le d_k: at most 128 associations per head in Qwen3.6, against 131,072 positions at 128K. Past that count, recall degrades through interference.

Eight models from four labs mix recurrent and softmax layers. Qwen3.5 and 3.6 repeat “3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE)” (Qwen3.5 card); Qwen3.8 keeps the pattern with sparse global layers. Kimi K3 (Moonshot) runs blocks of three KDA layers and one gated multi-head latent attention (MLA) layer, plus “an additional Gated MLA layer … at the end of the backbone”: 69 KDA and 24 MLA layers in all. MLA caches one latent vector of width dcd_c per token instead of per-head keys and values (T1). GLM-5.3-Flash (Z.ai) has 34 KDA and 11 sparse MLA layers, a ratio read from its config. Nemotron 3 Ultra and 3.5 (NVIDIA) put a few attention layers, dense, with two KV heads and no position code, among Mamba-2 layers at irregular spacing, 3.8:1 to 5:1 counting token-mixing layers only. NVIDIA calls them “sparse global Attention anchors”, sparse in the sense of few. Before these, Jamba (2024) found “virtually no performance difference” between 3:1 and 7:1 on a 1.3B model.

In Qwen3.6-35B-A3B a global layer caches 2⋅2⋅256=1,0242 \cdot 2 \cdot 256 = 1{,}024 numbers per token. A GDN layer keeps 32 value heads of 128×128128 \times 128 plus a convolution buffer of 8,192 channels by 4: 557,056 numbers, 1.06 MiB in BF16. So T∗=3⋅557,056/1,024=1,632T^* = 3 \cdot 557{,}056 / 1{,}024 = 1{,}632, and at 128K the model holds 2.53 GiB against 10 GiB all-global, a cut of 3.95 (computed); at the configured FP32 state, T∗T^* is 3,264. In Qwen’s ablation, on a 25B-A3B test model with one global layer in four, the GDN hybrid averaged 53.81 over nine short-context benchmarks, a window-128 hybrid 51.15 and the all-global model 49.87; the results “do not by themselves isolate which architectural component causes each improvement”.

Recurrent layers have other costs. The state is large even for a short prompt: Nemotron 3 Ultra’s “32-bit Mamba cache is larger than the FP8 KV cache at sequence lengths up to 64K”, which is T∗T^* with bR=4b_R = 4 and bKV=1b_{KV} = 1, 67,840 tokens (computed), and NVIDIA moved to 16-bit states. Speculative decoding, where a small drafter proposes a few tokens and the model checks them in one pass, must undo rejected tokens: a KV cache drops its last entries, but a state that absorbed them cannot be un-updated, so engines keep copies (below).

Where the global layers go

Two labs state a rule for the end of the stack. Kimi K3 adds its extra MLA layer so that “the final layer always performs global attention”, and Gemma 4’s card says the interleave is arranged “ensuring the final layer is always global”. Neither gives a reason, and six models in the map end on a cheap layer: Laguna-S-2.1 on a window, LFM3 on a convolution, GLM-5.3-Flash on KDA, and all three Nemotron models on Mamba-2. Longformer found wider windows near the top better than near the bottom on a small character model, the same direction.

At the start, MiMo puts a global layer first “to stabilize early representation learning”, as do Laguna and Step-3.7; Gemma 4, Kolibri and gpt-oss start with a window layer, and DeepSeek-V4-Flash and V4.1-Flash with two window-only layers.

Only one lab report measures where global layers go, and it argues against hybrids. In the GLM-5 report, Z.ai converts a 40-layer GLM-9B model, trained with full attention to 128K, so that half its layers become windows of 4,096 tokens. Converting every other layer, without retraining, drops RULER at 128K from 75.28 to 6.51. A beam search over which layers to convert, scoring each candidate on RULER at 16K, keeps 53.95. After 190B tokens of further training at 64K, the regular interleave recovers to 44.93, a drop of 30.35 points, and the searched pattern to 69.59, a drop of 5.69. No gradient passes through which layers are windows; the search chose the pattern, scored on the benchmark the table reports, and the error formula above suggests it found layers that put little attention mass beyond 4,096 tokens. The report concludes that even the best method keeps “an inherent accuracy gap on fine-grained retrieval tasks”, and GLM-5 uses sparse attention in every layer instead. This is a retrofit of a model trained as all-global, and Z.ai later shipped GLM-5.3-Flash as a hybrid.

Figure 3.GLM-5's retrofit study on a 9B model, converting half the layers to 4,096-token windows. A regular interleave collapses past the window without retraining (6.51 on RULER at 128K) and stays 30 points down after 190B tokens; layers picked by a search, which was scored on RULER at 16K, keep most of the score. Scores from the GLM-5 report; a retrofit, not a hybrid trained from scratch.

Two mixers in one layer

A hybrid can also combine branches inside a layer. DeepSeek-V4 compresses its cache in two ways (T3): Compressed Sparse Attention (CSA) merges every four tokens into one entry and lets an indexer pick 512 (Flash) or 1,024 (Pro) of them per query, and Heavily Compressed Attention (HCA) merges every 128 tokens. V4 alternates the two, a hybrid of two global mechanisms. Neither can see the query’s own, still uncompressed block, and “recent tokens usually possess greater relevance”, so each layer also keeps keys and values for the last 128 tokens, and the code puts window and compressed entries through one softmax with a sink. One softmax over two sets is a mixture weighted by each set’s share of the normalizer; Native Sparse Attention (T3) learns that gate from the query instead. Since no V4 layer is global without a window, we do not count V4 among the window interleaves.

Sharing across layers

Cross-layer sharing removes some layers’ caches instead of shrinking them. Cross-Layer Attention (CLA, Brandon et al., 2024) computes key and value projections in only some layers; the others attend with their own queries to a lower layer’s keys and values. It ties activations, not weights: layer ℓ\ell computes softmax⁡(qℓKj⊤) Vj\operatorname{softmax}(q_\ell K_j^\top)\,V_j over the keys and values of layer j<ℓj < \ell, so it keeps its own choice of which past tokens to weight, but sees every past token as it was at depth jj. It is MQA’s sharing across heads (T1) applied across layers. Sharing over pairs (CLA2) halves the cache: on 1B and 3B models trained on up to 100B tokens it cost “at worst a very modest (less than 1% change) degradation in perplexity”. The paper also notes that “even shared KV cache layers must be separately re-read from main memory in each attention layer”, so CLA cuts the memory a sequence holds, not the bytes a decode step reads.

You Only Cache Once (YOCO, Sun et al., 2024) shares more. The bottom half of the network uses cheap layers, and the hidden state at layer L/2L/2 is projected once into keys and values that every upper layer attends to. The upper layers write no keys or values of their own, and their outputs at prompt positions are discarded except at the last one, so prompt tokens can stop at the midpoint: “at least half prefilling latency reduction”, with the same result.

DeepSeek-V4.1-Flash’s Causal Encoder-Decoder (CED) builds on YOCO. Its 40 layers, numbered from 0, are a 20-layer encoder and a 20-layer decoder whose global keys and values are “projected directly from the hidden state of the (L/2)-th layer”. Like a classical decoder, CED’s equations give each decoder layer its own projection, but in the shipped configuration only layer 20 makes one and the other 19 read it: one global cache, as in YOCO. Every layer also keeps its own 128-token window, computed from its own hidden states, and that keeps prompt tokens from skipping the decoder. Exact window entries for the last 128 positions would depend, through the windows below, on the last 128+19⋅127=2,541128 + 19 \cdot 127 = 2{,}541; DeepSeek replays only 128, truncating each window to them, which “approximately reconstructs the required SWA KV states”. Prompt tokens thus run the 20 encoder layers and layer 20’s key and value projection in full (from the vLLM code). The report gives the saving as “from O(NL) to O(NL/2 + n_win × L/2)” (NN prompt length, nwinn_{win} window) and as “16B parameters per token during decode but only 8B parameters during prefill”. It calls the replay’s quality cost “negligible”, simulates the same truncation during post-training, and prints no numbers, nor a comparison for its “performance comparable to the baseline”.

  • computed
  • skipped
  • approximate replay (last window)
Plain stack
0L−1midprompt position →last window
YOCO
0L−1midprompt position →last windowone shared set ofkeys and values
DeepSeek-V4.1 CED
0L−1midprompt position →last windowkeys and values projectedfor every tokenreplayed, approximate

Schematic: 24 prompt positions by 8 layers, layer 0 at the bottom; not any model's sizes.

Figure 4.Which positions and layers prefill computes. YOCO's upper layers read one set of keys and values made at the midpoint, so prompt tokens stop there and the result is exact. CED's upper layers also keep their own windows, so the last window is replayed through them; DeepSeek replays only 128 tokens with truncated windows, an approximation. Schematic, not to scale.

Inside that stack, V4.1’s Compressed Sparse Attention 2 (CSA2) also shares the sparse selection. Each layer has one of three fixed modes: Full “generates global KV and performs indexing”, Reindex reuses a lower layer’s keys, values and indexer keys and picks fresh top-k indices with its own indexer query, and Reuse takes keys, values and indices from below; “each layer retains its own global Q and SWA KV”. The report assigns 4 Full, 4 Reindex and 30 Reuse layers after two window-only ones, so only the Full layers store global entries: three encoder layers at one entry per two tokens and layer 20 at one per token, 1,600 numbers per token with the indexer keys. At 128K that is 0.40 GiB in BF16, against 4.54 GiB if every compressed layer kept its own (computed); at FP4, half a byte each, it is 800 bytes, and with the formats’ scales (one byte per 16 key-value numbers, per 32 indexer numbers) 890, the report’s “890 bytes per token” (computed). The assignment is set by hand, and the report prints no ablation of it.

  • window only; left stripe: each layer's own window
  • writes global keys/values (Full)
  • reads them, own indexer query (Reindex)
  • reads keys/values and indices (Reuse)
  • stores indexer keys
  • own query over another layer's indexer keys
KV DeepSeek-V4.1-Flash indices layer 0: window only (own 128-token window), no compressed keys/values, no indexer0layer 1: window only (own 128-token window), no compressed keys/values, no indexer1layer 2: Full, compression 2: writes keys/values and indexer keys, runs its indexer; own 128-token windowFull2layer 3: Reuse: KV from layer 2, indices from layer 2; own 128-token window (source inferred from the config's field names)layer 4: Reuse: KV from layer 2, indices from layer 2; own 128-token window (source inferred from the config's field names)layer 5: Reuse: KV from layer 2, indices from layer 2; own 128-token window (source inferred from the config's field names)layer 6: Reuse: KV from layer 2, indices from layer 2; own 128-token window (source inferred from the config's field names)layer 7: Reuse: KV from layer 2, indices from layer 2; own 128-token window (source inferred from the config's field names)layer 8: Full, compression 2: writes keys/values and indexer keys, runs its indexer; own 128-token windowFull8layer 9: Reuse: KV from layer 8, indices from layer 8; own 128-token window (source inferred from the config's field names)layer 10: Reuse: KV from layer 8, indices from layer 8; own 128-token window (source inferred from the config's field names)layer 11: Reuse: KV from layer 8, indices from layer 8; own 128-token window (source inferred from the config's field names)layer 12: Reuse: KV from layer 8, indices from layer 8; own 128-token window (source inferred from the config's field names)layer 13: Reuse: KV from layer 8, indices from layer 8; own 128-token window (source inferred from the config's field names)layer 14: Full, compression 2: writes keys/values and indexer keys, runs its indexer; own 128-token windowFull14layer 15: Reuse: KV from layer 14, indices from layer 14; own 128-token window (source inferred from the config's field names)layer 16: Reuse: KV from layer 14, indices from layer 14; own 128-token window (source inferred from the config's field names)layer 17: Reuse: KV from layer 14, indices from layer 14; own 128-token window (source inferred from the config's field names)layer 18: Reuse: KV from layer 14, indices from layer 14; own 128-token window (source inferred from the config's field names)layer 19: Reuse: KV from layer 14, indices from layer 14; own 128-token window (source inferred from the config's field names)layer 20: Full, compression 1: writes keys/values and indexer keys, runs its indexer; publishes candidate blocks to the later indexers; in prefill it writes keys and values for every prompt token, the rest of it runs only on the last 128 tokens; own 128-token windowFull20layer 21: Reuse: KV from layer 20, indices from layer 20; own 128-token window (source inferred from the config's field names)layer 22: Reuse: KV from layer 20, indices from layer 20; own 128-token window (source inferred from the config's field names)layer 23: Reuse: KV from layer 20, indices from layer 20; own 128-token window (source inferred from the config's field names)layer 24: Reindex: KV and indexer keys from layer 20, own indexer query; own 128-token window (source inferred from the config's field names)Reindex24layer 25: Reuse: KV from layer 20, indices from layer 24; own 128-token window (source inferred from the config's field names)layer 26: Reuse: KV from layer 20, indices from layer 24; own 128-token window (source inferred from the config's field names)layer 27: Reuse: KV from layer 20, indices from layer 24; own 128-token window (source inferred from the config's field names)layer 28: Reindex: KV and indexer keys from layer 20, own indexer query; own 128-token window (source inferred from the config's field names)Reindex28layer 29: Reuse: KV from layer 20, indices from layer 28; own 128-token window (source inferred from the config's field names)layer 30: Reuse: KV from layer 20, indices from layer 28; own 128-token window (source inferred from the config's field names)layer 31: Reuse: KV from layer 20, indices from layer 28; own 128-token window (source inferred from the config's field names)layer 32: Reindex: KV and indexer keys from layer 20, own indexer query; own 128-token window (source inferred from the config's field names)Reindex32layer 33: Reuse: KV from layer 20, indices from layer 32; own 128-token window (source inferred from the config's field names)layer 34: Reuse: KV from layer 20, indices from layer 32; own 128-token window (source inferred from the config's field names)layer 35: Reuse: KV from layer 20, indices from layer 32; own 128-token window (source inferred from the config's field names)layer 36: Reindex: KV and indexer keys from layer 20, own indexer query; own 128-token window (source inferred from the config's field names)Reindex36layer 37: Reuse: KV from layer 20, indices from layer 36; own 128-token window (source inferred from the config's field names)layer 38: Reuse: KV from layer 20, indices from layer 36; own 128-token window (source inferred from the config's field names)layer 39: Reuse: KV from layer 20, indices from layer 36; own 128-token window (source inferred from the config's field names)39 encoder (layers 0–19)decoder (layers 20–39) layer 20 writes keys and values for everyprompt token; above, only the last 128tokens run (approximate replay) layer 0 at the bottom Mistral Large 4 Mistral Large 4, layer 0: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 1: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 2: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 3: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 4: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 5: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 6: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 7: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 8: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 9: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 10: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 11: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 12: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 13: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 14: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 15: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 16: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 17: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 18: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 19: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 20: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 21: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 22: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 23: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 24: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 25: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 26: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 27: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 28: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 29: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 30: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 31: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 32: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 33: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 34: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 35: writes its own latent KV, read by layers 39, 43, 47, 51, 55, 59, 63, 67, 71; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 36: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 37: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 38: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 39: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 40: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 41: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 42: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 43: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 44: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 45: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 46: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 47: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 48: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 49: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 50: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 51: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 52: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 53: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 54: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 55: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 56: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 57: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 58: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 59: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 60: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 61: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 62: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 63: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 64: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 65: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 66: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 67: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 68: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 69: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 70: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 71: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)35 71 0 Mistral Large 4: layer 35's KV, read by 9 upper layers Mistral Large 4, layer 0: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 1: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 2: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 3: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 4: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 5: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 6: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 7: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 8: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 9: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 10: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 11: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 12: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 13: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 14: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 15: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 16: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 17: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 18: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 19: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 20: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 21: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 22: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 23: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 24: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 25: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 26: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 27: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 28: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 29: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 30: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 31: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 32: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 33: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 34: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 35: writes its own latent KV, read by layers 39, 43, 47, 51, 55, 59, 63, 67, 71; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 36: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 37: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 38: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 39: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 40: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 41: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 42: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 43: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 44: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 45: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 46: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 47: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 48: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 49: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 50: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 51: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 52: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 53: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 54: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 55: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 56: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 57: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 58: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 59: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 60: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 61: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 62: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 63: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 64: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 65: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 66: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 67: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 68: writes its own latent KV; runs its own indexer (kv_source_layers, indexer_types)Mistral Large 4, layer 69: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 70: writes its own latent KV; reuses the previous indexer's top-k (kv_source_layers, indexer_types)Mistral Large 4, layer 71: reads layer 35's latent KV; runs its own indexer (kv_source_layers, indexer_types) 0 35 71

Hover or tap a layer for its mode and where its keys, values and indices come from.

Figure 5.Cross-layer sharing in DeepSeek-V4.1-Flash: four layers write global keys and values and indexer keys, four more compute their own indexer queries over them, and every layer keeps its own 128-token window. In prefill, prompt tokens stop after layer 20's key and value projection. Mistral Large 4 (right) keeps one source, layer 35, for nine upper layers. From the configs; which source each reusing layer reads is inferred from field names.

Quality losses from sharing

Mistral Large 4’s report is the corpus’s clearest account of sharing that went wrong. Its first design stacked one global layer in four, CLA between pairs of global layers, and YOCO for the second half, which “reduced the KV-cache size by a factor of 16 compared to a full-attention baseline” (4 × 2 × 2 by the rule above, our reading), with YOCO’s “two-stage prefilling … 1.7× faster at 1M context”. Early ablations put the cost at “negligible or small enough”. After fine-tuning, SWE-bench scores fell with the windows (see “Quality cost, and what reduces it”) and “regressed further when applying CLA”. Mistral concluded that “models that perform equally well on language-modeling loss can nevertheless perform very differently on downstream agentic tasks”, and saw no such gap on knowledge benchmarks.

Mistral then changed an existing checkpoint: the window layers became full-context layers, and the CLA consumers got key and value weights copied from the last layer that had them, which changes what they compute and had to be trained back. YOCO stayed: at a ten times larger test scale “keeping YOCO yielded better performance”. In the shipped model all 72 layers are global sparse-attention layers, 44 of them with their own indexer, and nine of them (39, 43, …, 71) still read layer 35’s cache, close to CSA2’s Reindex mode. The former window layers above 35 compute their own keys and values, so prompt tokens cannot stop at the midpoint: the YOCO sharing that remains reduces memory but not prefill compute. The report’s figures print no values.

Sharing sparse indices costs little memory and saves decode reads. IndexCache (Bai et al., 2026; Z.ai ships it as IndexShare) lets most sparse layers reuse the top-k indices of the nearest layer with its own indexer (T3); GLM-5.2, GLM-5.3 and Hy4-preview run 21 indexers in 78 layers. With latents of 576 and indexer keys of 128 numbers, the cache shrinks at most 1+128/576=1.221 + 128/576 = 1.22 times: 1.15 for GLM-5.2, and 1.08 for Mistral Large 4’s 44 indexers in 72 layers (1.21 with its nine KV-reusing layers). But the indexer is the only part of a sparse layer that scans every past token at each step, so GLM-5.2 at 128K reads about 3.2 times fewer attention bytes per step with 21 indexers than with 78 (computed with indexer keys and latents at one width; 2.8 times with FP8 indexer keys beside BF16 latents). The paper picks the layers that keep an indexer by greedy search on language-modelling loss, or trains each kept indexer against the averaged attention of the layers it serves. In a recurrent hybrid the global layers are four apart, and in a 35B-A3B test Qwen found that sharing one index between two of them “remains below the baseline” on RULER up to 1M.

One model, many caches

A hybrid model holds several kinds of cache at once, each with its own size and rule for when entries are dropped: growing keys and values, window ring buffers, fixed states, compressed entries waiting for mm tokens, and indexer keys. Most serving engines use PagedAttention, which stores keys and values in fixed-size blocks of tokens, allocated as a sequence grows. DeepSeek’s V4 report says its mix “violates fundamental assumptions behind PagedAttention”, and treats each layer’s window, with the tokens waiting for compression, “as a state-space model”: a fixed-size state per sequence from a preallocated pool.

The clearest case is prefix caching, which keeps a prompt’s cache so that a later request with the same beginning can skip its prefill. For global layers it is free: keep the entries up to the shared prefix. A window layer keeps only its last WW entries, so a prefix can be resumed only where the engine kept every window layer’s entries at that boundary. A recurrent state exists only at the position it was last updated, so the engine must save copies at chosen positions, and they are large: NVIDIA’s Nemotron 3.5 report says that in vLLM “one Mamba cache page occupies at least as many bytes as approximately 4K tokens of Attention KV cache”. A prefix is reused only up to the last whole block it matches, so blocks that large miss most shared prefixes. Kimi K3’s report found that a page size shared with the state forced blocks of 1,024 to 6,144 tokens, at which “caching is nearly useless”. Moonshot hashed prefixes in 512-token blocks instead and saved KDA checkpoints at some of those block ends, mostly at conversation-turn boundaries. Nemotron 3 Ultra handles rejected drafts the same way, “snapshotting the SSM state at every draft step”. DeepSeek-V4’s window entries, uncompressed and in every layer, made up “nearly half” of its persistent prefix cache, kept on SSD or in host memory (V4.1 report); V4.1 stores none and replays the last window approximately.

Parallelism adds constraints. Context parallelism splits a long prompt across GPUs, which is easier for cheap layers: window layers “exchange only the KV their queries can reach” (MiMo-V2.6) and linear layers pass on “a fixed-size recurrent state” (Kimi K3). Pipeline parallelism puts groups of layers on different GPUs, so layers that share a cache must sit in one group or pass it along; V4.1’s prefill shortcut runs only if the group holding layer 20 holds every layer above it, and only with a drafter that reads no hidden state outside the last window.

Comparison

LeverWhat it cachesSavesCostsIn the map
Window and global layersWW per window layer, TT per global layergrowing cache to 1/(r+1)1/(r+1) past T∗T^*no read past WW by content; agentic regression at 3:1 (Mistral)12 models, 8 labs; 1:1 to 6:1
Recurrent and global layersfixed state, TT per global layerthe sameat most dkd_k exact associations per head; large state below T∗T^*; harder prefix caching and rollback8 models, 4 labs; about 3:1 to 5:1
Placementnothing extranothingwrong layers lose retrieval (GLM-5 retrofit)rules differ by lab
Branches in one layerwindow plus compressed entrieslocal detail without a global layera window cache in every layerDeepSeek-V4, V4.1
KV sharing (CLA, YOCO, CED, CSA2)one source for several layersmemory; with YOCO or CED, prefillpast tokens seen at one depth; same decode reads; CLA regression (Mistral); CED replay approximateDeepSeek-V4.1, Mistral Large 4
Index sharing (IndexCache)indexer keys in fewer layersdecode reads, indexer computebelow baseline across distant global layers (Qwen test)GLM-5.2, GLM-5.3, Hy4, Mistral Large 4

Where the models in the map sit

The table lists the 35 language models whose code was checked, with caches computed from each config.json for one sequence at 128K tokens, every kind in BF16, and before tensor parallelism (which copies layers with few KV heads, and MLA latents, onto every GPU). “All global” gives every layer the model’s global layer shape. MLA layers count at their latent width, as serving engines store them (Kimi K3’s reference code caches expanded keys and values, 53 times more). DeepSeek-V4 and V4.1 are not counted as window interleaves, since every layer has a window branch. Reading the code changed six of the site’s stored code-check answers: five DeepSeek variants marked as interleaves or unclear, and one Nemotron model marked unclear.

ModelCheap layersCheap : globalWindowCache at 128KAll globalCut
gpt-oss-120bwindow18 : 181284.50 GiB9.00 GiB2.00
Laguna-S-2.1window36 : 125126.0724.003.95
Step-3.7-Flashwindow33 : 125126.0622.503.71
Step-5-Previewwindow (global layers sparse)69 : 235128.7334.503.95
LFM3-1.7Bconv + window18 : 62561.506.003.99
Kolibri-1window40 : 105132.5412.504.92
MiMo-V2.5, V2.6-Flashwindow39 : 91282.8415.005.29
Gemma 4 31Bwindow50 : 101,02410.7860.005.57
Inklingwindow55 : 115125.7233.005.77
MiMo-V2.5-Pro, V2.6-Prowindow60 : 101286.2943.756.96
Qwen3.5-397B-A17BGDN45 : 153.8415.003.90
Qwen3.6-35B-A3BGDN30 : 102.5310.003.95
Qwen3.8-FlashGDN (global layers sparse)36 : 123.1512.383.93
Kimi K3KDA69 : 243.6013.083.64
GLM-5.3-FlashKDA34 : 111.535.983.90
Nemotron-3-UltraMamba-248 : 121.697.504.43
Nemotron-3.5-LightningMamba-223 : 60.773.624.69
Nemotron-3.5-Super-VLMamba-240 : 81.086.005.55
Compression and sharing, against a dense layer with one 512-wide key that is also the value:
DeepSeek-V4-Flash (V4-Flash, -0731, -Vision-Exp)window branch in every layerCSA 21, HCA 20, 2 window-only1280.855.386.36
DeepSeek-V4-Pro (V4-Pro, -0813)window branch in every layerCSA 30, HCA 311281.217.626.30
DeepSeek-V4.1-Flash (and -1017)window branch, KV and index reuse4 KV sources in 401280.405.0012.64
Sharing in all-global stacks:
Mistral Large 4KV reuse in 9 layers, index reuse72 global10.2312.381.21
GLM-5.2, GLM-5.3, Hy4-previewindex reuse78 global11.6213.411.15
DeepSeek-V3.2, MiniMax-M3, M3.1-Flash, Hy3noneall global10.48 to 40.00same (MiniMax-M3: 16.88)1.00 (MiniMax-M3: 1.01)
context T =
  • window cheap layers
  • recurrent cheap layers
  • convolution + window (LFM3)
  • y = 1 / (1 + x)
0.80.850.90.951.000.050.10.15Kolibri-1: bound 5.00 (40:10), cut 4.92 at 128K, T* = 2,052 tokens (T*/T = 0.016, cut/bound = 0.985)LFM3-1.7B: bound 4.00 (18:6), cut 3.99 at 128K, T* = 244 tokens (T*/T = 0.002, cut/bound = 0.998)Laguna-S-2.1: bound 4.00 (36:12), cut 3.95 at 128K, T* = 1,536 tokens (T*/T = 0.012, cut/bound = 0.988)Step-3.7-Flash: bound 3.75 (33:12), cut 3.71 at 128K, T* = 1,408 tokens (T*/T = 0.011, cut/bound = 0.989)Step-5-Preview: bound 4.00 (69:23), cut 3.95 at 128K, T* = 1,536 tokens (T*/T = 0.012, cut/bound = 0.988)Inkling: bound 6.00 (55:11), cut 5.77 at 128K, T* = 5,308 tokens (T*/T = 0.040, cut/bound = 0.961)MiMo-V2.5 / MiMo-V2.6-Flash: bound 5.33 (39:9), cut 5.29 at 128K, T* = 1,109 tokens (T*/T = 0.008, cut/bound = 0.992)MiMo-V2.5-Pro / MiMo-V2.6-Pro: bound 7.00 (60:10), cut 6.96 at 128K, T* = 768 tokens (T*/T = 0.006, cut/bound = 0.994)Qwen3.5-397B-A17B: bound 4.00 (45:15), cut 3.90 at 128K, T* = 3,216 tokens (T*/T = 0.025, cut/bound = 0.976)Qwen3.6-35B-A3B: bound 4.00 (30:10), cut 3.95 at 128K, T* = 1,632 tokens (T*/T = 0.012, cut/bound = 0.988)Qwen3.8-Flash: bound 4.00 (36:12), cut 3.93 at 128K, T* = 2,351 tokens (T*/T = 0.018, cut/bound = 0.982)Kimi K3: bound 3.88 (69:24), cut 3.64 at 128K, T* = 8,587 tokens (T*/T = 0.066, cut/bound = 0.939)GLM-5.3-Flash: bound 4.09 (34:11), cut 3.90 at 128K, T* = 6,516 tokens (T*/T = 0.050, cut/bound = 0.953)Nemotron-3.5-Lightning: bound 4.83 (23:6), cut 4.69 at 128K, T* = 4,109 tokens (T*/T = 0.031, cut/bound = 0.970)gpt-oss-120b: bound 2.00 (18:18), cut 2.00 at 128K, T* = 128 tokens (T*/T = 0.001, cut/bound = 0.999)gpt-oss-120bGemma 4 31B: bound 6.00 (50:10), cut 5.57 at 128K, T* = 10,240 tokens (T*/T = 0.078, cut/bound = 0.928)Gemma 4 31BNemotron-3-Ultra: bound 5.00 (48:12), cut 4.43 at 128K, T* = 16,960 tokens (T*/T = 0.129, cut/bound = 0.885)Nemotron-3-UltraNemotron-3.5-Super-VL: bound 6.00 (40:8), cut 5.55 at 128K, T* = 10,640 tokens (T*/T = 0.081, cut/bound = 0.925)Nemotron-3.5-Super-VLT* / T (fixed caches over the growing one)cut / bound (L / n_G), axis from 0.8

Hover or tap a dot for its bound, cut and T*.

Figure 6.How close each hybrid gets to its bound r+1, one sequence in BF16. Every dot lies on the curve 1/(1 + T*/T). At 128K the farthest from the bound is Nemotron-3-Ultra, whose large Mamba-2 states put T* at about 17,000 tokens, then Nemotron-3.5-Super-VL (Mamba-2 states) and Gemma 4 (wide window layers). Computed from the configs.

Easy to get wrong

  • A 5:1 ratio does not cut the cache fivefold. Against an all-global stack the cut approaches six, and only at contexts well past T∗T^*. MiMo-V2-Flash’s “nearly 6×” counts the layers of one block; over the shipped 48-layer configs the computed cut is 5.29.
  • Only Kimi K3 and Gemma 4 state that the final layer is global. Six models in the map end on a cheap layer.
  • Sharing keys and values across layers cuts the memory a sequence holds; each layer still reads the shared cache at every decode step. Sharing indices does the reverse in sparse stacks: little memory, about three times fewer attention bytes per step for GLM-5.2 at 128K.
  • Language-modelling loss can miss a hybrid’s cost. Mistral’s window and CLA changes looked harmless in early pre-training ablations and lowered SWE-bench scores after fine-tuning.
  • The ablations behind these choices ran on test models at fixed ratios. No report of a model in the map sweeps the ratio of its own architecture; the nearest are Gemma 3’s plots (minimal perplexity change up to 7:1 on 2B and 9B models, perplexity only) and Jamba’s 3:1 against 7:1.

Check yourself

  1. Laguna-S-2.1 has 48 layers, 12 of them global, eight KV heads of width 128 in both kinds, and a window of 512. What is its T∗T^*, how large is its cache for one sequence at 1M tokens in BF16, and how close is the cut to its bound?
    AnswerEach layer caches 2⋅8⋅128=2,0482 \cdot 8 \cdot 128 = 2{,}048 numbers, 4 KiB, per token, and r=3r = 3, so T∗=3⋅512=1,536T^* = 3 \cdot 512 = 1{,}536. Global: 12⋅4 KiB⋅1,048,576=4812 \cdot 4\,\text{KiB} \cdot 1{,}048{,}576 = 48 GiB; window: 36⋅4 KiB⋅512=7236 \cdot 4\,\text{KiB} \cdot 512 = 72 MiB; total about 48.07 GiB. All global: 192 GiB, so a cut of 4/(1+1,536/1,048,576)≈3.994/(1 + 1{,}536/1{,}048{,}576) \approx 3.99.
  2. GLM-5.2 runs an indexer in 21 of its 78 sparse layers. Why does that shrink its cache by only 1.15 times, while a decode step at 128K reads about three times fewer attention bytes?
    AnswerEvery layer still stores its 576-wide latent; only the 128-wide indexer keys go from 78 layers to 21, at most a 1+128/576=1.221 + 128/576 = 1.22 cut. At each step an indexer reads its keys for all 131,072 past tokens, while attention reads only 2,048 selected latents, so 21 scans instead of 78 cut the reads about 3.2 times at one width (2.8 with FP8 indexer keys).
  3. Why can prompt tokens skip the upper half in YOCO but not in DeepSeek-V4.1’s CED, and why is CED’s shortcut approximate?
    AnswerYOCO’s upper layers write no keys or values of their own, so nothing needs their outputs at prompt positions except the last. CED’s upper layers keep their own 128-token windows, computed from their own hidden states, so the last window must run through them. Exact window entries there would depend on about 2,500 earlier positions through the windows below; DeepSeek runs only the last 128, with each window cut off at the start of that segment.

In the map

Methods

Taxonomy nodes

Further reading

  • Gemma 3 Technical Report (2025), arXiv:2503.19786, §5.2: the ratio and window plots, 2B and 9B.
  • Sun et al., You Only Cache Once (2024), arXiv:2405.05254: the prefill early exit, and how much of its speedup comes from the recurrent self-decoder.
  • Brandon et al., Cross-Layer Attention (2024), arXiv:2405.12981: sharing factors 2 to 4, and the non-uniform patterns that did worse.
  • MiMo-V2-Flash report (2026), arXiv:2601.02780, §2.2: the window and sink ablation tables.
  • GLM-5 report (2026), arXiv:2602.15763, §2.1.2: the placement search and retrofit tables.
  • Mistral Large 4 technical report, §3: the hybrid that was undone, with the SWE-bench plots.
  • DeepSeek-V4.1-Flash technical report, §2.2 to 2.3 and §3.2: CED, the CSA2 modes, bounded replay.
  • DeepSeek-V4 report §3.5 and Kimi K3 report §5.4.1: serving caches of mixed layer types.
↑