Open Models Almanac

Token mixer · T2

Fixed-state token mixers

Sliding windows, linear attention, Mamba-2, Gated DeltaNet and KDA trade exact recall for constant decode memory

Illustration of a long paper ribbon covered in small marks unrolling from a spool into a small square sand tray. In the tray the newest raked grooves are crisp, older ones fade, and a small rake smooths one strip flat.

Qwen3.5-397B has 60 layers. At the model’s maximum context of 262,144 tokens, each of its 15 full-attention layers stores 512 MiB of keys and values for one conversation. The other 45 store 4 MiB each at any length (computed from the config). They are Gated DeltaNet layers, each keeping a fixed-size matrix that summarizes the past and is rewritten as each token arrives.

Gated DeltaNet is one of several token mixers (the sublayer that moves information between positions) whose memory doesn’t grow with the context. Sliding-window attention keeps the last WW tokens; linear attention, Mamba-2, Gated DeltaNet and Kimi Delta Attention (KDA) compress the prefix into a matrix. Every mapped model that uses them is a hybrid with some full-attention layers.

The cost these layers remove

A model runs in two phases: prefill processes the prompt, all tokens at once, in large matrix multiplications, and decode generates one token at a time. A softmax-attention layer keeps the keys and values of every past token, the KV cache (T1), which grows by 2gdhb2 g d_h b bytes per token for gg key/value heads of dimension dhd_h at bb bytes per number. Qwen3.5-397B’s full-attention layers have g=2g = 2, dh=256d_h = 256 (num_key_value_heads, head_dim), so in BF16 (b=2b = 2) each grows by 2 KiB per token, 256 MiB at TT = 128K (computed). Each decoded token reads all of it, and prefill attention work grows with T2T^2. A fixed-state layer reads and updates a state SS of constant size once per token, and its prefill work grows with TT.

Window W

keysvalues / statehatched = cached state · one layer, one sequence, Qwen3.5-397B shape: BF16 cache, FP32 state

at
128K tokens
full attention
256 MiB
Gated DeltaNet
4 MiB
sliding window
256 KiB

full attention holds 64× the state. Hover or focus a line for its formula; the Gated DeltaNet layer also keeps a 72 KiB convolution buffer (crossover 2,084 with it).

Figure 1.Memory held by one layer for one sequence. The full-attention cache passes the Gated DeltaNet state at about 2,000 tokens and is 512 times larger at a million.

Sliding-window attention

In sliding-window attention (SWA; Beltagy et al., 2020; Jiang et al., 2023) each query attends to the WW most recent positions, its own included, with ordinary softmax. Recall inside the window is exact; anything older is gone from that layer. The layer holds 2Wgdhb2 W g d_h b bytes, 256 KiB for gpt-oss-120b (W=128W = 128 in sliding_window, 8 KV heads of 64; computed), and prefill work grows as T⋅WT \cdot W. A contiguous cache can be a ring buffer, the newest entry overwriting the oldest, with kk spare slots if kk speculated tokens may be rejected; paged engines free whole blocks instead.

Older information arrives by relay through positions that picked it up in lower layers: a query at layer LL reaches L(W−1)L(W-1) positions back, which the Mistral 7B paper rounds to “approximately 131K tokens” for 32 layers with WW = 4,096. Each hop passes through one position’s hidden state, which also carries that position’s own content, so the relay is lossy.

queryits windowwithin reachrelay path from the oldest reachable token

W = 4, 4 layers: the query at position 24 reaches 12 positions back (to 12), 4 × (W − 1). Position 13 arrives through 3 intermediate positions (16, 19, 22).

Hover a token to see how many hops it needs.

Figure 2.A query in an SWA stack sees further back at each layer down, through a relay of intermediate tokens. A global layer reaches every position directly, which is why the mapped models keep some (T5).

Windows in the map are 128 tokens (gpt-oss, four MiMo models, DeepSeek-V4’s SWA branch), 512 (Laguna-S-2.1, Step-3.7-Flash, Inkling, Kolibri-1) or 1,024 (Gemma 4). MiMo-V2-Flash found 128 better than 512 in its hybrids up to 32K context, and Kolibri-1 the same up to 64K at one scale, with 512 ahead at 256K. The three labs using 128 add an attention sink (T6; Xiaomi on SWA layers only), a learned logit per head that absorbs attention mass when no key in the window fits; MiMo-V2-Flash’s hybrid without one “suffers noticeable performance degradation”. Of the four labs at 512 or more in the code check, two gate the attention output per head instead. In the site’s code checks (each model’s code tested against a checklist), sliding-window-attention is present for 15 of 28 models: six DeepSeek-V4 models, whose every layer has a 128-token window, and nine that interleave whole SWA layers with global ones.

Linear attention

Softmax attention weights vsv_s by exp⁡(qt⊤ks)\exp(q_t^\top k_s) over a normalizing sum. That exponential doesn’t split into a function of qtq_t times a function of ksk_s, so the past can’t be summed up before the query arrives. Katharopoulos et al. (2020) used a feature map ϕ\phi with exp⁡(q⊤k)≈ϕ(q)⊤ϕ(k)\exp(q^\top k) \approx \phi(q)^\top \phi(k) and kept the normalizer ∑sϕ(qt)⊤ϕ(ks)\sum_s \phi(q_t)^\top \phi(k_s); current models use SiLU and L2 normalization as ϕ\phi, drop the normalizer and normalize each head’s output instead. Then

ot=∑s≤t(qt⊤ks) vs=∑s≤tvs ks⊤qt=(∑s≤tksvs⊤) ⁣⊤qt=St⊤qt,St=St−1+ktvt⊤,o_t = \sum_{s \le t} (q_t^\top k_s)\, v_s = \sum_{s \le t} v_s\, k_s^\top q_t = \Big(\sum_{s \le t} k_s v_s^\top\Big)^{\!\top} q_t = S_t^\top q_t, \qquad S_t = S_{t-1} + k_t v_t^\top ,

with St∈Rdk×dvS_t \in \mathbb{R}^{d_k \times d_v} for key width dkd_k and value width dvd_v. Each token adds an outer product, each query reads with a matrix-vector product, and the state never grows.

How much a dk×dvd_k \times d_v matrix can hold

Stack mm stored keys as the rows of KK and their values as the rows of VV. Exact recall through the linear read-out S⊤qS^\top q means KS=VKS = V, which for arbitrary values needs linearly independent keys, so m≤dkm \le d_k. Beyond that, any SS loses at least a share 1−dk/m1 - d_k/m of random values’ energy. The bound is per key head, and keys are shared: every linear layer in the map has dk=128d_k = 128, Qwen3.5-397B’s 64 value heads read 16 key heads, 2,048 key directions per layer, and Nemotron 3 Ultra’s 256 heads share 8 key groups, 1,024 (computed). Extra value heads widen what is stored under a key without adding slots.

Additive writing falls short of the bound. With S=K⊤VS = K^\top V, reading with kik_i returns

S⊤ki=∥ki∥2 vi+∑j≠i(kj⊤ki) vj,S^\top k_i = \|k_i\|^2\, v_i + \sum_{j \ne i} (k_j^\top k_i)\, v_j ,

the right value plus crosstalk, exact only for orthonormal keys. For random unit keys and values each kj⊤kik_j^\top k_i has variance 1/dk1/d_k, so the crosstalk’s expected squared norm is (m−1)/dk(m-1)/d_k times the signal’s, and the read-out’s cosine with viv_i is about (1+(m−1)/dk)−1/2(1 + (m-1)/d_k)^{-1/2}: above 0.9 only for m<1+0.23 dkm < 1 + 0.23\, d_k, about 16 at dk=64d_k = 64. Nothing is ever removed, so the crosstalk keeps growing with mm.

One bound for every layer

Every layer here computes O=ΩVO = \Omega V, values as rows of VV, with a lower-triangular mixing matrix Ω\Omega that depends on queries, keys and gates but not on the values. Writing a matrix-state update as St=AtSt−1+βtktvt⊤S_t = A_t S_{t-1} + \beta_t k_t v_t^\top, with a transition AtA_t (II for linear attention) and a write strength βt\beta_t (1 so far),

Ωts=qt⊤AtAt−1⋯As+1 βsks(s≤t).\Omega_{ts} = q_t^\top A_t A_{t-1} \cdots A_{s+1}\, \beta_s k_s \qquad (s \le t).

For a split point τ\tau and t>τ≥st > \tau \ge s the product splits at τ\tau into a length-dkd_k row vector that depends on tt times a column vector that depends on ss, so the past-to-future block Ω[τ+1:T,  1:τ]\Omega_{[\tau+1:T,\; 1:\tau]} has rank at most dkd_k. For SWA the block is nonzero only where t−s≤W−1t - s \le W - 1, rank at most W−1W - 1; for softmax attention nothing bounds it below min⁡(τ,T−τ)\min(\tau, T - \tau). Recalling mm arbitrary past values at later queries needs rank mm there, so the capacity bound holds for every write rule and gate at once, input by input. Matrices with low-rank off-diagonal blocks are called semiseparable; Mamba-2’s “state space duality” rests on them (Dao & Gu, 2024), and the chunkwise algorithm stores these blocks in factored form.

Sliding window, W = 6

Gated delta rule, d_k = 4

Full softmax attention

|Ωts|, 0 to each panel's largest entry (square-root scale); rows are queries t, columns keys s, T = 32. The outlined block is rows after τ, columns up to τ; dots below the dashed line (10−9 × the largest) do not count towards the rank.

Ranks: 5, 4, 16 (window, gated delta, full attention).

Figure 3.The same past-to-future block in three mixers. A window or a fixed state limits its rank, and with it how many independent past values later tokens can read; full attention has no such limit.

Parallel, recurrent and chunkwise

With queries, keys and values as rows of Q,K,VQ, K, V (rows are tokens), linear attention is O=(QK⊤⊙M)VO = (QK^\top \odot M)V for the causal 0/1 mask MM: O(T2)O(T^2) work in large matrix products. The recurrent form is O(T)O(T) work in TT small sequential steps, which suits decoding. Training and long prompts cut the sequence into chunks of CC tokens, with S[n]S_{[n]} the state entering chunk nn and MCM_C the C×CC \times C causal mask:

O[n]=Q[n]S[n]⏟earlier chunks+(Q[n]K[n]⊤⊙MC)V[n]⏟inside the chunk,S[n+1]=S[n]+K[n]⊤V[n].O_{[n]} = \underbrace{Q_{[n]} S_{[n]}}_{\text{earlier chunks}} + \underbrace{\big(Q_{[n]} K_{[n]}^\top \odot M_C\big) V_{[n]}}_{\text{inside the chunk}}, \qquad S_{[n+1]} = S_{[n]} + K_{[n]}^\top V_{[n]} .

Every operation is now a matrix product, the work of tensor cores, the GPU units that multiply small dense matrices (989 TFLOP/s in BF16 on an H100, against 67 for ordinary FP32 arithmetic). Per token the chunk part costs about C(dk+dv)C(d_k + d_v) multiply-adds and the state part 2dkdv2 d_k d_v, and the sequential chain shrinks to T/CT/C steps. Qwen’s code and the KDA kernels use C=64C = 64, Nemotron chunk_size: 128.

Parallel

Recurrent

Chunkwise, C = 4

querieskeysvalues / state (hatched = cached)computed as attentioncarried by a chunk state

sequential steps, T / C
16 / 4 = 4
intra-chunk multiply-adds per token, C(dk + dv)
4 × (128 + 128) = 1,024
state multiply-adds per token, 2 dk dv
2 × 128 × 128 = 32,768

C = 4: 4 diagonal blocks computed as small attention matrices, 4 chunk states in sequence.

T = 16 tokens drawn; the counters use dk = dv = 128, as in every linear layer in the map.

Figure 4.The same layer computed in three schedules. The chunk size trades quadratic work inside chunks against the length of the sequential chain between them.

No model in the map uses plain linear attention. Decay limits how much old content piles up, and the delta rule cancels the crosstalk on the key being written.

Decay

Multiplying the state by a decay αt∈(0,1)\alpha_t \in (0,1) before each write, St=αtSt−1+ktvt⊤S_t = \alpha_t S_{t-1} + k_t v_t^\top, weights token ss in the read at tt by ∏r=s+1tαr\prod_{r=s+1}^{t}\alpha_r, so the parallel form uses a decay mask Λ\Lambda with those entries (0 above the diagonal) in place of MM. Old pairs stop crowding new ones, and fade whether or not they mattered. RetNet (Sun et al., 2023) fixes α\alpha per head (a total weight 1/(1−α)1/(1-\alpha)). Gated linear attention (GLA; Yang et al., 2023) computes a decay per key channel from the input, Diag(αt)St−1\mathrm{Diag}(\alpha_t) S_{t-1}, and Mamba-2 one per head, so a value can be held across irrelevant text and dropped at a boundary.

A scalar decay keeps the chunkwise form cheap. With γi=∏r=1iαr\gamma_i = \prod_{r=1}^{i}\alpha_r the decay since the chunk start and Γ=diag(γ1,…,γC)\Gamma = \mathrm{diag}(\gamma_1, \dots, \gamma_C),

O[n]=Γ Q[n]S[n]+(Q[n]K[n]⊤⊙Λ[n])V[n],S[n+1]=γC S[n]+K[n]⊤diag(γC/γj) V[n],O_{[n]} = \Gamma\, Q_{[n]} S_{[n]} + \big(Q_{[n]} K_{[n]}^\top \odot \Lambda_{[n]}\big) V_{[n]}, \qquad S_{[n+1]} = \gamma_C\, S_{[n]} + K_{[n]}^\top \mathrm{diag}(\gamma_C/\gamma_j)\, V_{[n]},

with (Λ[n])ij=γi/γj(\Lambda_{[n]})_{ij} = \gamma_i/\gamma_j for j≤ij \le i. Every factor is at most 1, so nothing overflows. A decay per channel breaks this (KDA below).

Hover a cell for its product of decays. Queries after position 12 see little before it: α12 = 0.05.

Figure 5.The decay mask for three ways of choosing α. A dip in a data-dependent decay acts as a reset at a boundary in the text.

Mamba-2

A state-space model (SSM) is a linear system h˙=Ah+Bx\dot h = A h + B x, read out as y=Chy = C h, run in steps of size Δ\Delta: the transition becomes exp⁡(ΔA)\exp(\Delta A) and the input term, approximately, ΔB\Delta B. Mamba (Gu & Dao, 2023) made it selective, computing Δ\Delta, BB and CC from each token; its decay varies along both dimensions of the state, can’t be folded into a query or key, and runs as a scan, without matrix products. Mamba-2 (Dao & Gu, 2024) makes the transition a scalar times the identity per head. With BtB_t as the key, CtC_t as the query and the input xtx_t as the value, a head computes

St=eΔtA St−1+Δt ktvt⊤,ot=St⊤qt+D vt,S_t = e^{\Delta_t A}\, S_{t-1} + \Delta_t\, k_t v_t^\top, \qquad o_t = S_t^\top q_t + D\, v_t ,

with a learned A<0A < 0, Δt=softplus(⋅)>0\Delta_t = \mathrm{softplus}(\cdot) > 0 and a learned skip DD: decayed linear attention with αt=eΔtA\alpha_t = e^{\Delta_t A} and key Δtkt\Delta_t k_t. In Nemotron 3 Ultra dkd_k is ssm_state_size: 128 and dvd_v is mamba_head_dim: 64, over mamba_num_heads: 256.

One step size sets both forgetting and writing: the write weight Δt=ln⁡(1/αt)/∣A∣\Delta_t = \ln(1/\alpha_t)/|A| grows with how much is forgotten, so a head can’t hold its state and write strongly at once. Keys and queries are shared by groups of heads (n_groups: 8), Dao & Gu’s multi-value attention. A kernel-4 causal convolution (conv_kernel) precedes the state, and the output is gated, then normalized in groups.

The delta rule

Decay frees space by forgetting everything a little. The delta rule corrects one association. Before writing vtv_t under ktk_t it subtracts what the state already returns for ktk_t:

St=St−1+βt kt(vt−St−1⊤kt)⊤=(I−βtktkt⊤)St−1+βt ktvt⊤,S_t = S_{t-1} + \beta_t\, k_t \big(v_t - S_{t-1}^\top k_t\big)^\top = \big(I - \beta_t k_t k_t^\top\big) S_{t-1} + \beta_t\, k_t v_t^\top ,

with a write strength βt∈(0,1)\beta_t \in (0,1) from the input. This is one least-mean-squares gradient step on 12∥S⊤kt−vt∥2\tfrac12 \|S^\top k_t - v_t\|^2 (Widrow & Hoff, 1960; DeltaNet in Schlag et al., 2021). With ∥kt∥=1\|k_t\| = 1 and βt=1\beta_t = 1 the state then returns exactly vtv_t.

That holds at the moment of writing. With unit keys and β=1\beta = 1, writing pair jj moves the read at an older key kik_i by

Δ(S⊤ki)=(kj⊤ki) (vj−Sj−1⊤kj),\Delta\big(S^\top k_i\big) = (k_j^\top k_i)\,\big(v_j - S_{j-1}^\top k_j\big),

so after one pass pair ii is exact for arbitrary values only if every later key is orthogonal to kik_i. One pass of the delta rule therefore falls short of least squares and of the dkd_k bound, even for linearly independent keys. Recursive least squares (RLS) reaches the bound in one pass,

St=St−1+gt(vt−St−1⊤kt)⊤,gt=Pt−1kt1+kt⊤Pt−1kt,Pt=Pt−1−gtkt⊤Pt−1,S_t = S_{t-1} + g_t \big(v_t - S_{t-1}^\top k_t\big)^\top, \quad g_t = \frac{P_{t-1} k_t}{1 + k_t^\top P_{t-1} k_t}, \quad P_t = P_{t-1} - g_t k_t^\top P_{t-1},

with PP the inverse covariance of the keys so far. The delta rule is RLS with PP replaced by βI\beta I, which saves a dk×dkd_k \times d_k matrix per key head and O(dk2)O(d_k^2) work per token.

The factor’s eigenvalue along ktk_t, 1−βt∥kt∥21 - \beta_t \|k_t\|^2, leaves (−1,1)(-1, 1) once βt∥kt∥2>2\beta_t\|k_t\|^2 > 2, so every delta-rule layer in the map L2-normalizes ktk_t (and qtq_t, for the read-out scale). Diagonal transitions with entries in (0,1](0, 1], as in linear, decayed and Mamba-2 layers, commute and can’t track the parity of a bit string at finite precision (Merrill, Petty & Sabharwal, 2024); delta transitions don’t commute, and with β∈(0,2)\beta \in (0, 2) they can (Grazzi et al., 2024). The mapped models keep βt∈(0,1)\beta_t \in (0, 1).

value spaceAdditive: S += k vᵀvoldresultvalue spaceDelta rule, β = 1.00v − oldv = resultold
  • old read-out Sᵀk
  • new value v
  • correction v − old
  • read-out after the write

Additive: result = old + v, 1.42 from v (the whole old read-out stays).
Delta: result = old + β(v − old), 0.00 from v (β = 1: exactly v).

Drag the end of the “old” arrow (or focus it and use the arrow keys). Toy 2-D vectors.

Figure 6.Writing value v under a key the state already knows. Additive writing adds to what is there; the delta rule moves the read-out towards v by a fraction β of the error.

A chunk’s update is a product of CC different factors, which Yang et al. (2024) collapse with the UT transform. With the chunk’s keys and values as rows of KK and VV, let N=I+tril(diag(β)KK⊤,−1)N = I + \mathrm{tril}(\mathrm{diag}(\beta) K K^\top, -1) (strictly lower part), K~=N−1diag(β)K\tilde K = N^{-1}\mathrm{diag}(\beta) K and V~=N−1diag(β)V\tilde V = N^{-1}\mathrm{diag}(\beta) V. Then

∏i=C1(I−βikiki⊤)=I−K⊤K~,S[n+1]=S[n]+K⊤(V~−K~S[n]),\prod_{i=C}^{1}\big(I - \beta_i k_i k_i^\top\big) = I - K^\top \tilde K, \qquad S_{[n+1]} = S_{[n]} + K^\top\big(\tilde V - \tilde K S_{[n]}\big), O[n]=QS[n]+(QK⊤⊙MC)(V~−K~S[n]),O_{[n]} = Q S_{[n]} + \big(Q K^\top \odot M_C\big)\big(\tilde V - \tilde K S_{[n]}\big),

with MCM_C including the diagonal: one C×CC \times C triangular solve, then matrix products (torch.linalg.solve_triangular in Qwen’s code; FLA kernels for Kimi and GLM).

Gated DeltaNet

Gated DeltaNet (GDN; Yang, Kautz & Hatamizadeh, 2024) combines the two, with one scalar decay per head and token, so a head can clear its state at a new topic and still edit single pairs:

St=αt(I−βtktkt⊤)St−1+βt ktvt⊤.S_t = \alpha_t \big(I - \beta_t k_t k_t^\top\big) S_{t-1} + \beta_t\, k_t v_t^\top .

In the Qwen code log⁡αt=−eAlog⁡ softplus(at+b)\log\alpha_t = -e^{A_{\log}}\,\mathrm{softplus}(a_t + b) (Mamba-2’s form, learned Alog⁡A_{\log} and bb), and βt\beta_t is a sigmoid. Around the recurrence sit a kernel-4 causal convolution on q,k,vq, k, v (linear_conv_kernel_dim: 4) and a per-head RMSNorm followed by an output gate. Qwen3.5 and 3.6 gate with SiLU, as the original GDN does; Qwen3.8-Flash sets output_gate_type: "sigmoid", and the Qwen3.8-Next report that describes it says “we use the bounded sigmoid gate … and observe consistent improvements across our experiments”. A sigmoid can only scale a channel down, which limits activation outliers (T6).

Input token x_t, hidden_size 4,096.xt: 4,096in_proj_qkv: 4,096 → 2·16·128 + 64·128 = 12,288 (linear_num_key_heads, linear_key_head_dim, linear_num_value_heads, linear_value_head_dim).proj qkv→ 12,288 channelsin_proj_b: 4,096 → 64, one write strength per value head (linear_num_value_heads).proj b→ 64in_proj_a: 4,096 → 64, one decay logit per value head (linear_num_value_heads).proj a→ 64in_proj_z: 4,096 → 64·128 = 8,192, the output gate's input.proj z→ 8,192Causal depthwise convolution over the 12,288 qkv channels, kernel 4 (linear_conv_kernel_dim), then SiLU.conv, k = 4then SiLUCarried between tokens: the convolution's last 3 inputs, 3 × 12,288 channels × 2 B = 72 KiB per layer.last3 inputsβ_t = sigmoid(b), in (0, 1): write strength, one per value head (64).β = σ(b)64 headslog α_t = −exp(A_log) · softplus(a + dt_bias), Mamba-2's form; A_log and dt_bias learned per value head (64).decay αMamba-2 formq_t: 16 key heads × 128 (linear_num_key_heads × linear_key_head_dim); each serves 4 value heads.q16×128k_t: 16 key heads × 128; each serves 4 value heads.k16×128v_t: 64 value heads × 128 (linear_num_value_heads × linear_value_head_dim).v64×128q and k are L2-normalized per head (use_qk_l2norm_in_kernel), which keeps the delta rule's factor I − β k kᵀ contracting.L2Norm q, kCarried between tokens: the state, 64 value heads × 128 × 128 × 4 B (FP32, mamba_ssm_dtype) = 4 MiB per layer. Read-out o_t = S_tᵀ q_t.St = αt (I − βt kt ktᵀ) St−1 + βt kt vtᵀ64 heads × 128 × 128, FP324 MiB per layerot = Stᵀ qt: 64×128Gated RMSNorm over each head's 128 outputs: weight · x̂, weight initialised to 1 (in code; not the zero-centred 1 + w form of the other norms).RMSNorm per headweight · x̂z: 8,192Output gate: RMSNorm(o) ⊙ SiLU(z) (hard-coded in modeling_qwen3_5_moe.py). SiLU in Qwen3.5/3.6, sigmoid in Qwen3.8; a sigmoid only scales a channel down.⊙ SiLU(z)SiLU in Qwen3.5/3.6,sigmoid in Qwen3.8out_proj: 64·128 = 8,192 → 4,096 (hidden_size).out_proj → 4,096Input token x_t, hidden_size 2,560.xt: 2,560in_proj_qkv: 2,560 → 2·16·128 + 48·128 = 10,240 (linear_num_key_heads, linear_key_head_dim, linear_num_value_heads, linear_value_head_dim).proj qkv→ 10,240 channelsin_proj_b: 2,560 → 48, one write strength per value head (linear_num_value_heads).proj b→ 48in_proj_a: 2,560 → 48, one decay logit per value head (linear_num_value_heads).proj a→ 48in_proj_z: 2,560 → 48·128 = 6,144, the output gate's input.proj z→ 6,144Causal depthwise convolution over the 10,240 qkv channels, kernel 4 (linear_conv_kernel_dim), then SiLU.conv, k = 4then SiLUCarried between tokens: the convolution's last 3 inputs, 3 × 10,240 channels × 2 B = 60 KiB per layer.last3 inputsβ_t = sigmoid(b), in (0, 1): write strength, one per value head (48).β = σ(b)48 headslog α_t = −exp(A_log) · softplus(a + dt_bias), Mamba-2's form; A_log and dt_bias learned per value head (48).decay αMamba-2 formq_t: 16 key heads × 128 (linear_num_key_heads × linear_key_head_dim); each serves 3 value heads.q16×128k_t: 16 key heads × 128; each serves 3 value heads.k16×128v_t: 48 value heads × 128 (linear_num_value_heads × linear_value_head_dim).v48×128q and k are L2-normalized per head (use_qk_l2norm_in_kernel), which keeps the delta rule's factor I − β k kᵀ contracting.L2Norm q, kCarried between tokens: the state, 48 value heads × 128 × 128 × 4 B (FP32, mamba_ssm_dtype) = 3 MiB per layer. Read-out o_t = S_tᵀ q_t.St = αt (I − βt kt ktᵀ) St−1 + βt kt vtᵀ48 heads × 128 × 128, FP323 MiB per layerot = Stᵀ qt: 48×128Gated RMSNorm over each head's 128 outputs: weight · x̂, weight initialised to 1 (in code; not the zero-centred 1 + w form of the other norms).RMSNorm per headweight · x̂z: 6,144Output gate: RMSNorm(o) ⊙ sigmoid(z) (output_gate_type: "sigmoid"). SiLU in Qwen3.5/3.6, sigmoid in Qwen3.8; a sigmoid only scales a channel down.⊙ sigmoid(z)SiLU in Qwen3.5/3.6,sigmoid in Qwen3.8out_proj: 48·128 = 6,144 → 2,560 (hidden_size).out_proj → 2,560

carried to the next token (cached state)querykeyvalue

Qwen3.5-397B: carried per GDN layer = state 4 MiB (64 × 128 × 128 × 4 B) + convolution buffer 72 KiB (3 × 12,288 × 2 B). Nothing else survives the token.

Hover or focus a block for its config field and tensor shape.

Figure 7.One Gated DeltaNet layer as Qwen builds it. The hatched boxes are all it carries from one token to the next.

GDN is three of every four layers in all three Qwen models (full_attention_interval: 4). In Qwen3.8 the fourth is Qwen Sparse Attention (QSA, T3): an indexer scores blocks of 4 tokens, and each query attends to at most 2,048 selected tokens plus the last incomplete block (indexer_budget: 2048), while keys and values are still cached for every token. In the report’s ablation (a mixture of experts with 3B of 25B parameters active, trained up to 32K context), the GDN hybrid beat one with 128-token SWA layers, sink not stated, on seven of nine short-context benchmarks, results that “do not by themselves isolate which architectural component causes each improvement.”

Kimi Delta Attention

KDA, from Kimi Linear (Kimi Team, 2025), gives each key channel its own decay, as GLA did for additive linear attention:

St=(I−βtktkt⊤) Diag(αt) St−1+βt ktvt⊤,αt∈(0,1)dk.S_t = \big(I - \beta_t k_t k_t^\top\big)\,\mathrm{Diag}(\alpha_t)\, S_{t-1} + \beta_t\, k_t v_t^\top, \qquad \alpha_t \in (0,1)^{d_k}.

Each row of the state fades at its own rate, giving one head several time scales. The decay logits utu_t come from a rank-128 projection of the input. With Γt=∏r≤tαr∈(0,1]dk\Gamma_t = \prod_{r \le t}\alpha_r \in (0,1]^{d_k} counted from a reference position, key ss enters the read at tt with weight

qt⊤Diag(Γt/Γs) ks=(qt⊙Γt)⊤(ks⊘Γs),q_t^\top \mathrm{Diag}(\Gamma_t / \Gamma_s)\, k_s = (q_t \odot \Gamma_t)^\top (k_s \oslash \Gamma_s),

so rescaling queries by Γ\Gamma and keys by 1/Γ1/\Gamma restores one matrix product, but 1/Γ1/\Gamma can overflow. Kimi Linear splits each 64-token chunk into 16-token tiles. Below the diagonal the reference can sit at the tile boundary, keeping every factor at most 1; diagonal tiles need factors above 1 and ran position by position, “the main intra-chunk bottleneck” in the Kimi K3 report’s words.

K3 bounds the decay. In place of Kimi Linear’s unbounded log-decay ℓt=−eAlog⁡ softplus(ut)\ell_t = -e^{A_{\log}}\,\mathrm{softplus}(u_t) it uses

ℓt=ℓmin⁡ σ(eAlog⁡ut)∈(ℓmin⁡,0),αt=eℓt,\ell_t = \ell_{\min}\, \sigma\big(e^{A_{\log}} u_t\big) \in (\ell_{\min}, 0), \qquad \alpha_t = e^{\ell_t},

with ℓmin⁡=−5\ell_{\min} = -5 (gate_lower_bound: -5.0), so over a tile the cumulative log-decay stays above −80-80. The condition is 16 ∣ℓmin⁡∣<ln⁡(BF16 max)≈88.716\,|\ell_{\min}| < \ln(\text{BF16 max}) \approx 88.7, met up to ∣ℓmin⁡∣=5.54|\ell_{\min}| = 5.54, which leaves a factor of about 6,000 for the size of qq and kk (computed). Every tile can then use “dense Tensor Core matrix multiplications”, and decay alone removes at most a factor e−5e^{-5} per step (the delta term still overwrites at once).

  • Kimi K3: ℓ = ℓmin·sigmoid(u), bounded
  • Kimi Linear: ℓ = −softplus(u), no lower bound
0−2−4−6−8−10−50510gate logit ulog-decay ℓ per stepℓmin = −5no lower bound020406080100120140ln of rescaling factorunbounded80ln BF16 max = 88.7LinearK316-token tile

16 × |ℓ_min| = 16 × 5.00 = 80 ≤ 88.72 = ln(BF16 max): the largest factor e^80 fits, with headroom e^8.7 ≈ 6,118; the smallest, e^−80, stays above the smallest normal e^−87.3.

Both curves with Alog = 0. Off-diagonal tiles: every factor ≤ 1 with the reference at the tile boundary, so only the diagonal tiles need the bound.

Figure 8.Kimi K3's lower-bounded decay. A floor on decay keeps the chunkwise kernel's rescaling factors finite in BF16, so no tile needs a slow position-by-position path.

K3 also made the sigmoid output gate full-rank (use_full_rank_gate: true); the decay projection stays low-rank. Its 69 KDA and 24 Gated MLA layers (kda_layers) run three to one plus a final MLA layer, and the MLA layers apply no positional encoding, since the decay already weights each key by its distance (T4). GLM-5.3-Flash has 34 KDA and 11 sparse MLA layers (layer_types), the same bound and a low-rank output gate. The gated-deltanet check counts both as GDN.

The recall limit of a fixed state

We ran a toy test without training (computed). It writes mm key→value pairs once, keys and values Gaussian vectors normalized to the unit sphere in dk=dv=64d_k = d_v = 64, then queries each stored key; a recall is a read-out with cosine above 0.9 to the right value. One-pass least squares (RLS) recalls every pair up to m=64m = 64, 49% at 80 and none at 128. The rules in the models stay far below: at m=16m = 16 the additive state recalls 58% and the delta rule 78%, at m=32m = 32, 3% and 33%.

Counted in pairs, each one-pass rule holds a roughly fixed number whatever mm is: 8 to 12 for the delta rule, about 10 for gated delta, 6 for decay at 0.95, none for the additive state from m=64m = 64. They behave like soft windows. At m=256m = 256 the delta rule recalls each of its five newest pairs at least 90% of the time, drops below half at the ninth newest and recalls nothing older than the sixteenth; as a heuristic, each later write removes about a 1/dk1/d_k share of an older pair’s component. Softmax attention recalls all 256. Rewriting 16 of 32 keys, the delta rule returns the new value for 52%, since later writes disturb each exact rewrite.

On multi-query associative recall (MQAR), where trained models learn their own keys, fixed-state models fall behind attention as the number of pairs grows (Arora et al., 2023). Copying meets a separate bound, on the bits a state holds (Jelassi et al., 2024).

  • full attention
  • SWA, W = 32
  • least squares (RLS) = pseudoinverse oracle
  • delta, β = 1
  • gated delta, α = 0.99
  • decayed, α = 0.95
  • additive
0%25%50%75%100%8163264128256pairs stored m (log scale)fraction recalledA · fraction of pairs recalled (cosine > 0.9)m = dk = 64: no linear read-outholds more arbitrary pairsfull attention, m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 48: 100% recalled (48 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 64: 100% recalled (64 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 80: 100% recalled (80 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 96: 100% recalled (96 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 128: 100% recalled (128 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 192: 100% recalled (192 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 256: 100% recalled (256 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 48: 66.7% recalled (32 pairs), ± 0%; mean cosine 0.667 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 64: 50% recalled (32 pairs), ± 0%; mean cosine 0.500 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 80: 40% recalled (32 pairs), ± 0%; mean cosine 0.400 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 96: 33.3% recalled (32 pairs), ± 0%; mean cosine 0.333 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 128: 25% recalled (32 pairs), ± 0%; mean cosine 0.250 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 192: 16.7% recalled (32 pairs), ± 0%; mean cosine 0.167 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 256: 12.5% recalled (32 pairs), ± 0%; mean cosine 0.125 ± 0.000 (20 trials, ± 1 standard error).least squares (RLS), m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 48: 100% recalled (48 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 64: 100% recalled (64 pairs), ± 0%; mean cosine 0.996 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 80: 49.4% recalled (39.5 pairs), ± 1.3%; mean cosine 0.895 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 49.4%, mean cosine 0.895.least squares (RLS), m = 96: 2% recalled (1.9 pairs), ± 0.3%; mean cosine 0.817 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 2%, mean cosine 0.817.least squares (RLS), m = 128: 0% recalled (0 pairs), ± 0%; mean cosine 0.707 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.707.least squares (RLS), m = 192: 0% recalled (0 pairs), ± 0%; mean cosine 0.576 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.576.least squares (RLS), m = 256: 0% recalled (0 pairs), ± 0%; mean cosine 0.498 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.498.delta, β = 1, m = 8: 97.5% recalled (7.8 pairs), ± 1.1%; mean cosine 0.970 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 16: 77.8% recalled (12.4 pairs), ± 1.2%; mean cosine 0.931 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 24: 49.2% recalled (11.8 pairs), ± 1.6%; mean cosine 0.880 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 32: 33.3% recalled (10.7 pairs), ± 1.2%; mean cosine 0.842 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 48: 22.4% recalled (10.8 pairs), ± 0.5%; mean cosine 0.751 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 64: 15.7% recalled (10.1 pairs), ± 0.3%; mean cosine 0.676 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 80: 12.6% recalled (10.1 pairs), ± 0.5%; mean cosine 0.606 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 96: 9.5% recalled (9.1 pairs), ± 0.4%; mean cosine 0.547 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 128: 6.4% recalled (8.2 pairs), ± 0.3%; mean cosine 0.448 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 192: 4.5% recalled (8.7 pairs), ± 0.1%; mean cosine 0.321 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 256: 3.3% recalled (8.4 pairs), ± 0.2%; mean cosine 0.247 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 8: 97.5% recalled (7.8 pairs), ± 1.1%; mean cosine 0.969 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 16: 72.5% recalled (11.6 pairs), ± 1.8%; mean cosine 0.925 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 24: 46.5% recalled (11.2 pairs), ± 1.4%; mean cosine 0.866 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 32: 31.6% recalled (10.1 pairs), ± 1.1%; mean cosine 0.820 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 48: 22.1% recalled (10.6 pairs), ± 0.5%; mean cosine 0.711 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 64: 15.8% recalled (10.1 pairs), ± 0.3%; mean cosine 0.624 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 80: 13.1% recalled (10.5 pairs), ± 0.4%; mean cosine 0.546 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 96: 10.2% recalled (9.8 pairs), ± 0.4%; mean cosine 0.482 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 128: 7.5% recalled (9.6 pairs), ± 0.2%; mean cosine 0.381 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 192: 5.1% recalled (9.7 pairs), ± 0.2%; mean cosine 0.264 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 256: 4% recalled (10.2 pairs), ± 0.1%; mean cosine 0.198 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 8: 95.6% recalled (7.7 pairs), ± 1.6%; mean cosine 0.949 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 16: 51.6% recalled (8.2 pairs), ± 1.9%; mean cosine 0.887 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 24: 26.7% recalled (6.4 pairs), ± 1.9%; mean cosine 0.810 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 32: 18.3% recalled (5.8 pairs), ± 1.3%; mean cosine 0.740 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 48: 13.3% recalled (6.4 pairs), ± 0.7%; mean cosine 0.599 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 64: 9.1% recalled (5.8 pairs), ± 0.4%; mean cosine 0.489 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 80: 7.9% recalled (6.3 pairs), ± 0.3%; mean cosine 0.407 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 96: 5.7% recalled (5.5 pairs), ± 0.4%; mean cosine 0.343 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 128: 4.1% recalled (5.3 pairs), ± 0.2%; mean cosine 0.257 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 192: 3.2% recalled (6.1 pairs), ± 0.2%; mean cosine 0.175 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 256: 2.3% recalled (5.9 pairs), ± 0.2%; mean cosine 0.128 ± 0.002 (20 trials, ± 1 standard error).additive, m = 8: 98.8% recalled (7.9 pairs), ± 0.9%; mean cosine 0.951 ± 0.002 (20 trials, ± 1 standard error).additive, m = 16: 58.4% recalled (9.3 pairs), ± 2.9%; mean cosine 0.904 ± 0.002 (20 trials, ± 1 standard error).additive, m = 24: 14.8% recalled (3.5 pairs), ± 2%; mean cosine 0.856 ± 0.003 (20 trials, ± 1 standard error).additive, m = 32: 3.4% recalled (1.1 pairs), ± 0.7%; mean cosine 0.822 ± 0.002 (20 trials, ± 1 standard error).additive, m = 48: 0.1% recalled (0.1 pairs), ± 0.1%; mean cosine 0.760 ± 0.002 (20 trials, ± 1 standard error).additive, m = 64: 0% recalled (0 pairs), ± 0%; mean cosine 0.712 ± 0.002 (20 trials, ± 1 standard error).additive, m = 80: 0% recalled (0 pairs), ± 0%; mean cosine 0.673 ± 0.001 (20 trials, ± 1 standard error).additive, m = 96: 0% recalled (0 pairs), ± 0%; mean cosine 0.638 ± 0.001 (20 trials, ± 1 standard error).additive, m = 128: 0% recalled (0 pairs), ± 0%; mean cosine 0.579 ± 0.001 (20 trials, ± 1 standard error).additive, m = 192: 0% recalled (0 pairs), ± 0%; mean cosine 0.501 ± 0.001 (20 trials, ± 1 standard error).additive, m = 256: 0% recalled (0 pairs), ± 0%; mean cosine 0.448 ± 0.001 (20 trials, ± 1 standard error).attentionSWARLSdeltagated deltadecayedadditive0641281922568163264128256pairs stored m (log scale)pairs recalledA · number of pairs recalled (cosine > 0.9)m = dk = 64: no linear read-outholds more arbitrary pairsfull attention, m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 48: 100% recalled (48 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 64: 100% recalled (64 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 80: 100% recalled (80 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 96: 100% recalled (96 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 128: 100% recalled (128 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 192: 100% recalled (192 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).full attention, m = 256: 100% recalled (256 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 48: 66.7% recalled (32 pairs), ± 0%; mean cosine 0.667 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 64: 50% recalled (32 pairs), ± 0%; mean cosine 0.500 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 80: 40% recalled (32 pairs), ± 0%; mean cosine 0.400 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 96: 33.3% recalled (32 pairs), ± 0%; mean cosine 0.333 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 128: 25% recalled (32 pairs), ± 0%; mean cosine 0.250 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 192: 16.7% recalled (32 pairs), ± 0%; mean cosine 0.167 ± 0.000 (20 trials, ± 1 standard error).SWA, W = 32, m = 256: 12.5% recalled (32 pairs), ± 0%; mean cosine 0.125 ± 0.000 (20 trials, ± 1 standard error).least squares (RLS), m = 8: 100% recalled (8 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 16: 100% recalled (16 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 24: 100% recalled (24 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 32: 100% recalled (32 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 48: 100% recalled (48 pairs), ± 0%; mean cosine 1.000 ± 0.000 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 64: 100% recalled (64 pairs), ± 0%; mean cosine 0.996 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 100%, mean cosine 1.000.least squares (RLS), m = 80: 49.4% recalled (39.5 pairs), ± 1.3%; mean cosine 0.895 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 49.4%, mean cosine 0.895.least squares (RLS), m = 96: 2% recalled (1.9 pairs), ± 0.3%; mean cosine 0.817 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 2%, mean cosine 0.817.least squares (RLS), m = 128: 0% recalled (0 pairs), ± 0%; mean cosine 0.707 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.707.least squares (RLS), m = 192: 0% recalled (0 pairs), ± 0%; mean cosine 0.576 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.576.least squares (RLS), m = 256: 0% recalled (0 pairs), ± 0%; mean cosine 0.498 ± 0.001 (20 trials, ± 1 standard error); pseudoinverse oracle: 0%, mean cosine 0.498.delta, β = 1, m = 8: 97.5% recalled (7.8 pairs), ± 1.1%; mean cosine 0.970 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 16: 77.8% recalled (12.4 pairs), ± 1.2%; mean cosine 0.931 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 24: 49.2% recalled (11.8 pairs), ± 1.6%; mean cosine 0.880 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 32: 33.3% recalled (10.7 pairs), ± 1.2%; mean cosine 0.842 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 48: 22.4% recalled (10.8 pairs), ± 0.5%; mean cosine 0.751 ± 0.003 (20 trials, ± 1 standard error).delta, β = 1, m = 64: 15.7% recalled (10.1 pairs), ± 0.3%; mean cosine 0.676 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 80: 12.6% recalled (10.1 pairs), ± 0.5%; mean cosine 0.606 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 96: 9.5% recalled (9.1 pairs), ± 0.4%; mean cosine 0.547 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 128: 6.4% recalled (8.2 pairs), ± 0.3%; mean cosine 0.448 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 192: 4.5% recalled (8.7 pairs), ± 0.1%; mean cosine 0.321 ± 0.002 (20 trials, ± 1 standard error).delta, β = 1, m = 256: 3.3% recalled (8.4 pairs), ± 0.2%; mean cosine 0.247 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 8: 97.5% recalled (7.8 pairs), ± 1.1%; mean cosine 0.969 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 16: 72.5% recalled (11.6 pairs), ± 1.8%; mean cosine 0.925 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 24: 46.5% recalled (11.2 pairs), ± 1.4%; mean cosine 0.866 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 32: 31.6% recalled (10.1 pairs), ± 1.1%; mean cosine 0.820 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 48: 22.1% recalled (10.6 pairs), ± 0.5%; mean cosine 0.711 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 64: 15.8% recalled (10.1 pairs), ± 0.3%; mean cosine 0.624 ± 0.003 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 80: 13.1% recalled (10.5 pairs), ± 0.4%; mean cosine 0.546 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 96: 10.2% recalled (9.8 pairs), ± 0.4%; mean cosine 0.482 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 128: 7.5% recalled (9.6 pairs), ± 0.2%; mean cosine 0.381 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 192: 5.1% recalled (9.7 pairs), ± 0.2%; mean cosine 0.264 ± 0.002 (20 trials, ± 1 standard error).gated delta, α = 0.99, m = 256: 4% recalled (10.2 pairs), ± 0.1%; mean cosine 0.198 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 8: 95.6% recalled (7.7 pairs), ± 1.6%; mean cosine 0.949 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 16: 51.6% recalled (8.2 pairs), ± 1.9%; mean cosine 0.887 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 24: 26.7% recalled (6.4 pairs), ± 1.9%; mean cosine 0.810 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 32: 18.3% recalled (5.8 pairs), ± 1.3%; mean cosine 0.740 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 48: 13.3% recalled (6.4 pairs), ± 0.7%; mean cosine 0.599 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 64: 9.1% recalled (5.8 pairs), ± 0.4%; mean cosine 0.489 ± 0.003 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 80: 7.9% recalled (6.3 pairs), ± 0.3%; mean cosine 0.407 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 96: 5.7% recalled (5.5 pairs), ± 0.4%; mean cosine 0.343 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 128: 4.1% recalled (5.3 pairs), ± 0.2%; mean cosine 0.257 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 192: 3.2% recalled (6.1 pairs), ± 0.2%; mean cosine 0.175 ± 0.002 (20 trials, ± 1 standard error).decayed, α = 0.95, m = 256: 2.3% recalled (5.9 pairs), ± 0.2%; mean cosine 0.128 ± 0.002 (20 trials, ± 1 standard error).additive, m = 8: 98.8% recalled (7.9 pairs), ± 0.9%; mean cosine 0.951 ± 0.002 (20 trials, ± 1 standard error).additive, m = 16: 58.4% recalled (9.3 pairs), ± 2.9%; mean cosine 0.904 ± 0.002 (20 trials, ± 1 standard error).additive, m = 24: 14.8% recalled (3.5 pairs), ± 2%; mean cosine 0.856 ± 0.003 (20 trials, ± 1 standard error).additive, m = 32: 3.4% recalled (1.1 pairs), ± 0.7%; mean cosine 0.822 ± 0.002 (20 trials, ± 1 standard error).additive, m = 48: 0.1% recalled (0.1 pairs), ± 0.1%; mean cosine 0.760 ± 0.002 (20 trials, ± 1 standard error).additive, m = 64: 0% recalled (0 pairs), ± 0%; mean cosine 0.712 ± 0.002 (20 trials, ± 1 standard error).additive, m = 80: 0% recalled (0 pairs), ± 0%; mean cosine 0.673 ± 0.001 (20 trials, ± 1 standard error).additive, m = 96: 0% recalled (0 pairs), ± 0%; mean cosine 0.638 ± 0.001 (20 trials, ± 1 standard error).additive, m = 128: 0% recalled (0 pairs), ± 0%; mean cosine 0.579 ± 0.001 (20 trials, ± 1 standard error).additive, m = 192: 0% recalled (0 pairs), ± 0%; mean cosine 0.501 ± 0.001 (20 trials, ± 1 standard error).additive, m = 256: 0% recalled (0 pairs), ± 0%; mean cosine 0.448 ± 0.001 (20 trials, ± 1 standard error).attentionSWARLSdeltagated deltadecayedadditive0%50%100%0816243240age of the pair (0 = newest; linear)recalledB · recall by age, m = 256full attention, m = 256: the pair written 0 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 1 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 2 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 3 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 4 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 5 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 6 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 7 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 8 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 9 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 10 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 11 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 12 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 13 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 14 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 15 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 16 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 17 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 18 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 19 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 20 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 21 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 22 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 23 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 24 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 25 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 26 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 27 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 28 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 29 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 30 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 31 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 32 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 33 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 34 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 35 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 36 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 37 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 38 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 39 pairs before the end is recalled in 100% of 20 trials.full attention, m = 256: the pair written 40 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 0 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 1 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 2 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 3 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 4 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 5 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 6 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 7 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 8 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 9 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 10 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 11 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 12 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 13 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 14 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 15 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 16 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 17 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 18 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 19 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 20 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 21 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 22 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 23 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 24 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 25 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 26 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 27 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 28 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 29 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 30 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 31 pairs before the end is recalled in 100% of 20 trials.SWA, W = 32, m = 256: the pair written 32 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 33 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 34 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 35 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 36 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 37 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 38 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 39 pairs before the end is recalled in 0% of 20 trials.SWA, W = 32, m = 256: the pair written 40 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 0 pairs before the end is recalled in 100% of 20 trials.delta, β = 1, m = 256: the pair written 1 pairs before the end is recalled in 100% of 20 trials.delta, β = 1, m = 256: the pair written 2 pairs before the end is recalled in 95% of 20 trials.delta, β = 1, m = 256: the pair written 3 pairs before the end is recalled in 100% of 20 trials.delta, β = 1, m = 256: the pair written 4 pairs before the end is recalled in 90% of 20 trials.delta, β = 1, m = 256: the pair written 5 pairs before the end is recalled in 75% of 20 trials.delta, β = 1, m = 256: the pair written 6 pairs before the end is recalled in 55% of 20 trials.delta, β = 1, m = 256: the pair written 7 pairs before the end is recalled in 65% of 20 trials.delta, β = 1, m = 256: the pair written 8 pairs before the end is recalled in 45% of 20 trials.delta, β = 1, m = 256: the pair written 9 pairs before the end is recalled in 40% of 20 trials.delta, β = 1, m = 256: the pair written 10 pairs before the end is recalled in 30% of 20 trials.delta, β = 1, m = 256: the pair written 11 pairs before the end is recalled in 10% of 20 trials.delta, β = 1, m = 256: the pair written 12 pairs before the end is recalled in 15% of 20 trials.delta, β = 1, m = 256: the pair written 13 pairs before the end is recalled in 15% of 20 trials.delta, β = 1, m = 256: the pair written 14 pairs before the end is recalled in 5% of 20 trials.delta, β = 1, m = 256: the pair written 15 pairs before the end is recalled in 5% of 20 trials.delta, β = 1, m = 256: the pair written 16 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 17 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 18 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 19 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 20 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 21 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 22 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 23 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 24 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 25 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 26 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 27 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 28 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 29 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 30 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 31 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 32 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 33 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 34 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 35 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 36 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 37 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 38 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 39 pairs before the end is recalled in 0% of 20 trials.delta, β = 1, m = 256: the pair written 40 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 0 pairs before the end is recalled in 100% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 1 pairs before the end is recalled in 100% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 2 pairs before the end is recalled in 95% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 3 pairs before the end is recalled in 100% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 4 pairs before the end is recalled in 95% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 5 pairs before the end is recalled in 85% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 6 pairs before the end is recalled in 70% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 7 pairs before the end is recalled in 95% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 8 pairs before the end is recalled in 50% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 9 pairs before the end is recalled in 55% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 10 pairs before the end is recalled in 50% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 11 pairs before the end is recalled in 35% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 12 pairs before the end is recalled in 30% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 13 pairs before the end is recalled in 30% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 14 pairs before the end is recalled in 10% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 15 pairs before the end is recalled in 15% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 16 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 17 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 18 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 19 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 20 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 21 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 22 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 23 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 24 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 25 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 26 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 27 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 28 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 29 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 30 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 31 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 32 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 33 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 34 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 35 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 36 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 37 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 38 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 39 pairs before the end is recalled in 0% of 20 trials.gated delta, α = 0.99, m = 256: the pair written 40 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 0 pairs before the end is recalled in 85% of 20 trials.decayed, α = 0.95, m = 256: the pair written 1 pairs before the end is recalled in 95% of 20 trials.decayed, α = 0.95, m = 256: the pair written 2 pairs before the end is recalled in 80% of 20 trials.decayed, α = 0.95, m = 256: the pair written 3 pairs before the end is recalled in 65% of 20 trials.decayed, α = 0.95, m = 256: the pair written 4 pairs before the end is recalled in 65% of 20 trials.decayed, α = 0.95, m = 256: the pair written 5 pairs before the end is recalled in 55% of 20 trials.decayed, α = 0.95, m = 256: the pair written 6 pairs before the end is recalled in 45% of 20 trials.decayed, α = 0.95, m = 256: the pair written 7 pairs before the end is recalled in 50% of 20 trials.decayed, α = 0.95, m = 256: the pair written 8 pairs before the end is recalled in 20% of 20 trials.decayed, α = 0.95, m = 256: the pair written 9 pairs before the end is recalled in 15% of 20 trials.decayed, α = 0.95, m = 256: the pair written 10 pairs before the end is recalled in 5% of 20 trials.decayed, α = 0.95, m = 256: the pair written 11 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 12 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 13 pairs before the end is recalled in 10% of 20 trials.decayed, α = 0.95, m = 256: the pair written 14 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 15 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 16 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 17 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 18 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 19 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 20 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 21 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 22 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 23 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 24 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 25 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 26 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 27 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 28 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 29 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 30 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 31 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 32 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 33 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 34 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 35 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 36 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 37 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 38 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 39 pairs before the end is recalled in 0% of 20 trials.decayed, α = 0.95, m = 256: the pair written 40 pairs before the end is recalled in 0% of 20 trials.attentionSWAdeltagated deltadecayed

Hover a point for the mean cosine and its standard error.

Panel B leaves out least squares (RLS) and additive: at m = 256 they recall nothing at every age. Bands: ± 1 standard error over 20 trials. A toy: random unit keys and values in dk = dv = 64, one pass, fixed write rules, no training; not a measurement of any model.

Figure 9.A toy recall test with random keys and no training: write m key→value pairs once, then query each key. The write rules used in the models keep only their most recent pairs; least squares holds exactly d_k; attention holds all of them.

The mapped models handle recall in the stack, with one to six fixed-state layers per full-attention layer. Qwen3.8-Next keeps one in four because full attention “retains the direct token-level retrieval that no finite-state memory reproduces exactly”; in the shipped model those layers are QSA. T5 covers the ratio.

What a fixed state does to the serving stack

Decode is memory-bound: each byte read feeds only a few operations, while an H100 can do about 295 in the time it reads one (989 TFLOP/s against 3.35 TB/s; T1). A Qwen3.5-397B full-attention layer uses each key and value for h/g=32/2=16h/g = 32/2 = 16 query heads, about 16 FLOP per byte. A GDN layer reads and writes back its 4 MiB FP32 state per token, about 7dkdv7 d_k d_v operations on 8dkdv8 d_k d_v bytes per head: under one FLOP per byte, with constant traffic (computed). By its report, Nemotron 3 Ultra leads Qwen3.5-397B on a decode-heavy workload (large batch, no speculative decoding) “despite trailing on prefill”, because Mamba-2’s decode cost per step is constant.

The state is updated in place at every token and kept in FP32: in the Qwen and Nemotron configs, by a cast in GLM-5.3-Flash’s code, and by assumption for Kimi K3, whose code keeps what its FLA kernel returns. Nemotron 3 Ultra’s 32-bit state is larger than its FP8 KV cache “up to 64K” tokens, by its report (384 MiB against 6 KiB per token; computed), and its release stores FP16 with stochastic rounding, which rounds up or down at random in proportion to the remainder so that small updates survive on average.

Tensor parallelism (TP), splitting each layer’s heads over GPUs, splits a state by head, while a GQA cache with g=2g = 2 splits over at most two GPUs and an MLA latent is copied to all. At TP = 8 the per-GPU crossover falls from 6,252 to 1,563 tokens for Qwen3.5-397B and from 16,253 to 2,032 for Kimi K3 (computed).

Prefix caching reuses the cache of a prefix that many requests share. A KV cache can be cut at any token; a state exists only where it was saved, and kernels can save it only at chunk ends. Kimi K3 hashes prefixes in fine blocks (512 tokens in its example) and keeps KDA checkpoints of 429 MiB at a sparse subset of block ends, mostly turn boundaries. DeepSeek-V4 can also store no SWA entries and recompute the last L⋅WL \cdot W tokens, the span a relay can reach. In speculative decoding, rejected draft tokens have already entered the state, the convolution buffer and any SWA ring buffer: Nemotron 3 Ultra snapshots its state at every draft step, and Kimi K3 caches the drafts’ projected inputs and replays the accepted ones in one fused kernel.

In packed training the state and the convolution must restart at every document boundary (HF passes cu_seqlens; a plain nn.Conv1d fallback leaks across documents). Training and decode run different kernels, often in different dtypes, so trainer and server log-probabilities differ.

Prefill gains the most arithmetic: over a 128K prompt a Qwen3.5-397B GDN layer’s mixing core costs about 13 MFLOP per token, a full-attention layer 2.1 GFLOP (num_attention_heads: 32). With the 15 full layers and the projections, the model needs about 67 GFLOP per token against 163 for an all-full stack (computed).

Comparison

Full attentionSWALinear attentionDecayed (GLA, Mamba-2)Delta ruleGated DeltaNetKDA
State writeappend (k,v)(k, v)keep last WW (k,v)(k, v)S+kv⊤S + k v^\topαS\alpha S or Diag(α)S\mathrm{Diag}(\alpha) S, + kv⊤+\, k v^\top(I−βkk⊤)S+βkv⊤(I - \beta k k^\top) S + \beta k v^\topα(I−βkk⊤)S+βkv⊤\alpha (I - \beta k k^\top) S + \beta k v^\top(I−βkk⊤)Diag(α)S+βkv⊤(I - \beta k k^\top)\mathrm{Diag}(\alpha) S + \beta k v^\top
TransitionnonenoneIIdiagonal, commutinggeneralized Householder, non-commutingscalar × HouseholderHouseholder × diagonal
Forgettingnonehard cut at WWnoneper head or per channelby overwritingper head + overwritingper channel + overwriting
Memory per layer (computed)2Tgdhb2 T g d_h b; Qwen3.5 256 MiB at 128K2Wgdhb2 W g d_h b; gpt-oss 256 KiBhsdkdvbh_s d_k d_v bsame; Nemotron 3 Ultra 8 MiBsamesame; Qwen3.5-397B 4 MiBsame; Kimi K3 6 MiB
Rank of past→future blockup to min⁡(τ,T−τ)\min(\tau, T - \tau)≤W−1\le W - 1≤dk\le d_k per key head≤dk\le d_k per key head≤dk\le d_k per key head≤dk\le d_k per key head≤dk\le d_k per key head
Exact recall in one passeverythingeverything within WWorthonormal keys only; crosstalk grows with mmrecent pairsnewest pairs; older ones disturbed by later writesas delta, plus resetsas delta, several time scales
Main costmemory and prefill ∝T2\propto T^2nothing beyond WWcrosstalkforgets what matteredharder to parallelize (UT transform)kernel complexitynumerics; needs a bound for tensor-core tiles
In the mapglobal layers of every hybrid9 models + Kolibri-1; DeepSeek-V4 as a branchnoneNemotron 3 Ultra, 3.5 Lightningnone pureQwen3.5, 3.6, 3.8Kimi K3, GLM-5.3-Flash

hsh_s: state heads (value heads for GDN); b=4b = 4 (FP32), assumed for Kimi K3. A Householder matrix I−2kk⊤I - 2kk^\top (unit kk) is a reflection; I−βkk⊤I - \beta kk^\top generalizes it.

Where the models in the map sit

The table adds, per sequence, the fixed-state layers’ memory and the global layers’ KV cache, from each config.json: FP32 states with convolution buffers, BF16 KV, no MTP layers and no index caches (T3; Qwen3.8’s indexer adds 3 KiB per token). The per-GPU crossover assumes TP = 8. A † marks a 1M value beyond the stated context (config, or the 1M evaluations in the Nemotron 3 Ultra and Qwen3.8 reports). The table compares memory only; T5 compares quality.

ModelFixed-state layersGlobal layersFixed state (MiB)Global KV (KiB/token)Crossover (tokens)Crossover per GPU, TP = 8128K (GiB)1M (GiB)All global, 128K (GiB)
Nemotron 3 Ultra48 Mamba-212 GQA389.112.033,2008,3001.8812.47.5
Nemotron 3.5 Lightning23 Mamba-26 GQA46.86.07,9891,9970.806.0 †3.6
Kimi K369 KDA24 MLA428.627.016,2532,0323.7927.413.1
GLM-5.3-Flash34 KDA11 MLA (sparse)140.811.013,1051,6381.5111.15.6
Qwen3.5-397B45 GDN15 GQA183.230.06,2521,5633.9330.2 †15.0
Qwen3.8-Flash36 GDN12 QSA110.124.04,6981,1743.1124.112.0
Qwen3.6-35B30 GDN10 GQA61.420.03,1447862.5620.1 †10.0
Gemma 4 31B50 SWA, WW = 1,02410 K=V800.040.020,48010,2405.7840.8 †105.0
Inkling55 SWA, WW = 51211 GQA220.044.05,1205,1205.7144.260.5
Laguna-S-2.136 SWA, WW = 51212 GQA72.048.01,5361,5366.0748.124.0
Step-3.7-Flash33 SWA, WW = 51212 GQA66.048.01,4081,4086.0648.1 †22.5
MiMo-V2.5-Pro, V2.6-Pro60 SWA, WW = 12810 GQA37.550.07687686.2950.043.8
MiMo-V2.5, V2.6-Flash39 SWA, WW = 1289 GQA24.422.51,1095552.8422.527.2
gpt-oss-120b18 SWA, WW = 12818 GQA4.536.01281284.5036.0 †9.0

At 128K the fixed state is at most a fifth of any total (Nemotron 3 Ultra). The DeepSeek-V4 family is left out: its SWA branch adds 128 KiB per layer in BF16, 72 KiB in DeepSeek’s FP8/BF16 format (computed), and its other caches belong to T3.

  • fixed state
  • global KV: keys
  • values (MLA: latent)
  • all layers global
  • hatched = cached state · † beyond stated context
02468Mamba-2 hybridsNemotron 3 Ultra: 389.1 MiB fixed + 12 KiB per token × 131,072 = 1.88 GiB; fixed state is 20.2% of it; crossover 33,200 tokens.Nemotron 3 UltraNemotron 3 Ultra, fixed state (48 Mamba-2): 48 layers × (state 256 × 64 × 128 × 4 B + conv (256·64 + 2·8·128) × 3 × 2 B) = 389.1 MiB (mamba_num_heads, mamba_head_dim, ssm_state_size, n_groups, conv_kernel).Nemotron 3 Ultra, global-layer keys and values (12 GQA): 12 layers × 2 × 2 × 128 × 2 B = 12 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 1.50 GiB.Nemotron 3 Ultra, global-layer keys and values (12 GQA): 12 layers × 2 × 2 × 128 × 2 B = 12 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 1.50 GiB.Nemotron 3 Ultra, as if all layers were global: 60 layers × 2 × 2 × 128 × 2 B = 7.5 GiB at 128K.Nemotron 3 Ultra, as if all layers were global: 60 layers × 2 × 2 × 128 × 2 B = 7.5 GiB at 128K.1.88Nemotron 3.5 Lightning: 46.8 MiB fixed + 6 KiB per token × 131,072 = 0.80 GiB; fixed state is 5.7% of it; crossover 7,989 tokens.Nemotron 3.5 LightningNemotron 3.5 Lightning, fixed state (23 Mamba-2): 23 layers × (state 64 × 64 × 128 × 4 B + conv (64·64 + 2·8·128) × 3 × 2 B) = 46.8 MiB (mamba_num_heads, mamba_head_dim, ssm_state_size, n_groups, conv_kernel).Nemotron 3.5 Lightning, global-layer keys and values (6 GQA): 6 layers × 2 × 2 × 128 × 2 B = 6 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 0.75 GiB.Nemotron 3.5 Lightning, global-layer keys and values (6 GQA): 6 layers × 2 × 2 × 128 × 2 B = 6 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 0.75 GiB.Nemotron 3.5 Lightning, as if all layers were global: 29 layers × 2 × 2 × 128 × 2 B = 3.6 GiB at 128K.Nemotron 3.5 Lightning, as if all layers were global: 29 layers × 2 × 2 × 128 × 2 B = 3.6 GiB at 128K.0.80Kimi Delta Attention hybridsKimi K3: 428.6 MiB fixed + 27 KiB per token × 131,072 = 3.79 GiB; fixed state is 11% of it; crossover 16,253 tokens.Kimi K3Kimi K3, fixed state (69 KDA): 69 layers × (state 96 × 128 × 128 × 4 B + conv 3 × 96 × 128 × 3 × 2 B) = 428.6 MiB (linear_attn_config.num_heads, head_dim, short_conv_kernel_size).Kimi K3, global-layer RoPE key and latent (24 MLA): 24 layers × (512 + 64) × 2 B = 27 KiB per token (kv_lora_rank, qk_rope_head_dim) → × 131,072 tokens = 3.38 GiB.Kimi K3, global-layer RoPE key and latent (24 MLA): 24 layers × (512 + 64) × 2 B = 27 KiB per token (kv_lora_rank, qk_rope_head_dim) → × 131,072 tokens = 3.38 GiB.Kimi K3, as if all layers were global: 93 layers × (512 + 64) × 2 B = 13.1 GiB at 128K (off the axis).▸ 13.1Kimi K3, as if all layers were global: 93 layers × (512 + 64) × 2 B = 13.1 GiB at 128K (off the axis).3.79GLM-5.3-Flash: 140.8 MiB fixed + 11 KiB per token × 131,072 = 1.51 GiB; fixed state is 9.1% of it; crossover 13,105 tokens.GLM-5.3-FlashGLM-5.3-Flash, fixed state (34 KDA): 34 layers × (state 64 × 128 × 128 × 4 B + conv 3 × 64 × 128 × 3 × 2 B) = 140.8 MiB (linear_attn_config.num_heads, head_dim, short_conv_kernel_size).GLM-5.3-Flash, global-layer RoPE key and latent (11 MLA (sparse)): 11 layers × (512 + 0) × 2 B = 11 KiB per token (kv_lora_rank, qk_rope_head_dim) → × 131,072 tokens = 1.38 GiB.GLM-5.3-Flash, global-layer RoPE key and latent (11 MLA (sparse)): 11 layers × (512 + 0) × 2 B = 11 KiB per token (kv_lora_rank, qk_rope_head_dim) → × 131,072 tokens = 1.38 GiB.GLM-5.3-Flash, as if all layers were global: 45 layers × (512 + 0) × 2 B = 5.6 GiB at 128K.GLM-5.3-Flash, as if all layers were global: 45 layers × (512 + 0) × 2 B = 5.6 GiB at 128K.1.51Gated DeltaNet hybridsQwen3.5-397B: 183.2 MiB fixed + 30 KiB per token × 131,072 = 3.93 GiB; fixed state is 4.6% of it; crossover 6,252 tokens.Qwen3.5-397BQwen3.5-397B, fixed state (45 GDN): 45 layers × (state 64 × 128 × 128 × 4 B + conv (2·16·128 + 64·128) × 3 × 2 B) = 183.2 MiB (linear_num_value_heads, linear_key_head_dim, linear_value_head_dim, linear_num_key_heads, linear_conv_kernel_dim).Qwen3.5-397B, global-layer keys and values (15 GQA): 15 layers × 2 × 2 × 256 × 2 B = 30 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 3.75 GiB.Qwen3.5-397B, global-layer keys and values (15 GQA): 15 layers × 2 × 2 × 256 × 2 B = 30 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 3.75 GiB.Qwen3.5-397B, as if all layers were global: 60 layers × 2 × 2 × 256 × 2 B = 15 GiB at 128K (off the axis).▸ 15Qwen3.5-397B, as if all layers were global: 60 layers × 2 × 2 × 256 × 2 B = 15 GiB at 128K (off the axis).3.93Qwen3.8-Flash: 110.1 MiB fixed + 24 KiB per token × 131,072 = 3.11 GiB; fixed state is 3.5% of it; crossover 4,698 tokens.Qwen3.8-FlashQwen3.8-Flash, fixed state (36 GDN): 36 layers × (state 48 × 128 × 128 × 4 B + conv (2·16·128 + 48·128) × 3 × 2 B) = 110.1 MiB (linear_num_value_heads, linear_key_head_dim, linear_value_head_dim, linear_num_key_heads, linear_conv_kernel_dim).Qwen3.8-Flash, global-layer keys and values (12 QSA): 12 layers × 2 × 2 × 256 × 2 B = 24 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 3.00 GiB.Qwen3.8-Flash, global-layer keys and values (12 QSA): 12 layers × 2 × 2 × 256 × 2 B = 24 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 3.00 GiB.Qwen3.8-Flash, as if all layers were global: 48 layers × 2 × 2 × 256 × 2 B = 12 GiB at 128K (off the axis).▸ 12Qwen3.8-Flash, as if all layers were global: 48 layers × 2 × 2 × 256 × 2 B = 12 GiB at 128K (off the axis).3.11Qwen3.6-35B: 61.4 MiB fixed + 20 KiB per token × 131,072 = 2.56 GiB; fixed state is 2.3% of it; crossover 3,144 tokens.Qwen3.6-35BQwen3.6-35B, fixed state (30 GDN): 30 layers × (state 32 × 128 × 128 × 4 B + conv (2·16·128 + 32·128) × 3 × 2 B) = 61.4 MiB (linear_num_value_heads, linear_key_head_dim, linear_value_head_dim, linear_num_key_heads, linear_conv_kernel_dim).Qwen3.6-35B, global-layer keys and values (10 GQA): 10 layers × 2 × 2 × 256 × 2 B = 20 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 2.50 GiB.Qwen3.6-35B, global-layer keys and values (10 GQA): 10 layers × 2 × 2 × 256 × 2 B = 20 KiB per token (num_key_value_heads, head_dim) → × 131,072 tokens = 2.50 GiB.Qwen3.6-35B, as if all layers were global: 40 layers × 2 × 2 × 256 × 2 B = 10 GiB at 128K (off the axis).▸ 10Qwen3.6-35B, as if all layers were global: 40 layers × 2 × 2 × 256 × 2 B = 10 GiB at 128K (off the axis).2.56sliding-window hybridsGemma 4 31B: 800 MiB fixed + 40 KiB per token × 131,072 = 5.78 GiB; fixed state is 13.5% of it; crossover 20,480 tokens.Gemma 4 31BGemma 4 31B, fixed state (50 SWA, W = 1,024): 50 layers × W 1024 × 2 × 16 × 256 × 2 B = 800 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).Gemma 4 31B, global-layer K=V (10 K=V): 10 layers × 4 × 512 × 2 B = 40 KiB per token (K=V: num_global_key_value_heads, global_head_dim; one tensor is key and value) → × 131,072 tokens = 5.00 GiB.Gemma 4 31B, global-layer K=V (10 K=V): 10 layers × 4 × 512 × 2 B = 40 KiB per token (K=V: num_global_key_value_heads, global_head_dim; one tensor is key and value) → × 131,072 tokens = 5.00 GiB.Gemma 4 31B, as if all layers were global: 50 sliding layers × 2 × 16 × 256 × 2 B + the global layers' KV = 105 GiB at 128K (off the axis).▸ 105Gemma 4 31B, as if all layers were global: 50 sliding layers × 2 × 16 × 256 × 2 B + the global layers' KV = 105 GiB at 128K (off the axis).5.78Inkling: 220 MiB fixed + 44 KiB per token × 131,072 = 5.71 GiB; fixed state is 3.8% of it; crossover 5,120 tokens.InklingInkling, fixed state (55 SWA, W = 512): 55 layers × W 512 × 2 × 16 × 128 × 2 B = 220 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).Inkling, global-layer keys and values (11 GQA): 11 layers × 2 × 8 × 128 × 2 B = 44 KiB per token → × 131,072 tokens = 5.50 GiB.Inkling, global-layer keys and values (11 GQA): 11 layers × 2 × 8 × 128 × 2 B = 44 KiB per token → × 131,072 tokens = 5.50 GiB.Inkling, as if all layers were global: 55 sliding layers × 2 × 16 × 128 × 2 B + the global layers' KV = 60.5 GiB at 128K (off the axis).▸ 60.5Inkling, as if all layers were global: 55 sliding layers × 2 × 16 × 128 × 2 B + the global layers' KV = 60.5 GiB at 128K (off the axis).5.71Laguna-S-2.1: 72 MiB fixed + 48 KiB per token × 131,072 = 6.07 GiB; fixed state is 1.2% of it; crossover 1,536 tokens.Laguna-S-2.1Laguna-S-2.1, fixed state (36 SWA, W = 512): 36 layers × W 512 × 2 × 8 × 128 × 2 B = 72 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).Laguna-S-2.1, global-layer keys and values (12 GQA): 12 layers × 2 × 8 × 128 × 2 B = 48 KiB per token → × 131,072 tokens = 6.00 GiB.Laguna-S-2.1, global-layer keys and values (12 GQA): 12 layers × 2 × 8 × 128 × 2 B = 48 KiB per token → × 131,072 tokens = 6.00 GiB.Laguna-S-2.1, as if all layers were global: 36 sliding layers × 2 × 8 × 128 × 2 B + the global layers' KV = 24 GiB at 128K (off the axis).▸ 24Laguna-S-2.1, as if all layers were global: 36 sliding layers × 2 × 8 × 128 × 2 B + the global layers' KV = 24 GiB at 128K (off the axis).6.07Step-3.7-Flash: 66 MiB fixed + 48 KiB per token × 131,072 = 6.06 GiB; fixed state is 1.1% of it; crossover 1,408 tokens.Step-3.7-FlashStep-3.7-Flash, fixed state (33 SWA, W = 512): 33 layers × W 512 × 2 × 8 × 128 × 2 B = 66 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).Step-3.7-Flash, global-layer keys and values (12 GQA): 12 layers × 2 × 8 × 128 × 2 B = 48 KiB per token → × 131,072 tokens = 6.00 GiB.Step-3.7-Flash, global-layer keys and values (12 GQA): 12 layers × 2 × 8 × 128 × 2 B = 48 KiB per token → × 131,072 tokens = 6.00 GiB.Step-3.7-Flash, as if all layers were global: 33 sliding layers × 2 × 8 × 128 × 2 B + the global layers' KV = 22.5 GiB at 128K (off the axis).▸ 22.5Step-3.7-Flash, as if all layers were global: 33 sliding layers × 2 × 8 × 128 × 2 B + the global layers' KV = 22.5 GiB at 128K (off the axis).6.06MiMo-V2.5-Pro, V2.6-Pro: 37.5 MiB fixed + 50 KiB per token × 131,072 = 6.29 GiB; fixed state is 0.58% of it; crossover 768 tokens.MiMo-V2.5-Pro, V2.6-ProMiMo-V2.5-Pro, V2.6-Pro, fixed state (60 SWA, W = 128): 60 layers × W 128 × 8 × (192 + 128) × 2 B = 37.5 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).MiMo-V2.5-Pro, V2.6-Pro, global-layer keys and values (10 GQA): 10 layers × 8 × (192 + 128) × 2 B = 50 KiB per token → × 131,072 tokens = 6.25 GiB.MiMo-V2.5-Pro, V2.6-Pro, global-layer keys and values (10 GQA): 10 layers × 8 × (192 + 128) × 2 B = 50 KiB per token → × 131,072 tokens = 6.25 GiB.MiMo-V2.5-Pro, V2.6-Pro, as if all layers were global: 60 sliding layers × 8 × (192 + 128) × 2 B + the global layers' KV = 43.8 GiB at 128K (off the axis).▸ 43.8MiMo-V2.5-Pro, V2.6-Pro, as if all layers were global: 60 sliding layers × 8 × (192 + 128) × 2 B + the global layers' KV = 43.8 GiB at 128K (off the axis).6.29MiMo-V2.5, V2.6-Flash: 24.4 MiB fixed + 22.5 KiB per token × 131,072 = 2.84 GiB; fixed state is 0.84% of it; crossover 1,109 tokens.MiMo-V2.5, V2.6-FlashMiMo-V2.5, V2.6-Flash, fixed state (39 SWA, W = 128): 39 layers × W 128 × 8 × (192 + 128) × 2 B = 24.4 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).MiMo-V2.5, V2.6-Flash, global-layer keys and values (9 GQA): 9 layers × 4 × (192 + 128) × 2 B = 22.5 KiB per token → × 131,072 tokens = 2.81 GiB.MiMo-V2.5, V2.6-Flash, global-layer keys and values (9 GQA): 9 layers × 4 × (192 + 128) × 2 B = 22.5 KiB per token → × 131,072 tokens = 2.81 GiB.MiMo-V2.5, V2.6-Flash, as if all layers were global: 39 sliding layers × 8 × (192 + 128) × 2 B + the global layers' KV = 27.2 GiB at 128K (off the axis).▸ 27.2MiMo-V2.5, V2.6-Flash, as if all layers were global: 39 sliding layers × 8 × (192 + 128) × 2 B + the global layers' KV = 27.2 GiB at 128K (off the axis).2.84gpt-oss-120b: 4.5 MiB fixed + 36 KiB per token × 131,072 = 4.50 GiB; fixed state is 0.098% of it; crossover 128 tokens.gpt-oss-120bgpt-oss-120b, fixed state (18 SWA, W = 128): 18 layers × W 128 × 2 × 8 × 64 × 2 B = 4.5 MiB (the window's KV cache; sliding window and the sliding layers' KV heads and head dims).gpt-oss-120b, global-layer keys and values (18 GQA): 18 layers × 2 × 8 × 64 × 2 B = 36 KiB per token → × 131,072 tokens = 4.50 GiB.gpt-oss-120b, global-layer keys and values (18 GQA): 18 layers × 2 × 8 × 64 × 2 B = 36 KiB per token → × 131,072 tokens = 4.50 GiB.gpt-oss-120b, as if all layers were global: 18 sliding layers × 2 × 8 × 64 × 2 B + the global layers' KV = 9 GiB at 128K (off the axis).▸ 9gpt-oss-120b, as if all layers were global: 18 sliding layers × 2 × 8 × 64 × 2 B + the global layers' KV = 9 GiB at 128K (off the axis).4.50GiB per sequence at 128K tokens (linear)

At 128K tokens, BF16 KV (2 B): totals from 0.80 GiB (Nemotron 3.5 Lightning) to 6.29 GiB (MiMo-V2.5-Pro, V2.6-Pro); the fixed state is 0.098–20.2% of each total.

Hover or focus a bar for its formula and config fields.

FP32 states with their convolution buffers; a sliding window's cache counts as fixed state and stays BF16 under the FP8 toggle. No MTP layers, no index caches. A ▸ at the right edge gives an all-global value off the axis, in GiB. Per GPU: state heads split 8 ways, GQA KV heads min(g, 8) ways, the MLA latent not at all.

Figure 10.Memory per sequence for the hybrids in the map: fixed state plus the KV cache of the global layers, BF16 KV, FP32 state, no index caches.

Easy to get wrong

  • “Linear in sequence length” describes prefill. At decode a fixed-state layer costs the same for every token and is still memory-bound, at lower intensity than the GQA layer it replaces.
  • Below the crossover length a fixed state is larger than the KV cache it replaces: up to 64K tokens for Nemotron 3 Ultra against FP8 KV (its report).
  • One pass of the delta rule keeps the newest pairs exact and lets older ones drift. Only a least-squares write such as RLS reaches dkd_k pairs per key head.

Old ideas in new places

Linear attention is the correlation-matrix memory of Anderson, Kohonen and Nakano (1972); the delta rule with unit keys is normalized LMS (Nagumo & Noda, 1967), a Kaczmarz step, and falls short of Kohonen’s optimal linear associative memory, which RLS reaches. With a forgetting factor RLS is a Kalman filter for a drifting state, and GDN its LMS approximation. Data-dependent decay is the LSTM forget gate (Gers et al., 2000), KDA’s a forget-gate vector, Mamba-2’s step size the GRU’s coupled update gate. The rank bound is Ho & Kalman’s (1966) realization theory. Wang, Shi & Fox (2025) treat all these layers as regression solved at test time.

Check yourself

  1. Nemotron 3.5 Lightning has 23 Mamba-2 layers with 64 heads, mamba_head_dim: 64 and ssm_state_size: 128, and 6 attention layers with 2 KV heads of 128. How large is its FP32 state, and at what context does its BF16 KV cache reach that size?
    Answer64 × 64 × 128 × 4 B = 2 MiB per layer, 46 MiB in all. KV: 6 × 2 × 2 × 128 × 2 B = 6 KiB per token, so about 7,850 tokens.
  2. Starting from an empty state, a layer writes key kk (∥k∥=1\|k\| = 1) with value v1v_1, then kk again with v2v_2, at β=1\beta = 1. What does reading with kk return under the additive rule and under the delta rule?
    AnswerAdditive: v1+v2v_1 + v_2. Delta: the second write first removes S⊤k=v1S^\top k = v_1, so v2v_2, until a later key with kj⊤k≠0k_j^\top k \ne 0 is written.
  3. For dk=128d_k = 128 and random unit keys and values, at about what mm does an additive state’s read-out fall below cosine 0.9, and why doesn’t the delta rule hold 128 pairs after one pass?
    Answer(1+(m−1)/128)−1/2>0.9(1 + (m-1)/128)^{-1/2} > 0.9 needs m<1+0.23⋅128≈31m < 1 + 0.23 \cdot 128 \approx 31. Later writes shift older reads by (kj⊤ki)(k_j^\top k_i) times their error, and random keys aren’t orthogonal.

In the map

Methods

Taxonomy nodes

Further reading

↑