Qwen3.5-397B has 60 layers. At the model’s maximum context of 262,144 tokens, each of its 15
full-attention layers stores 512 MiB of keys and values for one conversation. The other 45 store
4 MiB each at any length (computed from the config). They are Gated DeltaNet layers, each keeping
a fixed-size matrix that summarizes the past and is rewritten as each token arrives.
Gated DeltaNet is one of several token mixers (the sublayer that moves information between
positions) whose memory doesn’t grow with the context. Sliding-window attention keeps the last
W tokens; linear attention, Mamba-2, Gated DeltaNet and Kimi Delta Attention (KDA) compress the
prefix into a matrix. Every mapped model that uses them is a hybrid with some
full-attention layers.
The cost these layers remove
A model runs in two phases: prefill processes the prompt, all tokens at once, in large matrix
multiplications, and decode generates one token at a time. A softmax-attention layer keeps the
keys and values of every past token, the KV cache (T1), which grows by 2gdhb bytes per token
for g key/value heads of dimension dh at b bytes per number. Qwen3.5-397B’s full-attention
layers have g=2, dh=256 (num_key_value_heads, head_dim), so in BF16 (b=2) each
grows by 2 KiB per token, 256 MiB at T = 128K (computed). Each decoded token reads all of it,
and prefill attention work grows with T2. A fixed-state layer reads and updates a state S of
constant size once per token, and its prefill work grows with T.
Window W
keysvalues / statehatched = cached state · one layer, one sequence, Qwen3.5-397B shape: BF16 cache, FP32 state
at
128K tokens
full attention
256 MiB
Gated DeltaNet
4 MiB
sliding window
256 KiB
full attention holds 64× the state. Hover or focus a line for its formula; the Gated DeltaNet layer also keeps a 72 KiB convolution buffer (crossover 2,084 with it).
Figure 1.Memory held by one layer for one sequence. The full-attention cache passes the Gated DeltaNet state at about 2,000 tokens and is 512 times larger at a million.
Sliding-window attention
In sliding-window attention (SWA; Beltagy et al., 2020; Jiang et al., 2023) each query attends to
the W most recent positions, its own included, with ordinary softmax. Recall inside the window
is exact; anything older is gone from that layer. The layer holds 2Wgdhb bytes, 256 KiB for
gpt-oss-120b (W=128 in sliding_window, 8 KV heads of 64; computed), and prefill work grows
as T⋅W. A contiguous cache can be a ring buffer, the newest entry overwriting the oldest, with k
spare slots if k speculated tokens may be rejected; paged engines free whole blocks instead.
Older information arrives by relay through positions that picked it up in lower layers: a query
at layer L reaches L(W−1) positions back, which the Mistral 7B paper rounds to “approximately
131K tokens” for 32 layers with W = 4,096. Each hop passes through one position’s hidden state,
which also carries that position’s own content, so the relay is lossy.
queryits windowwithin reachrelay path from the oldest reachable token
W = 4, 4 layers: the query at position 24 reaches 12 positions back (to 12), 4 × (W − 1). Position 13 arrives through 3 intermediate positions (16, 19, 22).
Hover a token to see how many hops it needs.
Figure 2.A query in an SWA stack sees further back at each layer down, through a relay of intermediate tokens. A global layer reaches every position directly, which is why the mapped models keep some (T5).
Windows in the map are 128 tokens (gpt-oss, four MiMo models, DeepSeek-V4’s SWA branch), 512
(Laguna-S-2.1, Step-3.7-Flash, Inkling, Kolibri-1) or 1,024 (Gemma 4). MiMo-V2-Flash found 128 better than 512
in its hybrids up to 32K context, and Kolibri-1 the same up to 64K at one scale, with 512
ahead at 256K. The three labs using 128 add an attention sink (T6; Xiaomi on SWA layers only), a
learned logit per head that absorbs attention mass when no key in the window fits; MiMo-V2-Flash’s hybrid without one “suffers
noticeable performance degradation”. Of the four labs at 512 or more in the code check, two
gate the attention output per head instead. In the site’s code checks (each model’s code tested against a checklist),
sliding-window-attention is present for 15 of 28 models: six DeepSeek-V4 models, whose every
layer has a 128-token window, and nine that interleave whole SWA layers with global ones.
Linear attention
Softmax attention weights vs by exp(qt⊤ks) over a normalizing sum. That exponential
doesn’t split into a function of qt times a function of ks, so the past can’t be summed up
before the query arrives. Katharopoulos et al. (2020) used a feature map ϕ with
exp(q⊤k)≈ϕ(q)⊤ϕ(k) and kept the normalizer
∑sϕ(qt)⊤ϕ(ks); current models use SiLU and L2 normalization as ϕ, drop
the normalizer and normalize each head’s output instead. Then
with St∈Rdk×dv for key width dk and value width dv. Each token
adds an outer product, each query reads with a matrix-vector product, and the state never grows.
How much a dk×dv matrix can hold
Stack m stored keys as the rows of K and their values as the rows of V. Exact recall through
the linear read-out S⊤q means KS=V, which for arbitrary values needs linearly
independent keys, so m≤dk. Beyond that, any S loses at least a share 1−dk/m of random
values’ energy. The bound is per key head, and keys are shared:
every linear layer in the map has dk=128, Qwen3.5-397B’s 64 value heads read 16 key heads,
2,048 key directions per layer, and Nemotron 3 Ultra’s 256 heads share 8 key groups, 1,024
(computed). Extra value heads widen what is stored under a key without adding slots.
Additive writing falls short of the bound. With S=K⊤V, reading with ki returns
S⊤ki=∥ki∥2vi+j=i∑(kj⊤ki)vj,
the right value plus crosstalk, exact only for orthonormal keys. For random unit keys and values
each kj⊤ki has variance 1/dk, so the crosstalk’s expected squared norm is
(m−1)/dk times the signal’s, and the read-out’s cosine with vi is about
(1+(m−1)/dk)−1/2: above 0.9 only for m<1+0.23dk, about 16 at dk=64.
Nothing is ever removed, so the crosstalk keeps growing with m.
One bound for every layer
Every layer here computes O=ΩV, values as rows of V, with a lower-triangular mixing
matrix Ω that depends on queries, keys and gates but not on the values. Writing a
matrix-state update as St=AtSt−1+βtktvt⊤, with a transition At (I
for linear attention) and a write strength βt (1 so far),
Ωts=qt⊤AtAt−1⋯As+1βsks(s≤t).
For a split point τ and t>τ≥s the product splits at τ into a length-dk row
vector that depends on t times a column vector that depends on s, so the past-to-future block
Ω[τ+1:T,1:τ] has rank at most dk. For SWA the block is nonzero only where
t−s≤W−1, rank at most W−1; for softmax attention nothing bounds it below
min(τ,T−τ). Recalling m arbitrary past values at later queries needs rank m there,
so the capacity bound holds for every write rule and gate at once, input by input. Matrices with
low-rank off-diagonal blocks are called semiseparable; Mamba-2’s “state space duality” rests on
them (Dao & Gu, 2024), and the chunkwise algorithm stores these blocks in factored form.
Sliding window, W = 6
Gated delta rule, d_k = 4
Full softmax attention
|Ωts|, 0 to each panel's largest entry (square-root scale); rows are queries t, columns keys s, T = 32. The outlined block is rows after τ, columns up to τ; dots below the dashed line (10−9 × the largest) do not count towards the rank.
Ranks: 5, 4, 16 (window, gated delta, full attention).
Figure 3.The same past-to-future block in three mixers. A window or a fixed state limits its rank, and with it how many independent past values later tokens can read; full attention has no such limit.
Parallel, recurrent and chunkwise
With queries, keys and values as rows of Q,K,V (rows are tokens), linear attention is
O=(QK⊤⊙M)V for the causal 0/1 mask M: O(T2) work in large matrix products. The
recurrent form is O(T) work in T small sequential steps, which suits decoding. Training and
long prompts cut the sequence into chunks of C tokens, with S[n] the state entering chunk
n and MC the C×C causal mask:
O[n]=earlier chunksQ[n]S[n]+inside the chunk(Q[n]K[n]⊤⊙MC)V[n],S[n+1]=S[n]+K[n]⊤V[n].
Every operation is now a matrix product, the work of tensor cores, the GPU units that multiply
small dense matrices (989 TFLOP/s in BF16 on an H100, against 67 for ordinary FP32 arithmetic). Per token the
chunk part costs about C(dk+dv) multiply-adds and the state part 2dkdv, and the
sequential chain shrinks to T/C steps. Qwen’s code and the KDA kernels use C=64, Nemotron
chunk_size: 128.
Parallel
Recurrent
Chunkwise, C = 4
querieskeysvalues / state (hatched = cached)computed as attentioncarried by a chunk state
sequential steps, T / C
16 / 4 = 4
intra-chunk multiply-adds per token, C(dk + dv)
4 × (128 + 128) = 1,024
state multiply-adds per token, 2 dk dv
2 × 128 × 128 = 32,768
C = 4: 4 diagonal blocks computed as small attention matrices, 4 chunk states in sequence.
T = 16 tokens drawn; the counters use dk = dv = 128, as in every linear layer in the map.
Figure 4.The same layer computed in three schedules. The chunk size trades quadratic work inside chunks against the length of the sequential chain between them.
No model in the map uses plain linear attention. Decay limits how much old content piles up, and
the delta rule cancels the crosstalk on the key being written.
Decay
Multiplying the state by a decay αt∈(0,1) before each write,
St=αtSt−1+ktvt⊤, weights token s in the read at t by
∏r=s+1tαr, so the parallel form uses a decay mask Λ with those entries
(0 above the diagonal) in place of M. Old pairs stop crowding new ones, and fade whether or not
they mattered. RetNet (Sun et al., 2023) fixes α per head (a total weight
1/(1−α)). Gated linear attention (GLA; Yang et al., 2023) computes a decay
per key channel from the input, Diag(αt)St−1, and Mamba-2 one per head, so
a value can be held across irrelevant text and dropped at a boundary.
A scalar decay keeps the chunkwise form cheap. With γi=∏r=1iαr the decay
since the chunk start and Γ=diag(γ1,…,γC),
with (Λ[n])ij=γi/γj for j≤i. Every factor is at most 1, so
nothing overflows. A decay per channel breaks this (KDA below).
Hover a cell for its product of decays. Queries after position 12 see little before it: α12 = 0.05.
Figure 5.The decay mask for three ways of choosing α. A dip in a data-dependent decay acts as a reset at a boundary in the text.
Mamba-2
A state-space model (SSM) is a linear system h˙=Ah+Bx, read out as y=Ch, run in
steps of size Δ: the transition becomes exp(ΔA) and the input term, approximately,
ΔB. Mamba (Gu & Dao, 2023) made it selective, computing Δ, B and C from each
token; its decay varies along both dimensions of the state, can’t be folded into a query or key,
and runs as a scan, without matrix products. Mamba-2 (Dao & Gu, 2024) makes the transition a
scalar times the identity per head. With Bt as the key, Ct as the query and the input xt
as the value, a head computes
St=eΔtASt−1+Δtktvt⊤,ot=St⊤qt+Dvt,
with a learned A<0, Δt=softplus(⋅)>0 and a learned skip D: decayed
linear attention with αt=eΔtA and key Δtkt. In Nemotron 3 Ultradk is
ssm_state_size: 128 and dv is mamba_head_dim: 64, over mamba_num_heads: 256.
One step size sets both forgetting and writing: the write weight Δt=ln(1/αt)/∣A∣
grows with how much is forgotten, so a head can’t hold its state and write strongly at once. Keys
and queries are shared by groups of heads (n_groups: 8), Dao & Gu’s multi-value attention. A
kernel-4 causal convolution (conv_kernel) precedes the state, and the output is gated, then
normalized in groups.
The delta rule
Decay frees space by forgetting everything a little. The delta rule corrects one association.
Before writing vt under kt it subtracts what the state already returns for kt:
with a write strength βt∈(0,1) from the input. This is one least-mean-squares
gradient step on 21∥S⊤kt−vt∥2 (Widrow & Hoff, 1960; DeltaNet in Schlag et
al., 2021). With ∥kt∥=1 and βt=1 the state then returns
exactly vt.
That holds at the moment of writing. With unit keys and β=1, writing pair j moves the
read at an older key ki by
Δ(S⊤ki)=(kj⊤ki)(vj−Sj−1⊤kj),
so after one pass pair i is exact for arbitrary values only if every later key is orthogonal to
ki. One pass of the delta rule therefore falls short of least squares and of the dk bound, even
for linearly independent keys. Recursive least squares (RLS) reaches the bound in one pass,
with P the inverse covariance of the keys so far. The delta rule is RLS with P replaced by
βI, which saves a dk×dk matrix per key head and O(dk2) work per token.
The factor’s eigenvalue along kt, 1−βt∥kt∥2, leaves (−1,1) once
βt∥kt∥2>2, so every delta-rule layer in the map L2-normalizes kt (and qt, for
the read-out scale). Diagonal transitions with entries in (0,1], as in linear, decayed and
Mamba-2 layers, commute and can’t track the parity of a bit string at finite precision (Merrill,
Petty & Sabharwal, 2024); delta transitions don’t commute, and with β∈(0,2) they can
(Grazzi et al., 2024). The mapped models keep βt∈(0,1).
old read-out Sᵀk
new value v
correction v − old
read-out after the write
Additive: result = old + v, 1.42 from v (the whole old read-out stays). Delta: result = old + β(v − old), 0.00 from v (β = 1: exactly v).
Drag the end of the “old” arrow (or focus it and use the arrow keys). Toy 2-D vectors.
Figure 6.Writing value v under a key the state already knows. Additive writing adds to what is there; the delta rule moves the read-out towards v by a fraction β of the error.
A chunk’s update is a product of C different factors, which Yang et al. (2024) collapse with the
UT transform. With the chunk’s keys and values as rows of K and V, let
N=I+tril(diag(β)KK⊤,−1) (strictly lower part),
K~=N−1diag(β)K and V~=N−1diag(β)V. Then
with MC including the diagonal: one C×C triangular solve, then matrix products
(torch.linalg.solve_triangular in Qwen’s code; FLA kernels for Kimi and GLM).
Gated DeltaNet
Gated DeltaNet (GDN; Yang, Kautz & Hatamizadeh, 2024) combines the two, with one scalar decay per
head and token, so a head can clear its state at a new topic and still edit single pairs:
St=αt(I−βtktkt⊤)St−1+βtktvt⊤.
In the Qwen code logαt=−eAlogsoftplus(at+b) (Mamba-2’s form,
learned Alog and b), and βt is a sigmoid. Around the recurrence sit a
kernel-4 causal convolution on q,k,v (linear_conv_kernel_dim: 4) and a per-head RMSNorm
followed by an output gate. Qwen3.5 and 3.6 gate with
SiLU, as the original GDN does; Qwen3.8-Flash sets output_gate_type: "sigmoid", and the
Qwen3.8-Next report that describes it says “we use the bounded sigmoid gate … and observe
consistent improvements across our experiments”. A sigmoid can only scale a channel down, which
limits activation outliers (T6).
carried to the next token (cached state)querykeyvalue
Qwen3.5-397B: carried per GDN layer = state 4 MiB (64 × 128 × 128 × 4 B) + convolution buffer 72 KiB (3 × 12,288 × 2 B). Nothing else survives the token.
Hover or focus a block for its config field and tensor shape.
Figure 7.One Gated DeltaNet layer as Qwen builds it. The hatched boxes are all it carries from one token to the next.
GDN is three of every four layers in all three Qwen models (full_attention_interval: 4). In
Qwen3.8 the fourth is Qwen Sparse Attention (QSA, T3): an indexer scores blocks of 4 tokens, and
each query attends to at most 2,048 selected tokens plus the last incomplete block
(indexer_budget: 2048), while keys and values are still cached for every token. In the report’s
ablation (a mixture of experts with 3B of 25B parameters active, trained up to 32K context), the
GDN hybrid beat one with 128-token SWA layers, sink not stated, on seven of nine short-context
benchmarks, results that “do not by themselves isolate which architectural component causes each
improvement.”
Kimi Delta Attention
KDA, from Kimi Linear (Kimi Team, 2025), gives each key channel its own decay, as GLA did for
additive linear attention:
Each row of the state fades at its own rate, giving one head several time scales. The decay
logits ut come from a rank-128 projection of the input. With
Γt=∏r≤tαr∈(0,1]dk counted from a reference position, key s
enters the read at t with weight
qt⊤Diag(Γt/Γs)ks=(qt⊙Γt)⊤(ks⊘Γs),
so rescaling queries by Γ and keys by 1/Γ restores one matrix product, but
1/Γ can overflow. Kimi Linear splits each 64-token chunk into 16-token tiles.
Below the diagonal the reference can sit at the tile boundary, keeping every factor at most 1;
diagonal tiles need factors above 1 and ran position by position, “the main intra-chunk
bottleneck” in the Kimi K3 report’s words.
K3 bounds the decay. In place of Kimi Linear’s unbounded log-decay
ℓt=−eAlogsoftplus(ut) it uses
ℓt=ℓminσ(eAlogut)∈(ℓmin,0),αt=eℓt,
with ℓmin=−5 (gate_lower_bound: -5.0), so over a tile the cumulative log-decay stays
above −80. The condition is 16∣ℓmin∣<ln(BF16 max)≈88.7, met up to
∣ℓmin∣=5.54, which leaves a factor of about 6,000 for the size of q and k
(computed). Every tile can then use “dense
Tensor Core matrix multiplications”, and decay alone removes at most a factor e−5 per step
(the delta term still overwrites at once).
Kimi K3: ℓ = ℓmin·sigmoid(u), bounded
Kimi Linear: ℓ = −softplus(u), no lower bound
16 × |ℓ_min| = 16 × 5.00 = 80 ≤ 88.72 = ln(BF16 max): the largest factor e^80 fits, with headroom e^8.7 ≈ 6,118; the smallest, e^−80, stays above the smallest normal e^−87.3.
Both curves with Alog = 0. Off-diagonal tiles: every factor ≤ 1 with the reference at the tile boundary, so only the diagonal tiles need the bound.
Figure 8.Kimi K3's lower-bounded decay. A floor on decay keeps the chunkwise kernel's rescaling factors finite in BF16, so no tile needs a slow position-by-position path.
K3 also made the sigmoid output gate full-rank (use_full_rank_gate: true); the decay projection
stays low-rank. Its 69 KDA and 24 Gated MLA layers (kda_layers) run three to one plus a final
MLA layer, and the MLA layers apply no positional encoding, since the decay already weights each
key by its distance (T4). GLM-5.3-Flash has 34 KDA and 11 sparse MLA layers (layer_types), the
same bound and a low-rank output gate. The gated-deltanet check counts both as GDN.
The recall limit of a fixed state
We ran a toy test without training (computed). It writes m key→value
pairs once, keys and values Gaussian vectors normalized to the unit sphere in dk=dv=64,
then queries each stored key; a recall is a read-out with cosine above 0.9 to the right value.
One-pass least squares (RLS) recalls every pair up to m=64, 49% at 80 and none at 128. The
rules in the models stay far below: at m=16 the additive state recalls 58% and the delta rule
78%, at m=32, 3% and 33%.
Counted in pairs, each one-pass rule holds a roughly fixed number whatever m is: 8 to 12 for the
delta rule, about 10 for gated delta, 6 for decay at 0.95, none for the additive state from
m=64. They behave like soft windows. At m=256 the delta rule recalls each of its five
newest pairs at least 90% of the time, drops below half at the ninth newest and recalls nothing
older than the sixteenth; as a
heuristic, each later write removes about a 1/dk share of an older pair’s component. Softmax
attention recalls all 256. Rewriting 16 of 32 keys, the delta rule
returns the new value for 52%, since later writes disturb each exact rewrite.
On multi-query associative recall (MQAR), where trained models learn their own keys, fixed-state
models fall behind attention as the number of pairs grows (Arora et al., 2023). Copying meets a
separate bound, on the bits a state holds (Jelassi et al., 2024).
full attention
SWA, W = 32
least squares (RLS) = pseudoinverse oracle
delta, β = 1
gated delta, α = 0.99
decayed, α = 0.95
additive
Hover a point for the mean cosine and its standard error.
Panel B leaves out least squares (RLS) and additive: at m = 256 they recall nothing at every age. Bands: ± 1 standard error over 20 trials. A toy: random unit keys and values in dk = dv = 64, one pass, fixed write rules, no training; not a measurement of any model.
Figure 9.A toy recall test with random keys and no training: write m key→value pairs once, then query each key. The write rules used in the models keep only their most recent pairs; least squares holds exactly d_k; attention holds all of them.
The mapped models handle recall in the stack, with one to six fixed-state layers per
full-attention layer. Qwen3.8-Next keeps one in four because full attention “retains the direct token-level retrieval that no finite-state
memory reproduces exactly”; in the shipped model those layers are QSA. T5 covers the ratio.
What a fixed state does to the serving stack
Decode is memory-bound: each byte read feeds only a few operations, while an H100 can do about 295
in the time it reads one (989 TFLOP/s against 3.35 TB/s; T1). A Qwen3.5-397B full-attention layer
uses each key and value for h/g=32/2=16 query heads, about 16 FLOP per byte. A GDN layer
reads and writes back its 4 MiB FP32 state per token, about 7dkdv operations on 8dkdv
bytes per head: under one FLOP per byte, with constant traffic (computed). By its report,
Nemotron 3 Ultra leads Qwen3.5-397B on a decode-heavy workload (large batch, no speculative
decoding) “despite trailing on prefill”, because Mamba-2’s decode cost per step is constant.
The state is updated in place at every token and kept in FP32: in the Qwen and Nemotron configs,
by a cast in GLM-5.3-Flash’s code, and by assumption for Kimi K3, whose code keeps what its FLA
kernel returns. Nemotron 3 Ultra’s 32-bit state is larger than its FP8 KV
cache “up to 64K” tokens, by its report (384 MiB against 6 KiB per token; computed), and its
release stores FP16 with stochastic rounding, which rounds up or down at random in proportion to
the remainder so that small updates survive on average.
Tensor parallelism (TP), splitting each layer’s heads over GPUs, splits a state by head, while a
GQA cache with g=2 splits over at most two GPUs and an MLA latent is copied to all. At TP = 8
the per-GPU crossover falls from 6,252 to 1,563 tokens for Qwen3.5-397B and from 16,253 to 2,032
for Kimi K3 (computed).
Prefix caching reuses the cache of a prefix that many requests share. A KV cache can be cut at
any token; a state exists only where it was saved, and kernels can save it only at chunk ends. Kimi
K3 hashes prefixes in fine blocks (512 tokens in its example) and keeps KDA checkpoints of 429 MiB
at a sparse subset of block ends, mostly turn boundaries. DeepSeek-V4 can also store no SWA entries and
recompute the last L⋅W tokens, the span a relay can reach. In speculative decoding,
rejected draft tokens have already entered the state, the convolution buffer and any SWA ring
buffer: Nemotron 3 Ultra snapshots its state at every draft step, and Kimi K3 caches the drafts’
projected inputs and replays the accepted ones in one fused kernel.
In packed training the state and the convolution must restart at every document boundary (HF
passes cu_seqlens; a plain nn.Conv1d fallback leaks across documents). Training and decode run
different kernels, often in different dtypes, so trainer and server log-probabilities differ.
Prefill gains the most arithmetic: over a 128K prompt a Qwen3.5-397B GDN layer’s mixing core costs
about 13 MFLOP per token, a full-attention layer 2.1 GFLOP (num_attention_heads: 32). With the
15 full layers and the projections, the model needs about 67 GFLOP per token against 163 for an
all-full stack (computed).
hs: state heads (value heads for GDN); b=4 (FP32), assumed for Kimi K3. A Householder
matrix I−2kk⊤ (unit k) is a reflection; I−βkk⊤ generalizes it.
Where the models in the map sit
The table adds, per sequence, the fixed-state layers’ memory and the global layers’ KV cache, from
each config.json: FP32 states with convolution buffers, BF16 KV, no MTP layers and no index
caches (T3; Qwen3.8’s indexer adds 3 KiB per token). The per-GPU crossover assumes TP = 8. A †
marks a 1M value beyond the stated context (config, or the 1M evaluations in the Nemotron 3 Ultra
and Qwen3.8 reports). The table compares memory only; T5 compares quality.
At 128K the fixed state is at most a fifth of any total (Nemotron 3 Ultra). The DeepSeek-V4 family is left
out: its SWA branch adds 128 KiB per layer in BF16, 72 KiB in DeepSeek’s FP8/BF16 format
(computed), and its other caches belong to T3.
fixed state
global KV: keys
values (MLA: latent)
all layers global
hatched = cached state · † beyond stated context
At 128K tokens, BF16 KV (2 B): totals from 0.80 GiB (Nemotron 3.5 Lightning) to 6.29 GiB (MiMo-V2.5-Pro, V2.6-Pro); the fixed state is 0.098–20.2% of each total.
Hover or focus a bar for its formula and config fields.
FP32 states with their convolution buffers; a sliding window's cache counts as fixed state and stays BF16 under the FP8 toggle. No MTP layers, no index caches. A ▸ at the right edge gives an all-global value off the axis, in GiB. Per GPU: state heads split 8 ways, GQA KV heads min(g, 8) ways, the MLA latent not at all.
Figure 10.Memory per sequence for the hybrids in the map: fixed state plus the KV cache of the global layers, BF16 KV, FP32 state, no index caches.
Easy to get wrong
“Linear in sequence length” describes prefill. At decode a fixed-state layer costs the same for
every token and is still memory-bound, at lower intensity than the GQA layer it replaces.
Below the crossover length a fixed state is larger than the KV cache it replaces: up to 64K
tokens for Nemotron 3 Ultra against FP8 KV (its report).
One pass of the delta rule keeps the newest pairs exact and lets older ones drift. Only a
least-squares write such as RLS reaches dk pairs per key head.
Old ideas in new places
Linear attention is the correlation-matrix memory of Anderson, Kohonen
and Nakano (1972); the delta rule with unit keys is normalized LMS (Nagumo & Noda, 1967), a
Kaczmarz step, and falls short of Kohonen’s optimal linear associative memory, which RLS reaches.
With a forgetting factor RLS is a Kalman filter for a drifting state, and GDN its LMS
approximation. Data-dependent decay is the LSTM forget gate (Gers et al., 2000), KDA’s a
forget-gate vector, Mamba-2’s step size the GRU’s coupled update gate. The rank bound is Ho &
Kalman’s (1966) realization theory. Wang, Shi & Fox (2025) treat all these
layers as regression solved at test time.
Check yourself
Nemotron 3.5 Lightning has 23 Mamba-2 layers with 64 heads, mamba_head_dim: 64 and
ssm_state_size: 128, and 6 attention layers with 2 KV heads of 128. How large is its FP32
state, and at what context does its BF16 KV cache reach that size?
Answer64 × 64 × 128 × 4 B = 2 MiB per layer, 46 MiB in all. KV:
6 × 2 × 2 × 128 × 2 B = 6 KiB per token, so about 7,850 tokens.
Starting from an empty state, a layer writes key k (∥k∥=1) with value v1, then k
again with v2, at β=1. What does reading with k return under the additive rule and
under the delta rule?
AnswerAdditive: v1+v2. Delta: the second write first removes
S⊤k=v1, so v2, until a later key with kj⊤k=0 is written.
For dk=128 and random unit keys and values, at about what m does an additive state’s
read-out fall below cosine 0.9, and why doesn’t the delta rule hold 128 pairs after one pass?
Answer(1+(m−1)/128)−1/2>0.9 needs
m<1+0.23⋅128≈31. Later writes shift older reads by (kj⊤ki) times their error, and random
keys aren’t orthogonal.