Gated DeltaNet is a linear-attention token mixer that compresses the prefix into a fixed-size recurrent state and updates that state according to the current content.1 It applies a gated delta rule whose delta term writes only the residual error between the new value and the value already stored, so repeated or similar keys update an existing association instead of accumulating outer products.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Also called Gated Delta Net (GDN) module, Gated Delta Network linear attention, linear attention via Gated Delta Networks, Gated DeltaNet mechanism, Gated DeltaNet (GDN), Gated DeltaNet linear attention.
Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule write (chunk_kimi_delta_attention/recurrent_kimi_delta_attention) and a decay gate from the forget gate module, active on all linear_attention layers.
Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers.
KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers.
Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used in linear_attention layers.
No gated delta-rule recurrent layers (Gated DeltaNet / Kimi Delta Attention) are implemented or configured; the model uses softmax sparse MLA attention throughout.
The config has mamba-specific keys (mamba_num_heads, conv_kernel, ssm_state_size) rather than linear_num_key_heads/linear_conv_kernel_dim used by Gated DeltaNet, and the mixer implementation file is not provided, so it is unclear whether the mamba layers use the gated delta rule versus standard Mamba-2 SSD.
No gated delta-rule recurrent mixer exists; the only non-attention mixing component is a short depthwise convolution (sconv), not a delta-rule gated state.
The recurrent mixer is a Mamba-2 SSM block (selective state-space with conv1d), not a gated delta-rule recurrence; no delta-rule write or Kimi Delta Attention code is present.
No gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention with compressors/indexer.
No gated delta-rule or Kimi Delta Attention mechanism or associated config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in the files or config.
GDN compresses the prefix into a fixed-size recurrent state and updates that state according to the current content, while the interleaved global-attention layers
this architecture includes the Gated Delta Net (GDN) module, which is particularly effective for accelerating the modeling of long audio-video sequences.
The 3:1 interleave of Gated DeltaNet to full-attention layers is the same ratio Qwen3.5 uses, keeping most layers on linear attention whose state does not grow with sequence length.
coremodel architecturein Qwen3.8-Flash-NextQwen Team
Gated DeltaNet (GDN) [54]: A linear attention variant that replaces the quadratic softmax attention computation with a gated linear recurrence, reducing the computational cost of attention from quadratic to linear in sequence length.