specific method · filed under model architecture
Gated DeltaNet
A linear-attention method using a gated recurrence and delta-rule state updates, with recurrent state used to represent sequence history.
Also called Gated Delta Net (GDN) module, Gated Delta Network linear attention, linear attention via Gated Delta Networks, gated delta networks hybrid attention, Gated DeltaNet mechanism, Gated DeltaNet (GDN).
- sources
- 13
- models
- 5
- labs adopt it
- 2
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Gated Delta Networks: Improving Mamba2 with Delta Rule (Yang et al., 2024) paper arxiv.orgintroduces Gated DeltaNet
In the code
Checked as “Gated DeltaNet layers” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | Glm5NextTextLinearAttention implements Kimi Delta Attention (a gated delta-rule variant) with a forget gate decay and delta-rule state update, used in linear_attention layers. modeling_glm5_next.py#L508 |
| DeepSeek-V4.1-Flash | not in its code | not stated | No gated delta-rule recurrent mixer code exists in the given files; attention layers are exclusively sparse/sliding-window MLA. |
| Hy4-preview | not in its code | not stated | No gated delta-rule recurrent mixer code or config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model. |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | No gated delta-rule recurrent module exists in the code; all layers use DeepseekV4Attention softmax attention. |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | The recurrent layers implement the Mamba-2 selective SSM (chunked scan with segment sums), not a gated delta-rule mixer like Gated DeltaNet or Kimi Delta Attention. modeling_nemotron_h.py#L366 |
| MiMo-V2.6-Flash | not in its code | not stated | No gated delta-rule recurrent mixer is implemented; only softmax attention layers exist. |
| DeepSeek-V4-Flash | not in its code | not stated | No gated delta-rule recurrent module exists in the code; all layers use DeepseekV4Attention (softmax attention with optional compression). |
| MiMo-V2.5 | not in its code | not stated | No gated delta rule / recurrent state update mechanism is implemented in the code. |
| GLM-5.3 | not in its code | not stated | No gated delta-rule recurrent mixer or linear_num_key_heads/linear_conv_kernel_dim config exists in this codebase. |
| Hy3 | not in its code | not stated | No gated delta-rule recurrent module is implemented anywhere in the file |
| GLM-5.2 | not in its code | evaluated | No delta-rule recurrent state or decay gate mechanism is implemented anywhere in the modeling file. |
| MiniMax-M3 | not in its code | not stated | No gated delta-rule recurrent mixer is implemented; only standard attention and the lightning indexer exist. |
| DeepSeek-V3.2 | not in its code | not stated | No gated delta rule / recurrent state mixer is implemented; only MLA attention layers exist. |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | No gated delta-rule recurrent mixer is implemented anywhere in this file. |
| DeepSeek-V4-Pro | not in its code | not stated | No gated delta-rule recurrent mixer implementation is present in the given files |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | No gated delta-rule recurrent module exists in the code; layer_types only cover sliding/CSA/HCA softmax-attention variants. |
| Gemma 4 31B | not in its code | not stated | No gated delta-rule recurrent mixer is implemented; the model uses only softmax attention layer types. |
| Inkling | not in its code | not stated | No gated delta-rule recurrent mixer is implemented; the short convolution (sconv) is a plain depthwise causal conv1d with residual add, not a delta-rule/decay-gated state update. |
| Kimi K3 | in its code | used | KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers. modeling_kimi_linear.py#L520 |
| Laguna-S-2.1 | not in its code | not stated | No delta-rule recurrent state update or decay gate code exists; all layers are standard softmax attention. |
| MiMo-V2.5-Pro | not in its code | not stated | No gated delta-rule recurrent mixer implementation or config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. |
| MiMo-V2.6-Pro | not in its code | not stated | No gated delta-rule recurrent module is implemented; all token mixing is standard softmax attention. |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | The linear-attention layers implement Mamba-2 style SSM scan (segment sum, chunked SSD), not a gated delta rule / Kimi Delta Attention recurrence. |
| Qwen3.5-397B-A17B | in its code | core | Linear-attention layers implement the gated delta rule (Qwen3_5MoeGatedDeltaNet) with a decay gate g and delta-rule chunked/recurrent update, configured via linear_num_key_heads/linear_conv_kernel_dim. modeling_qwen3_5_moe.py#L621 |
| Qwen3.6-35B-A3B | in its code | core | Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and beta-weighted delta update, used for all linear_attention layers. modeling_qwen3_5_moe.py#L621 |
| Qwen3.8-Flash-Next | in its code | core | Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers. modeling_qwen4_exp.py#L581 |
| Step-3.7-Flash | not in its code | not stated | No gated delta-rule recurrent module exists in the given code; layer_types only distinguishes full and sliding softmax attention. |
| gpt-oss-120b | not in its code | not stated | No gated delta-rule recurrent layers are implemented; all layers use GptOssAttention. |
Evidence
13 spans quoted from the sources, strongest treatment first.
It combines Mamba2’s gated decay mechanism with a delta rule for updating hidden states.
Gated DeltaNet:- Number of Linear Attention Heads: 64 for V and 16 for QK
Gated DeltaNet:- Number of Linear Attention Heads: 32 for V and 16 for QK
The 3:1 interleave of Gated DeltaNet to full-attention layers is the same ratio Qwen3.5 uses, keeping most layers on linear attention whose state does not grow with sequence length.
GDN compresses the prefix into a fixed-size recurrent state and updates that state according to the current content
It uses the same gated delta networks hybrid attention as its MoE siblings
three of every four layers use Gated DeltaNet to compress history
linear attention via Gated Delta Networks
Gated DeltaNet: - Number of Linear Attention Heads: 32 for V and 16 for QK
Gated DeltaNet (GDN) compresses the history efficiently
similar to the Gated DeltaNet used for Olmo Hybrid
this architecture includes the Gated Delta Net (GDN) module, which is particularly effective for accelerating the modeling of long audio-video sequences.
Gated DeltaNet (GDN) [54]: A linear attention variant that replaces the quadratic softmax attention computation with a gated linear recurrence, reducing the computational cost of attention from quadratic to linear in sequence length.
Filed alongside
Other methods under model architecture :: token mixer :: linear attention & state space :: gated delta network.