Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spacegated delta network

methods · linear attention & state space

gated delta network

12 methods filed at this node or below it, from the sources of 6 models.

In practice

Checked as “Gated DeltaNet layers” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
GLM-5.3-Flashin its codenot stated
linear_attn_config={"num_heads": 64, "gate_lower_bound…

Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule write (chunk_kimi_delta_attention/recurrent_kimi_delta_attention) and a decay gate from the forget gate module, active on all linear_attention layers.

text_config.linear_attn_config = {"num_heads": 64, "gate_lower_bound": -5.0, "head_dim": 128, "short_conv_kern…
        last_recurrent_state = last_recurrent_state * g_i
        kv_mem = (last_recurrent_state * k_i[..., None]).sum(dim=-2)
        delta = (v_i - kv_mem) * b_i

        last_recurrent_state = last_recurrent_state + k_i.unsqueeze(-1) * delta.unsqueeze(-2)
modeling_glm5_next.py · L506–510
    """Kimi-style KDA (Kimi Linear Attention) for GLM-5.3-Flash."""
modeling_glm5_next.py · L621
Qwen3.8-Flash-Nextin its codeadopted
linear_num_key_heads=16 · linear_conv_kernel_dim=4

Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers.

text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
modeling_qwen4_exp.py · L572
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
modeling_qwen4_exp.py · L376
Kimi K3in its codeadopted
num_heads=96 · short_conv_kernel_size=4

KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers.

text_config.linear_attn_config.num_heads = 96text_config.linear_attn_config.short_conv_kernel_size = 4
self.A_log = torch.nn.Parameter(torch.log(torch.empty(
            self.num_heads, dtype=torch.float32).uniform_(1, 16)))

        self.f_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
        self.f_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
modeling_kimi_linear.py · L520–524
o, recurrent_state = chunk_kda(
                q=q,
                k=k,
                v=v,
                g=g,
                beta=beta,
                A_log=self.A_log,
                dt_bias=self.dt_bias,
modeling_kimi_linear.py · L610–617
Qwen3.6-35B-A3Bin its codeadopted
linear_num_key_heads=16 · linear_conv_kernel_dim=4

Qwen3_5MoeGatedDeltaNet implements the gated delta rule with decay gate g and beta update, used in all linear_attention layers.

text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
modeling_qwen3_5_moe.py · L406
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
modeling_qwen3_5_moe.py · L602
Qwen3.5-397B-A17Bin its codeadopted
linear_num_key_heads=16 · linear_conv_kernel_dim=4

Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used in linear_attention layers.

text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
modeling_qwen3_5_moe.py · L602
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
modeling_qwen3_5_moe.py · L406
Mistral-Large-4not in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model's code or config.

LFM3-1.7Bnot in its codenot stated
verdict and lines

No delta-rule recurrent state update with a decay gate is implemented; the non-attention layers are plain causal convolutions.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

There is no gated delta rule recurrent mixer implemented in the given files.

DeepSeek-V4.1-Flash-1017not in its codenot stated
verdict and lines

No gated delta-rule recurrent layers (Gated DeltaNet / Kimi Delta Attention) are implemented or configured; the model uses softmax sparse MLA attention throughout.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

The config has mamba-specific keys (mamba_num_heads, conv_kernel, ssm_state_size) rather than linear_num_key_heads/linear_conv_kernel_dim used by Gated DeltaNet, and the mixer implementation file is not provided, so it is unclear whether the mamba layers use the gated delta rule versus standard Mamba-2 SSD.

Step-5-Previewnot in its codenot stated
verdict and lines

No gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and full_attention.

Kolibri-1not in its codenot stated
verdict and lines

No delta-rule recurrent update or decay gate is implemented anywhere in the modeling file.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer is implemented; only softmax attention layers exist.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

No gated delta-rule recurrent module is implemented; all token mixing is standard softmax attention.

DeepSeek-V4.1-Flashnot in its codenot stated
verdict and lines

No gated delta rule / delta-net recurrent layer implementation appears in the given files; the model's token mixer is MLA attention throughout.

Hy4-previewnot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present.

DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention implementation.

GLM-5.3not in its codenot stated
verdict and lines

No gated delta rule / recurrent state update code exists in the given files.

DeepSeek-V4-Pro-0813not in its codenot stated
verdict and lines

No gated delta-rule recurrent state update is implemented anywhere in the modeling file

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

The recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form exists.

DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer is implemented anywhere in the modeling file.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer or linear_num_key_heads/linear_conv_kernel_dim config is present in the code.

Laguna-S-2.1not in its codenot stated
verdict and lines

No gated delta-rule recurrent mechanism or linear_num_key_heads/linear_conv_kernel_dim config exists in this model.

Inklingnot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer exists; the only non-attention mixing component is a short depthwise convolution (sconv), not a delta-rule gated state.

Hy3not in its codenot stated
verdict and lines

No delta-rule recurrent state or decay gate implementation is present in the code.

GLM-5.2not in its codediscussed
verdict and lines

No gated delta-rule recurrent mixer is implemented; attention layers are MLA/indexer-sparse, not Gated DeltaNet.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

The recurrent mixer is a Mamba-2 SSM block (selective state-space with conv1d), not a gated delta-rule recurrence; no delta-rule write or Kimi Delta Attention code is present.

MiniMax-M3not in its codenot stated
verdict and lines

No gated delta rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in this model.

Step-3.7-Flashnot in its codenot stated
verdict and lines

No gated delta rule / recurrent state module exists in the provided code; layer_types contains only full_attention and sliding_attention.

MolmoAct 2not in its codenot stated
verdict and lines

No gated delta rule or recurrent state update mechanism exists in the text model code; attention layers are standard softmax attention.

DeepSeek-V4-Flashnot in its codenot stated
verdict and lines

No gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention with compressors/indexer.

DeepSeek-V4-Pronot in its codenot stated
verdict and lines

There is no gated delta-rule recurrent state update implemented anywhere in this file; all token mixing is softmax attention.

MiMo-V2.5not in its codenot stated
verdict and lines

No gated delta rule / recurrent state update mechanism is implemented in the code.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer implementation or config keys (linear_num_key_heads, linear_conv_kernel_dim) are present.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No gated delta rule / recurrent state update mechanism is implemented in these files.

DeepSeek-V3.2not in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present.

X-VLAnot in its codenot stated
verdict and lines

No gated delta-rule recurrent mixer is implemented anywhere in the provided files.

gpt-oss-120bnot in its codenot stated
verdict and lines

No gated delta-rule recurrent layer implementation is present in this file

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No gated delta-rule or Kimi Delta Attention mechanism or associated config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in the files or config.

In this branch 12

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 12

Gated DeltaNet core · 12 sources · 12 quotes
Kimi Delta Attention core · 6 sources · 6 quotes
KDA core · 2 sources · 2 quotes
8 single mentions
Gated attention with reduced KV heads core · 1 source · 1 quote
Gated DeltaNet hybrid attention core · 1 source · 1 quote
Hybrid linear attention mechanism core · 1 source · 1 quote
Bounded sigmoid output gate used · 1 source · 1 quote
Input-dependent full-rank output gate used · 1 source · 1 quote
Lower-bounded scaled-sigmoid log-decay used · 1 source · 1 quote
SimpleGDN evaluated · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑