methods · linear attention & state space
gated delta network
12 methods filed at this node or below it, from the sources of 6 models.
In practice
Checked as “Gated DeltaNet layers” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | linear_attn_config={"num_heads": 64, "gate_lower_bound…Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule write (chunk_kimi_delta_attention/recurrent_kimi_delta_attention) and a decay gate from the forget gate module, active on all linear_attention layers. text_config.linear_attn_config = {"num_heads": 64, "gate_lower_bound": -5.0, "head_dim": 128, "short_conv_kern… | code → |
| Qwen3.8-Flash-Next | in its code | adopted | linear_num_key_heads=16 · linear_conv_kernel_dim=4Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers. text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 | code → |
| Kimi K3 | in its code | adopted | num_heads=96 · short_conv_kernel_size=4KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers. text_config.linear_attn_config.num_heads = 96text_config.linear_attn_config.short_conv_kernel_size = 4 | code → |
| Qwen3.6-35B-A3B | in its code | adopted | linear_num_key_heads=16 · linear_conv_kernel_dim=4Qwen3_5MoeGatedDeltaNet implements the gated delta rule with decay gate g and beta update, used in all linear_attention layers. text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 | code → |
| Qwen3.5-397B-A17B | in its code | adopted | linear_num_key_heads=16 · linear_conv_kernel_dim=4Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used in linear_attention layers. text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model's code or config. | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo delta-rule recurrent state update with a decay gate is implemented; the non-attention layers are plain causal convolutions. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesThere is no gated delta rule recurrent mixer implemented in the given files. | code → |
| DeepSeek-V4.1-Flash-1017 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent layers (Gated DeltaNet / Kimi Delta Attention) are implemented or configured; the model uses softmax sparse MLA attention throughout. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesThe config has mamba-specific keys (mamba_num_heads, conv_kernel, ssm_state_size) rather than linear_num_key_heads/linear_conv_kernel_dim used by Gated DeltaNet, and the mixer implementation file is not provided, so it is unclear whether the mamba layers use the gated delta rule versus standard Mamba-2 SSD. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and full_attention. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesNo delta-rule recurrent update or decay gate is implemented anywhere in the modeling file. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer is implemented; only softmax attention layers exist. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesNo gated delta-rule recurrent module is implemented; all token mixing is standard softmax attention. | code → |
| DeepSeek-V4.1-Flash | not in its code | not stated | verdict and linesNo gated delta rule / delta-net recurrent layer implementation appears in the given files; the model's token mixer is MLA attention throughout. | code → |
| Hy4-preview | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention implementation. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesNo gated delta rule / recurrent state update code exists in the given files. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent state update is implemented anywhere in the modeling file | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesThe recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form exists. | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer is implemented anywhere in the modeling file. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer or linear_num_key_heads/linear_conv_kernel_dim config is present in the code. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mechanism or linear_num_key_heads/linear_conv_kernel_dim config exists in this model. | code → |
| Inkling | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer exists; the only non-attention mixing component is a short depthwise convolution (sconv), not a delta-rule gated state. | code → |
| Hy3 | not in its code | not stated | verdict and linesNo delta-rule recurrent state or decay gate implementation is present in the code. | code → |
| GLM-5.2 | not in its code | discussed | verdict and linesNo gated delta-rule recurrent mixer is implemented; attention layers are MLA/indexer-sparse, not Gated DeltaNet. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesThe recurrent mixer is a Mamba-2 SSM block (selective state-space with conv1d), not a gated delta-rule recurrence; no delta-rule write or Kimi Delta Attention code is present. | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesNo gated delta rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in this model. | code → |
| Step-3.7-Flash | not in its code | not stated | verdict and linesNo gated delta rule / recurrent state module exists in the provided code; layer_types contains only full_attention and sliding_attention. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo gated delta rule or recurrent state update mechanism exists in the text model code; attention layers are standard softmax attention. | code → |
| DeepSeek-V4-Flash | not in its code | not stated | verdict and linesNo gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention with compressors/indexer. | code → |
| DeepSeek-V4-Pro | not in its code | not stated | verdict and linesThere is no gated delta-rule recurrent state update implemented anywhere in this file; all token mixing is softmax attention. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesNo gated delta rule / recurrent state update mechanism is implemented in the code. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer implementation or config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo gated delta rule / recurrent state update mechanism is implemented in these files. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo gated delta-rule recurrent mixer is implemented anywhere in the provided files. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo gated delta-rule recurrent layer implementation is present in this file | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo gated delta-rule or Kimi Delta Attention mechanism or associated config keys (linear_num_key_heads, linear_conv_kernel_dim) appear in the files or config. | code → |
In this branch 12
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 12
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| GLM-5.3-Flash core | Kimi Delta Attention core— |
| GLM-5.2 evaluated | Gated DeltaNet evaluatedSimpleGDN evaluated— |
| Kimi K3 core | Hybrid linear attention mechanism coreKDA coreKimi Delta Attention coreKimi Delta Attention–Attention Residuals architecture coreGated DeltaNet usedInput-dependent full-rank output gate usedLower-bounded scaled-sigmoid log-decay used— |
| Qwen3.5-397B-A17B core | Gated attention with reduced KV heads coreGated DeltaNet coreGated DeltaNet–sparse Mixture-of-Experts hybrid core— |
| Qwen3.6-35B-A3B core | Gated DeltaNet coreGated DeltaNet hybrid attention core— |
| Qwen3.8-Flash-Next core | Gated DeltaNet coreBounded sigmoid output gate used— |