Model techniques map
Techniquesmodel architecturetoken mixerlinear attention & state spacegated delta network

ambiguous · filed under model architecture

Hybrid linear attention

A method described only as replacing standard quadratic attention in a subset of layers, without evidence identifying its specific mechanism.

Also called Hybrid Linear Attention Mechanism.

source
1
model
1
lab adopt it
1
strongest
core

How sources treat it

One count per evidence span, weakest treatment to strongest.

core 1

Documented in

Kimi K3

In the code

Checked as “Linear-attention or state-space layers alongside full attention” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.

ModelCodeIts documentsWhat the code does
GLM-5.3-Flash in its codenot statedlayer_types alternates linear_attention (KDA) layers with deepseek_sparse_attention (MLA+indexer) softmax layers across the 45 decoder layers. modeling_glm5_next.py#L1285
DeepSeek-V4.1-Flash not in its codenot statedNo linear-attention or SSM token-mixing layers are implemented; every decoder layer uses the same DeepseekV4Attention (sparse/sliding-window MLA) mechanism.
Hy4-preview not in its codenot statedAll layers use the same DeepSeek sparse attention mechanism; there is no linear-attention or SSM layer type in layer_types.
DeepSeek-V4-Flash-0731 not in its codenot statedAll layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants using DeepseekV4Attention; there is no linear-attention/SSM mixer layer type in the code.
NVIDIA-Nemotron-3-Ultra-550B-A55B in its codenot statedlayers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers, dispatched via MIXER_TYPES. modeling_nemotron_h.py#L893
MiMo-V2.6-Flash not in its codenot statedAll token-mixing layers in this file are softmax attention (full or sliding-window); there is no linear-attention, Mamba, or state-space mixer implementation.
DeepSeek-V4-Flash not in its codenot statedAll layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention module with a sliding-window cache and optional compressor; there is no linear-attention or SSM token mixer.
MiMo-V2.5 not in its codenot statedAll decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers.
GLM-5.3 not in its codenot statedAll layers use the same GlmMoeDsaAttention (MLA + indexer sparse attention); there is no linear-attention/state-space mixer class or layer-type switch in this file.
Hy3 not in its codenot statedAll decoder layers use the same HYV3Attention softmax attention module; there is no linear-attention or state-space mixer variant in the code modeling_hy_v3.py#L383
GLM-5.2 not in its codenot statedNo linear-attention, Mamba, or state-space mixer layers exist in the code; every layer uses GlmMoeDsaAttention (MLA + indexer sparse attention).
MiniMax-M3 not in its codenot statedNo recurrent linear-attention or SSM layer type appears in layer_types or the code; all token-mixing layers are softmax attention (full or sparse-indexed).
DeepSeek-V3.2 not in its codenot statedThere are no linear-attention or SSM layer types; every decoder layer uses the same DeepseekV32Attention (MLA with indexer).
DeepSeek-V4-Flash-Vision-Exp not in its codenot statedNo linear-attention or SSM token-mixing layers exist; all layers use the same softmax DeepseekV4Attention class with sliding-window plus optional compressor branch.
DeepSeek-V4-Pro not in its codenot statedNo linear-attention, Gated DeltaNet, KDA, or Mamba token-mixing layers exist in this file; all layer_types variants are softmax attention with sliding-window plus compressor branches
DeepSeek-V4-Pro-0813 not in its codenot statedNo linear-attention or state-space recurrent mixer layers are implemented; all layer types (sliding, CSA, HCA) are variants of softmax attention with different KV compression, not linear-complexity recurrence.
Gemma 4 31B not in its codenot statedNo linear-attention, Gated DeltaNet, Kimi Delta Attention, or Mamba layer types appear; layer_types only contains sliding_attention and full_attention (softmax attention).
Inkling not in its codenot statedAll token-mixing layers are softmax attention (global or sliding-window) plus short convolutions on residual streams; there is no linear-attention/SSM mixer layer type. configs.py#L173
Kimi K3 in its code coreThe text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers. modeling_kimi_linear.py#L883
Laguna-S-2.1 not in its codenot statedAll decoder layers use the same LagunaAttention softmax attention class; there is no linear-attention or SSM mixer variant. modeling_laguna.py#L427
MiMo-V2.5-Pro not in its codenot statedAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type. modeling_mimo_v2.py#L397
MiMo-V2.6-Pro not in its codenot statedAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type.
NVIDIA-Nemotron-3.5-Lightning-30B-A3B in its codenot statedlayers_block_type mixes NemotronHMamba2Mixer linear-attention layers with NemotronHAttention full_attention layers and moe/mlp layers, dispatched through MIXER_TYPES. modeling_nemotron_h.py#L893
Qwen3.5-397B-A17B in its codenot statedlayer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (3 linear then 1 full). modeling_qwen3_5_moe.py#L952
Qwen3.6-35B-A3B in its codenot statedlayer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed 4-layer interleave pattern. modeling_qwen3_5_moe.py#L952
Qwen3.8-Flash-Next in its codenot statedlayer_types alternates linear_attention (GatedDeltaNet) layers with full/indexed softmax attention layers every 4th layer. modeling_qwen4_exp.py#L1261
Step-3.7-Flash not in its codenot statedlayer_types only contains full_attention and sliding_attention values for softmax attention layers; there is no linear-attention or state-space layer type or module in the modeling file.
gpt-oss-120b not in its codenot statedAll token-mixing layers are softmax attention (full or sliding); there are no linear-attention or SSM layers.

Evidence

1 span quoted from the sources, strongest treatment first.

A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers.

coremodel architecturein Kimi K3Moonshot AI
Kimi K3 Model Overview: 2.8T Parameters, MXFP4 Quantization, and What the Open Weights Mean for the Community·Hugging Face community blog·2026-07-17

Filed alongside

Other methods under model architecture :: token mixer :: linear attention & state space :: gated delta network.

Gated DeltaNetKimi Delta AttentionGated DeltaNet–sparse MoE hybridKDAGated DeltaNet with bounded sigmoid output gateGated DeltaNet with reduced KV-head configurationKimi Delta Attention and Attention Residuals architectureKimi Delta Attention with input-dependent full-rank output gateKimi Delta Attention with lower-bounded log-decaySimpleGDN