single mention · model architecture
Layerwise Hybrid Linear and Global Attention
Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA.1 Each block contains 3 KDA layers followed by 1 Gated MLA layer, giving a 3:1 mixing ratio.2
Trade-off Combined with refined training and data recipes, they yield an approximately 2.5× improvement in overall scaling efficiency over Kimi K2.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA
— k3 tech reportEach block contains 3 KDA layers followed by 1 Gated MLA layer, giving a 3:1 mixing ratio
— k3 tech reportCombined with refined training and data recipes, they yield an approximately 2.5× improvement in overall scaling efficiency over Kimi K2
— k3 tech report
Also called Hybrid Attention.
Filed under model architecture › token mixer › hybrid layer stacking
In practice
Checked as “Linear-attention or state-space layers alongside full attention” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| GLM-5.3-Flash | in its code | not stated | layer_types=["linear_attention", "linear_attent…layer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across the 45 decoder layers. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse… | code → |
| Qwen3.8-Flash-Next | in its code | not stated | layer_types=["linear_attention", "linear_attent…layer_types mixes linear_attention (Gated DeltaNet) layers with full/indexed-attention softmax layers in a fixed pattern. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not stated | layers_block_type=["mamba", "moe", "mamba", "moe", "m…layers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers across the 52-layer stack. layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",… | code → |
| Kimi K3 | in its code | adopted | full_attn_layers=[4, 8, 12, 16, 20, 24, 28, 32, 36, … · kda_layers=[1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 1…The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers. text_config.linear_attn_config.kda_layers = [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27,…text_config.linear_attn_config.full_attn_layers = [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80… | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not stated | layers_block_type=["mamba", "moe", "mamba", "moe", "m…layers_block_type mixes mamba (linear_attention) layers with full_attention and moe layers, dispatched via MIXER_TYPES based on block type. layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "moe", "mamba", "attention", "moe",… | code → |
| Qwen3.6-35B-A3B | in its code | not stated | layer_types=["linear_attention", "linear_attent…layer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per full_attention_interval. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… | code → |
| Qwen3.5-397B-A17B | in its code | not stated | layer_types=["linear_attention", "linear_attent… · full_attention_interval=4layer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (every 4th layer is full attention). text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…text_config.full_attention_interval = 4 | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThere is no linear-attention or SSM mixer implementation; every layer is the same MLA/DSA attention module per layer_types. | code → |
| LFM3-1.7B | not in its code | not stated | layer_types=["conv", "conv", "full_attention", …Non-attention layers use a short causal convolution (Lfm3ShortConv), not a linear-attention or SSM recurrent mixer, alongside full/sliding softmax attention. layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full… | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesNo linear-attention, Mamba, or SSM layer type exists in this decoder; all layers are softmax attention (full or sparse-indexer selected). | code → |
| DeepSeek-V4.1-Flash-1017 | not in its code | not stated | verdict and linesNo linear-attention, Mamba, or recurrent SSM mixer layers are present; all token-mixing layers use (sparse) softmax MLA attention. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and lineslayers_block_type interleaves 'mamba' (a linear-complexity SSM mixer) with 'moe' feed-forward blocks and occasional 'attention' softmax layers in the text_config. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax attention variants. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesAll decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type is present. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesAll token-mixing layers in this file are softmax attention (full or sliding-window); there is no linear-attention, Mamba, or state-space mixer implementation. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type. | code → |
| DeepSeek-V4.1-Flash | not in its code | not stated | verdict and linesNo linear-attention, SSM, Mamba, or lightning-attention token mixer is present; all layers use (sparse) softmax MLA attention, varying only in compression/window pattern. | code → |
| Hy4-preview | not in its code | not stated | layer_types=["deepseek_sparse_attention", "deep…No linear-attention, Mamba, or Gated DeltaNet layer types or configuration keys exist; all layers are DeepSeek sparse (softmax) attention. layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a… | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | verdict and linesNo linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention class. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesAll layers use the same GlmMoeDsaAttention (MLA with indexer sparsity); there is no linear-attention/SSM mixer variant in this file. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | verdict and linesNo recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants with different KV compression, not linear-complexity recurrent mixers | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesNo linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with sliding window plus optional compressors. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesAll decoder layers use the same Qwen3VLTextAttention softmax attention; no linear-attention or SSM layer variants exist. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesAll token-mixing layers in LagunaDecoderLayer use LagunaAttention (softmax attention); no linear-attention or SSM layer type exists in layer_types or the code. | code → |
| Inkling | not in its code | not stated | verdict and linesAll token-mixing layers use the same relative-position softmax attention (full or sliding-window variants); there are no linear-attention or SSM mixer layers in this model. | code → |
| Hy3 | not in its code | not stated | verdict and linesAll decoder layers use the same HYV3Attention softmax attention; there is no recurrent/linear mixer layer type. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesNo linear-attention or SSM mixer layers exist in this file; every decoder layer uses GlmMoeDsaAttention (MLA with indexer sparsity). | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesThere is no recurrent/state-space linear-attention layer type; only softmax attention (full or sparse-indexer-selected) layers exist. | code → |
| Step-3.7-Flash | not in its code | not stated | layer_types=["full_attention", "sliding_attenti…All layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type appears in this config's layer_types list. text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesAll text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer type in the code. | code → |
| DeepSeek-V4-Flash | not in its code | not stated | verdict and linesAll layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with compression/indexing, not linear-complexity recurrent mixers like linear attention or SSMs; no Mamba/linear-attention module is present. | code → |
| DeepSeek-V4-Pro | not in its code | not stated | verdict and linesNo linear-attention or state-space recurrent mixer layers exist; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants implemented by DeepseekV4Attention. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesAll decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesAll decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesAll token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in layer_types. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesAll layers use the same DeepseekV32Attention (MLA + indexer); there are no linear-attention/state-space mixer layers in this file. | code → |
| X-VLA | not in its code | not stated | verdict and linesAll token-mixing layers (Florence2 encoder/decoder, action transformer) use standard softmax attention; no linear-attention or state-space mixers are implemented. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesAll layers use the same GptOssAttention softmax attention class; no linear-attention or SSM mixer layers exist | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | model_type=llamaNo linear-attention/state-space layer config (layer_types, hybrid_override_pattern, linear_attn_config) or code is present; the model uses a standard Llama transformer. text_config.model_type = "llama" | code → |