Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

single mention · model architecture

Layerwise Hybrid Linear and Global Attention

Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA.1 Each block contains 3 KDA layers followed by 1 Gated MLA layer, giving a 3:1 mixing ratio.2

Trade-off Combined with refined training and data recipes, they yield an approximately 2.5× improvement in overall scaling efficiency over Kimi K2.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA — k3 tech report
  2. Each block contains 3 KDA layers followed by 1 Gated MLA layer, giving a 3:1 mixing ratio — k3 tech report
  3. Combined with refined training and data recipes, they yield an approximately 2.5× improvement in overall scaling efficiency over Kimi K2 — k3 tech report

Also called Hybrid Attention.

Filed under model architecture › token mixer › hybrid layer stacking

In practice

Checked as “Linear-attention or state-space layers alongside full attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
GLM-5.3-Flashin its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across the 45 decoder layers.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse…
        self.self_attn = (
            Glm5NextTextLinearAttention(config, layer_idx)
            if self.block_type == "linear_attention"
            else Glm5NextTextAttention(config, layer_idx)
        )
modeling_glm5_next.py · L1284–1288
Qwen3.8-Flash-Nextin its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types mixes linear_attention (Gated DeltaNet) layers with full/indexed-attention softmax layers in a fixed pattern.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
if self.layer_type == "linear_attention":
            self.linear_attn = Qwen4ExpTextGatedDeltaNet(config, layer_idx)
        else:
            self.self_attn = Qwen4ExpTextAttention(config, layer_idx)
modeling_qwen4_exp.py · L1251–1254
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot stated
layers_block_type=["mamba", "moe", "mamba", "moe", "m…

layers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers across the 52-layer stack.

layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L923–928
Kimi K3in its codeadopted
full_attn_layers=[4, 8, 12, 16, 20, 24, 28, 32, 36, … · kda_layers=[1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 1…

The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers.

text_config.linear_attn_config.kda_layers = [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27,…text_config.linear_attn_config.full_attn_layers = [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80…
if config.is_kda_layer(layer_idx):
            self.is_linear_attn = True
            self.self_attn = KimiDeltaAttention(
                config=config, layer_idx=layer_idx)
        elif config.is_mla:
            self.is_linear_attn = False
            self.self_attn = KimiMLAAttention(
                config=config, layer_idx=layer_idx)
modeling_kimi_linear.py · L883–890
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot stated
layers_block_type=["mamba", "moe", "mamba", "moe", "m…

layers_block_type mixes mamba (linear_attention) layers with full_attention and moe layers, dispatched via MIXER_TYPES based on block type.

layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "moe", "mamba", "attention", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L923–928
Qwen3.6-35B-A3Bin its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per full_attention_interval.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
if self.block_type == "linear_attention":
            self.linear_attn = Qwen3_5MoeGatedDeltaNet(config, layer_idx)
        elif self.block_type == "full_attention":
            self.self_attn = Qwen3_5MoeAttention(config, layer_idx)
modeling_qwen3_5_moe.py · L932–935
Qwen3.5-397B-A17Bin its codenot stated
layer_types=["linear_attention", "linear_attent… · full_attention_interval=4

layer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (every 4th layer is full attention).

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…text_config.full_attention_interval = 4
if self.block_type == "linear_attention":
            self.linear_attn = Qwen3_5MoeGatedDeltaNet(config, layer_idx)
        elif self.block_type == "full_attention":
            self.self_attn = Qwen3_5MoeAttention(config, layer_idx)
modeling_qwen3_5_moe.py · L932–935
Mistral-Large-4not in its codenot stated
verdict and lines

There is no linear-attention or SSM mixer implementation; every layer is the same MLA/DSA attention module per layer_types.

LFM3-1.7Bnot in its codenot stated
layer_types=["conv", "conv", "full_attention", …

Non-attention layers use a short causal convolution (Lfm3ShortConv), not a linear-attention or SSM recurrent mixer, alongside full/sliding softmax attention.

layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full…
class Lfm3ShortConv(nn.Module):
modeling_lfm3.py · L341
MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

No linear-attention, Mamba, or SSM layer type exists in this decoder; all layers are softmax attention (full or sparse-indexer selected).

DeepSeek-V4.1-Flash-1017not in its codenot stated
verdict and lines

No linear-attention, Mamba, or recurrent SSM mixer layers are present; all token-mixing layers use (sparse) softmax MLA attention.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

layers_block_type interleaves 'mamba' (a linear-complexity SSM mixer) with 'moe' feed-forward blocks and occasional 'attention' softmax layers in the text_config.

Step-5-Previewnot in its codenot stated
verdict and lines

No linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax attention variants.

Kolibri-1not in its codenot stated
verdict and lines

All decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type is present.

        self.self_attn = Kolibri1Attention(config=config, layer_idx=layer_idx)
        self.mlp = Kolibri1SparseMoeBlock(config)
modeling_kolibri1.py · L321–322
MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

All token-mixing layers in this file are softmax attention (full or sliding-window); there is no linear-attention, Mamba, or state-space mixer implementation.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

All decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type.

DeepSeek-V4.1-Flashnot in its codenot stated
verdict and lines

No linear-attention, SSM, Mamba, or lightning-attention token mixer is present; all layers use (sparse) softmax MLA attention, varying only in compression/window pattern.

Hy4-previewnot in its codenot stated
layer_types=["deepseek_sparse_attention", "deep…

No linear-attention, Mamba, or Gated DeltaNet layer types or configuration keys exist; all layers are DeepSeek sparse (softmax) attention.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
verdict and lines

No linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention class.

self.self_attn = DeepseekV4Attention(config, layer_idx)
modeling_deepseek_v4.py · L1168
GLM-5.3not in its codenot stated
verdict and lines

All layers use the same GlmMoeDsaAttention (MLA with indexer sparsity); there is no linear-attention/SSM mixer variant in this file.

DeepSeek-V4-Pro-0813not in its codenot stated
verdict and lines

No recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants with different KV compression, not linear-complexity recurrent mixers

DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

No linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with sliding window plus optional compressors.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

All decoder layers use the same Qwen3VLTextAttention softmax attention; no linear-attention or SSM layer variants exist.

Laguna-S-2.1not in its codenot stated
verdict and lines

All token-mixing layers in LagunaDecoderLayer use LagunaAttention (softmax attention); no linear-attention or SSM layer type exists in layer_types or the code.

Inklingnot in its codenot stated
verdict and lines

All token-mixing layers use the same relative-position softmax attention (full or sliding-window variants); there are no linear-attention or SSM mixer layers in this model.

Hy3not in its codenot stated
verdict and lines

All decoder layers use the same HYV3Attention softmax attention; there is no recurrent/linear mixer layer type.

self.self_attn = HYV3Attention(config=config, layer_idx=layer_idx)
modeling_hy_v3.py · L372
GLM-5.2not in its codenot stated
verdict and lines

No linear-attention or SSM mixer layers exist in this file; every decoder layer uses GlmMoeDsaAttention (MLA with indexer sparsity).

MiniMax-M3not in its codenot stated
verdict and lines

There is no recurrent/state-space linear-attention layer type; only softmax attention (full or sparse-indexer-selected) layers exist.

Step-3.7-Flashnot in its codenot stated
layer_types=["full_attention", "sliding_attenti…

All layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type appears in this config's layer_types list.

text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
MolmoAct 2not in its codenot stated
verdict and lines

All text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer type in the code.

DeepSeek-V4-Flashnot in its codenot stated
verdict and lines

All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with compression/indexing, not linear-complexity recurrent mixers like linear attention or SSMs; no Mamba/linear-attention module is present.

DeepSeek-V4-Pronot in its codenot stated
verdict and lines

No linear-attention or state-space recurrent mixer layers exist; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants implemented by DeepseekV4Attention.

MiMo-V2.5not in its codenot stated
verdict and lines

All decoder layers use MiMoV2Attention (softmax attention, full or sliding-window); there are no linear-attention or SSM mixer layers.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

All decoder layers use MiMoV2Attention (softmax attention, either full or sliding-window); there is no linear-attention or SSM mixer layer type.

self.self_attn = MiMoV2Attention(
            config, is_swa_layer, layer_idx, projection_layout=attention_projection_layout
        )
modeling_mimo_v2.py · L397–399
Gemma 4 31Bnot in its codenot stated
verdict and lines

All token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in layer_types.

DeepSeek-V3.2not in its codenot stated
verdict and lines

All layers use the same DeepseekV32Attention (MLA + indexer); there are no linear-attention/state-space mixer layers in this file.

X-VLAnot in its codenot stated
verdict and lines

All token-mixing layers (Florence2 encoder/decoder, action transformer) use standard softmax attention; no linear-attention or state-space mixers are implemented.

gpt-oss-120bnot in its codenot stated
verdict and lines

All layers use the same GptOssAttention softmax attention class; no linear-attention or SSM mixer layers exist

OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

No linear-attention/state-space layer config (layer_types, hybrid_override_pattern, linear_attn_config) or code is present; the model uses a standard Llama transformer.

text_config.model_type = "llama"