Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

methods · token mixer

hybrid layer stacking

12 methods filed at this node or below it, from the sources of 22 models.

In practice

Checked as “Interleaved sliding-window and global attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codenot stated
layer_types=["conv", "conv", "full_attention", …

layer_types interleaves full_attention and sliding_attention layers (plus conv layers) in a fixed pattern, with separate mask builders per type.

layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full…
"full_attention": lambda: create_causal_mask(**mask_kwargs),
                "sliding_attention": lambda: create_sliding_window_causal_mask(**mask_kwargs),
                "conv": lambda: create_recurrent_attention_mask(**mask_kwargs),
modeling_lfm3.py · L553–555
DeepSeek-V4.1-Flash-1017in its codenot stated
compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…

compress_ratios interleaves layers with compress_ratio 0 (SWA-only, no compressed/global KV) against layers with ratio 2 or 1 (which read a compressed, effectively global, KV cache), giving a fixed interleaving of local and global-context attention.

text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…
swa_only = self.compress_ratio == 0
        # SWA-only layers have no compressed KV cache; consumers read the kv
        # source's cache.
        self_kv_cache = None if swa_only else self._compressed_kv_cache()
flashinfer_sparse.py · L313–316
Step-5-Previewin its codenot stated
layer_types=["sliding_attention", "sliding_atte…

layer_types fixes a repeating pattern of three sliding_attention layers followed by one full_attention layer across the 92-layer stack.

text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti…
default_layer_type = (
            "sliding_attention" if self.layer_idx % 2 == 0 else "full_attention"
        )
        layer_type = _per_layer_value(
            layer_types,
            self.layer_idx,
            name="layer_types",
            default=default_layer_type,
        )
model.py · L886–894
Kolibri-1in its codeadopted
layer_types=["sliding_attention", "sliding_atte…

layer_types interleaves four sliding_attention layers followed by one full_attention layer, repeating across all 50 layers.

layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
            causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_kolibri1.py · L498–501
MiMo-V2.6-Flashin its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48 decoder layers.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
MiMo-V2.6-Proin its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1…

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 70 layers.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
DeepSeek-V4.1-Flashin its codenot stated
compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2… · kv_source_layer_ids=[2, 8, 14, 20]

compress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV source (ratio 1 or 2), giving a fixed interleaving of local and global attention across the 40 layers.

text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…text_config.kv_source_layer_ids = [2, 8, 14, 20]
        cut = max(config.kv_source_layer_ids)
        if (
            cut < self.end_layer - 1
            and self._decoder_replay_supported(vllm_config, cut)
model.py · L882–885
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types.

interleave = [
                "compressed_sparse_attention" if i % 2 else "heavily_compressed_attention"
                for i in range(max(n - 2, 0))
            ]
            self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave
configuration_deepseek_v4.py · L270–274
self.layer_type = config.layer_types[layer_idx]
modeling_deepseek_v4.py · L785
Laguna-S-2.1in its codeadopted
layer_types=["full_attention", "sliding_attenti…

layer_types fixes a pattern of one full_attention layer followed by three sliding_attention layers, repeated across all 48 layers.

layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
mask_creation_functions = {
                "full_attention": lambda: create_causal_mask(**mask_kwargs),
                "sliding_attention": lambda: create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_laguna.py · L549–552
Inklingin its codeadopted
local_layer_ids=[0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12,… · num_hidden_layers=66

local_layer_ids selects most layers as sliding-window (swa_*) while layers not listed (e.g. 5, 11, 17...) use full attention, giving a fixed interleaving of local and global attention layers.

text_config.local_layer_ids = [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 2…text_config.num_hidden_layers = 66
self.attn = InklingAttention(
            config,
            num_heads=(
                config.swa_num_attention_heads
                if is_local
                else config.num_attention_heads
            ),
model.py · L152–158
Step-3.7-Flashin its codenot stated
layer_types=["full_attention", "sliding_attenti…

layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each.

text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**sliding_mask_kwargs),
            }
modeling_step3p7.py · L889–892
DeepSeek-V4-Flashin its codenot stated
compress_ratios=[0, 0, 4, 128, 4, 128, 4, 128, 4, 1…

layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config.

compress_ratios = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128…
self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
MiMo-V2.5in its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
MiMo-V2.5-Proin its codeadopted
verdict and lines

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across 70 layers.

is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L395–396
Gemma 4 31Bin its codeadopted
layer_types=["sliding_attention", "sliding_atte…

layer_types interleaves 5 sliding_attention layers followed by 1 full_attention layer repeatedly across the 60 layers.

text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gemma4.py · L1658–1661
gpt-oss-120bin its codeadopted
layer_types=["sliding_attention", "full_attenti…

layer_types alternates sliding_attention and full_attention across the 36 layers, each built with its own causal mask

layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gpt_oss.py · L466–469
Mistral-Large-4not in its codenot stated
verdict and lines

layer_types lists only 'indexed_attention' for every layer, with no sliding-window variant interleaved.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

layer_types only distinguishes full_attention from the sparse-indexer attention type; there is no sliding-window layer variant interleaved with global attention.

self.layer_types = [
                "minimax_m3_sparse" if f else "full_attention" for f in sparse_cfg["sparse_attention_freq"]
            ]
configuration_minimax_m3_vl.py · L142–144
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
sliding_window=null

Since sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention, not windowed vs global attention.

llm_config.sliding_window = null
Hy4-previewnot in its codenot stated
layer_types=["deepseek_sparse_attention", "deep…

There is no interleaving of sliding-window and global layers; every layer_types entry is the same 'deepseek_sparse_attention' type.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
GLM-5.3-Flashnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse…
Qwen3.8-Flash-Nextnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no sliding-window/global interleaving.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
self.layer_types = [
                "linear_attention" if (i + 1) % interval_pattern else "indexed_attention"
                for i in range(self.num_hidden_layers)
            ]
configuration_qwen4_exp.py · L175–178
GLM-5.3not in its codenot stated
verdict and lines

layer_types is uniformly "indexed_attention" for all layers with no sliding/global interleaving pattern present.

        if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_glm_moe_dsa.py · L158–159
DeepSeek-V4-Pro-0813not in its codenot stated
sliding_window=128

All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
sliding_window=null

layers_block_type only distinguishes mamba/moe/attention (full_attention), with no sliding-window layer type or pattern present.

sliding_window = null
DeepSeek-V4-Flash-0731not in its codenot stated
sliding_window=128

All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers.

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern config, and no code that interleaves local and global attention layers.

Kimi K3not in its codenot stated
verdict and lines

There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention.

Hy3not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern config, and all attention layers use the same full causal mask.

GLM-5.2not in its codenot stated
verdict and lines

There is no layer_types/sliding_window_pattern distinguishing local vs global attention; all 78 layers share the same "indexed_attention" type.

if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_glm_moe_dsa.py · L158–159
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
sliding_window=null

There is no sliding-window layer type in layers_block_type (only mamba, moe, attention), so no interleaving of local/global attention occurs.

sliding_window = null
MiniMax-M3not in its codenot stated
verdict and lines

layer_types alternates only between full_attention and the lightning-indexer sparse attention, not sliding-window vs global attention.

MolmoAct 2not in its codenot stated
verdict and lines

No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention.

DeepSeek-V4-Procode unclearnot stated
sliding_window=128

layer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to compressed_sparse_attention/heavily_compressed_attention), but all layer types still apply the same sliding_window base cache, and compress-attention layers additionally concatenate long-range compressed KV rather than attending to the genuinely full context, so whether any layer is truly 'full-context' versus all being sliding-window-based cannot be settled from the given files.

sliding_window = 128
_COMPRESS_RATIO_TO_LAYER_TYPE = {
    0: "sliding_attention",
    4: "compressed_sparse_attention",
    128: "heavily_compressed_attention",
}
configuration_deepseek_v4.py · L28–32
Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

layer_types interleaves linear_attention and full_attention, not sliding-window and global attention; no sliding-window mechanism exists.

causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1346–1349
Qwen3.5-397B-A17Bnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types interleaves full_attention and linear_attention (not sliding-window) layers, so this is a linear/full hybrid, not sliding-window/full.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1346–1349
DeepSeek-V3.2not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern distinguishing local vs global attention; every layer is uniformly 'indexed_attention'.

if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_deepseek_v32.py · L138–139
X-VLAnot in its codenot stated
verdict and lines

There is no interleaving of sliding and full-attention layers; all layers use the same uniform full/causal attention pattern.

OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

No layer_types or sliding_window_pattern configuration exists, and no interleaving code is present; attention layers are uniform.

text_config.model_type = "llama"

In this branch 12

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 12

Hybrid Sliding-Window and Global Attention core · 21 sources · 23 quotes
Gated DeltaNet and full attention core · 7 sources · 9 quotes
Hybrid Mamba-Transformer core · 6 sources · 9 quotes
Hybrid Mamba-Attention core · 3 sources · 4 quotes
Hybrid Sparse and Linear Attention core · 3 sources · 3 quotes
Hybrid compressed attention (CSA and HCA) used · 3 sources · 3 quotes
4 single mentions
Search-Based Sliding-Window Attention Pattern evaluated · 1 source · 1 quote
Dense Attention Fallback unclear · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash coreHybrid Sparse and Linear Attention core—
MiMo-V2.6-Flash coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreHybrid Mamba-Attention coreHybrid Mamba-Transformer core—
DeepSeek-V4-Flash usedHybrid compressed attention (CSA and HCA) used—
DeepSeek-V4-Pro usedHybrid compressed attention (CSA and HCA) used—
GLM-5.2 evaluatedSearch-Based Sliding-Window Attention Pattern evaluated—
Gemma 4 31B coreHybrid Sliding-Window and Global Attention core—
Inkling coreHybrid Sliding-Window and Global Attention core—
Kimi K3 coreLayerwise Hybrid Linear and Global Attention core—
Kolibri-1 coreHybrid Sliding-Window and Global Attention core—
Laguna-S-2.1 coreHybrid Sliding-Window and Global Attention coreGated DeltaNet and full attention used—
MiMo-V2.5 coreHybrid Sliding-Window and Global Attention core—
MiMo-V2.5-Pro coreHybrid Sliding-Window and Global Attention core—
MiMo-V2.6-Pro coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
MiniMax-M3 unclearDense Attention Fallback unclear—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreHybrid Mamba-2 and Mixture-of-Experts Architecture coreHybrid Mamba-Transformer core—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreHybrid Mamba-Transformer core—
Qwen3.5-397B-A17B coreGated DeltaNet and full attention core—
Qwen3.6-35B-A3B coreGated DeltaNet and full attention core—
Qwen3.8-Flash-Next coreGated DeltaNet and full attention core—
SmolVLA coreInterleaved Self-Attention and Cross-Attention core—
gpt-oss-120b usedHybrid Sliding-Window and Global Attention used—
↑