Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

general family · filed under model architecture

Hybrid Sliding-Window and Global Attention

Hybrid Sliding-Window and Global Attention interleaves local sliding-window attention layers with global attention layers in a Transformer.1 Sliding-window layers restrict each token's attention to a local window, cutting compute and memory, while interleaved global layers preserve long-range modeling.2

Trade-off Overly aggressive use of sliding-window attention, such as very small windows or high SWA:GA ratios, can substantially degrade performance, especially on long-context tasks.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA — MiMo-V2-Flash Technical Report
  2. restricts each token’s attention scope to a local window rather than the entire sequence, thereby reducing both computational and memory complexity dramatically — MiMo-V2-Flash Technical Report
  3. overly aggressive use of SWA, such as very small sliding window sizes or high SWA:GA ratios, can lead to substantial degradation in model performance — MiMo-V2-Flash Technical Report

Also called Hybrid Attention, Hybrid Attention Architecture, Hybrid attention (sliding window + global), Hybrid local and global attention, Hybrid local and global attention layers, hybrid of local and global layers.

sources
21
models
9
labs adopt it
6
strongest
core

In practice

Checked as “Interleaved sliding-window and global attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codenot stated
layer_types=["conv", "conv", "full_attention", …

layer_types interleaves full_attention and sliding_attention layers (plus conv layers) in a fixed pattern, with separate mask builders per type.

layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full…
"full_attention": lambda: create_causal_mask(**mask_kwargs),
                "sliding_attention": lambda: create_sliding_window_causal_mask(**mask_kwargs),
                "conv": lambda: create_recurrent_attention_mask(**mask_kwargs),
modeling_lfm3.py · L553–555
DeepSeek-V4.1-Flash-1017in its codenot stated
compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…

compress_ratios interleaves layers with compress_ratio 0 (SWA-only, no compressed/global KV) against layers with ratio 2 or 1 (which read a compressed, effectively global, KV cache), giving a fixed interleaving of local and global-context attention.

text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…
swa_only = self.compress_ratio == 0
        # SWA-only layers have no compressed KV cache; consumers read the kv
        # source's cache.
        self_kv_cache = None if swa_only else self._compressed_kv_cache()
flashinfer_sparse.py · L313–316
Step-5-Previewin its codenot stated
layer_types=["sliding_attention", "sliding_atte…

layer_types fixes a repeating pattern of three sliding_attention layers followed by one full_attention layer across the 92-layer stack.

text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti…
default_layer_type = (
            "sliding_attention" if self.layer_idx % 2 == 0 else "full_attention"
        )
        layer_type = _per_layer_value(
            layer_types,
            self.layer_idx,
            name="layer_types",
            default=default_layer_type,
        )
model.py · L886–894
Kolibri-1in its codeadopted
layer_types=["sliding_attention", "sliding_atte…

layer_types interleaves four sliding_attention layers followed by one full_attention layer, repeating across all 50 layers.

layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
            causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_kolibri1.py · L498–501
MiMo-V2.6-Flashin its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48 decoder layers.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
MiMo-V2.6-Proin its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1…

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 70 layers.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
DeepSeek-V4.1-Flashin its codenot stated
compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2… · kv_source_layer_ids=[2, 8, 14, 20]

compress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV source (ratio 1 or 2), giving a fixed interleaving of local and global attention across the 40 layers.

text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…text_config.kv_source_layer_ids = [2, 8, 14, 20]
        cut = max(config.kv_source_layer_ids)
        if (
            cut < self.end_layer - 1
            and self._decoder_replay_supported(vllm_config, cut)
model.py · L882–885
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types.

interleave = [
                "compressed_sparse_attention" if i % 2 else "heavily_compressed_attention"
                for i in range(max(n - 2, 0))
            ]
            self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave
configuration_deepseek_v4.py · L270–274
self.layer_type = config.layer_types[layer_idx]
modeling_deepseek_v4.py · L785
Laguna-S-2.1in its codeadopted
layer_types=["full_attention", "sliding_attenti…

layer_types fixes a pattern of one full_attention layer followed by three sliding_attention layers, repeated across all 48 layers.

layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
mask_creation_functions = {
                "full_attention": lambda: create_causal_mask(**mask_kwargs),
                "sliding_attention": lambda: create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_laguna.py · L549–552
Inklingin its codeadopted
local_layer_ids=[0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12,… · num_hidden_layers=66

local_layer_ids selects most layers as sliding-window (swa_*) while layers not listed (e.g. 5, 11, 17...) use full attention, giving a fixed interleaving of local and global attention layers.

text_config.local_layer_ids = [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 2…text_config.num_hidden_layers = 66
self.attn = InklingAttention(
            config,
            num_heads=(
                config.swa_num_attention_heads
                if is_local
                else config.num_attention_heads
            ),
model.py · L152–158
Step-3.7-Flashin its codenot stated
layer_types=["full_attention", "sliding_attenti…

layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each.

text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**sliding_mask_kwargs),
            }
modeling_step3p7.py · L889–892
DeepSeek-V4-Flashin its codenot stated
compress_ratios=[0, 0, 4, 128, 4, 128, 4, 128, 4, 1…

layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config.

compress_ratios = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128…
self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
MiMo-V2.5in its codeadopted
hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly.

hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L400–401
MiMo-V2.5-Proin its codeadopted
verdict and lines

hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across 70 layers.

is_swa_layer = config.hybrid_layer_pattern[layer_idx] == 1
        self.attention_type = "sliding_window_attention" if is_swa_layer else "full_attention"
modeling_mimo_v2.py · L395–396
Gemma 4 31Bin its codeadopted
layer_types=["sliding_attention", "sliding_atte…

layer_types interleaves 5 sliding_attention layers followed by 1 full_attention layer repeatedly across the 60 layers.

text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gemma4.py · L1658–1661
gpt-oss-120bin its codeadopted
layer_types=["sliding_attention", "full_attenti…

layer_types alternates sliding_attention and full_attention across the 36 layers, each built with its own causal mask

layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gpt_oss.py · L466–469
Mistral-Large-4not in its codenot stated
verdict and lines

layer_types lists only 'indexed_attention' for every layer, with no sliding-window variant interleaved.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

layer_types only distinguishes full_attention from the sparse-indexer attention type; there is no sliding-window layer variant interleaved with global attention.

self.layer_types = [
                "minimax_m3_sparse" if f else "full_attention" for f in sparse_cfg["sparse_attention_freq"]
            ]
configuration_minimax_m3_vl.py · L142–144
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
sliding_window=null

Since sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention, not windowed vs global attention.

llm_config.sliding_window = null
Hy4-previewnot in its codenot stated
layer_types=["deepseek_sparse_attention", "deep…

There is no interleaving of sliding-window and global layers; every layer_types entry is the same 'deepseek_sparse_attention' type.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
GLM-5.3-Flashnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse…
Qwen3.8-Flash-Nextnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no sliding-window/global interleaving.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
self.layer_types = [
                "linear_attention" if (i + 1) % interval_pattern else "indexed_attention"
                for i in range(self.num_hidden_layers)
            ]
configuration_qwen4_exp.py · L175–178
GLM-5.3not in its codenot stated
verdict and lines

layer_types is uniformly "indexed_attention" for all layers with no sliding/global interleaving pattern present.

        if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_glm_moe_dsa.py · L158–159
DeepSeek-V4-Pro-0813not in its codenot stated
sliding_window=128

All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
sliding_window=null

layers_block_type only distinguishes mamba/moe/attention (full_attention), with no sliding-window layer type or pattern present.

sliding_window = null
DeepSeek-V4-Flash-0731not in its codenot stated
sliding_window=128

All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers.

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern config, and no code that interleaves local and global attention layers.

Kimi K3not in its codenot stated
verdict and lines

There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention.

Hy3not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern config, and all attention layers use the same full causal mask.

GLM-5.2not in its codenot stated
verdict and lines

There is no layer_types/sliding_window_pattern distinguishing local vs global attention; all 78 layers share the same "indexed_attention" type.

if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_glm_moe_dsa.py · L158–159
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
sliding_window=null

There is no sliding-window layer type in layers_block_type (only mamba, moe, attention), so no interleaving of local/global attention occurs.

sliding_window = null
MiniMax-M3not in its codenot stated
verdict and lines

layer_types alternates only between full_attention and the lightning-indexer sparse attention, not sliding-window vs global attention.

MolmoAct 2not in its codenot stated
verdict and lines

No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention.

DeepSeek-V4-Procode unclearnot stated
sliding_window=128

layer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to compressed_sparse_attention/heavily_compressed_attention), but all layer types still apply the same sliding_window base cache, and compress-attention layers additionally concatenate long-range compressed KV rather than attending to the genuinely full context, so whether any layer is truly 'full-context' versus all being sliding-window-based cannot be settled from the given files.

sliding_window = 128
_COMPRESS_RATIO_TO_LAYER_TYPE = {
    0: "sliding_attention",
    4: "compressed_sparse_attention",
    128: "heavily_compressed_attention",
}
configuration_deepseek_v4.py · L28–32
Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

layer_types interleaves linear_attention and full_attention, not sliding-window and global attention; no sliding-window mechanism exists.

causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1346–1349
Qwen3.5-397B-A17Bnot in its codenot stated
layer_types=["linear_attention", "linear_attent…

layer_types interleaves full_attention and linear_attention (not sliding-window) layers, so this is a linear/full hybrid, not sliding-window/full.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1346–1349
DeepSeek-V3.2not in its codenot stated
verdict and lines

There is no layer_types or sliding_window_pattern distinguishing local vs global attention; every layer is uniformly 'indexed_attention'.

if self.layer_types is None:
            self.layer_types = ["indexed_attention"] * self.num_hidden_layers
configuration_deepseek_v32.py · L138–139
X-VLAnot in its codenot stated
verdict and lines

There is no interleaving of sliding and full-attention layers; all layers use the same uniform full/causal attention pattern.

OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

No layer_types or sliding_window_pattern configuration exists, and no interleaving code is present; attention layers are uniform.

text_config.model_type = "llama"

Documented in

Evidence

23 quotes from 21 sources, one card per model, each opening with its strongest.

How sources treat it
used 5core 18

Inkling

Thinking Machines Labadopted4 sources

Attention is a hybrid of local and global layers.

coremodel architecturein InklingThinking Machines Lab
3 more from this model

Attention is a hybrid of local and global layers.

coremodel architecturein InklingThinking Machines Lab

Attention is a hybrid of local and global layers

coremodel architecturein InklingThinking Machines Lab

For attention, we interleave sliding-window and global layers at a 5:1 ratio with 8 KV heads.

usedmodel architecturein InklingThinking Machines Lab

MiMo-V2.5

Xiaomiadopted4 sources

Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.

coremodel architecturein MiMo-V2.5Xiaomi
3 more from this model

Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.

coremodel architecturein MiMo-V2.5Xiaomi

MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention

coremodel architecturein MiMo-V2-FlashXiaomi

hybrid attention (SWA-128 + full-attention at 5:1 ratio)

usedmodel architecturein MiMo-V2.5Xiaomi

Gemma 4 31B

Google DeepMindadopted4 sources

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

coremodel architecturein Gemma 4Google DeepMind
3 more from this model

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

coremodel architecturein Gemma 4Google DeepMind

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.

coremodel architecturein Gemma 4Google DeepMind

We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3B model)

usedmodel architecturein Gemma 4Google DeepMind

Kolibri-1

Aleph Alphaadopted3 sources

Kolibri uses full attention in every fifth block (10 of 50) and SWA in the other 40.

coremodel architecturein KolibriAleph Alpha
2 more from this model

Out of 50 total layers, only 10 process full context, while the remaining 40 use a tight 512-token focused window.

coremodel architecturein KolibriAleph Alpha

most attention layers focus on nearby text, while a smaller number attend across the whole context.

coremodel architecturein KolibriAleph Alpha

MiMo-V2.6-Pro

Xiaomiadopted2 sources

repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA)

coremodel architecturein MiMo-V2.6Xiaomi
3 more from this model

a Transformer with a causal hybrid attention architecture that interleaves SWA and GA layers

coremodel architecturein MiMo-V2.6 Audio TokenizerXiaomi

MiMo-V2.6 interleaves 128-token sliding-window layers with full attention.

coremodel architecturein MiMo-V2.6Xiaomi

Remaining blocks interleave local SWA and GA

coremodel architecturein MiMo-V2.6-Pro-RLXiaomi

Laguna-S-2.1

Poolsideadopted2 sources

Mixed SWA and global attention layout: 48 layers in a 1:3 global-to-SWA ratio (12 global attention layers, 36 sliding-window layers, window 512)

coremodel architecturein Laguna S 2.1Poolside
1 more from this model

LAGUNA XS.2 uses interleaved Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio.

usedmodel architecturein Laguna XS.2Poolside

MiMo-V2.6-Flash

Xiaomiadopted1 source

repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA)

coremodel architecturein MiMo-V2.6Xiaomi
2 more from this model

a Transformer with a causal hybrid attention architecture that interleaves SWA and GA layers

coremodel architecturein MiMo-V2.6 Audio TokenizerXiaomi

MiMo-V2.6 interleaves 128-token sliding-window layers with full attention.

coremodel architecturein MiMo-V2.6Xiaomi

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑