Hybrid Sliding-Window and Global Attention interleaves local sliding-window attention layers with global attention layers in a Transformer.1 Sliding-window layers restrict each token's attention to a local window, cutting compute and memory, while interleaved global layers preserve long-range modeling.2
Trade-off Overly aggressive use of sliding-window attention, such as very small windows or high SWA:GA ratios, can substantially degrade performance, especially on long-context tasks.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA — MiMo-V2-Flash Technical Report
restricts each token’s attention scope to a local window rather than the entire sequence, thereby reducing both computational and memory complexity dramatically — MiMo-V2-Flash Technical Report
overly aggressive use of SWA, such as very small sliding window sizes or high SWA:GA ratios, can lead to substantial degradation in model performance — MiMo-V2-Flash Technical Report
Also called Hybrid Attention, Hybrid Attention Architecture, Hybrid attention (sliding window + global), Hybrid local and global attention, Hybrid local and global attention layers, hybrid of local and global layers.
compress_ratios interleaves layers with compress_ratio 0 (SWA-only, no compressed/global KV) against layers with ratio 2 or 1 (which read a compressed, effectively global, KV cache), giving a fixed interleaving of local and global-context attention.
compress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV source (ratio 1 or 2), giving a fixed interleaving of local and global attention across the 40 layers.
layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types.
interleave = [
"compressed_sparse_attention" if i % 2 else "heavily_compressed_attention"
for i in range(max(n - 2, 0))
]
self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave
local_layer_ids selects most layers as sliding-window (swa_*) while layers not listed (e.g. 5, 11, 17...) use full attention, giving a fixed interleaving of local and global attention layers.
layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each.
layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config.
hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly.
layer_types only distinguishes full_attention from the sparse-indexer attention type; there is no sliding-window layer variant interleaved with global attention.
self.layer_types = [
"minimax_m3_sparse" if f else "full_attention" for f in sparse_cfg["sparse_attention_freq"]
]
Since sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention, not windowed vs global attention.
There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention.
layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no sliding-window/global interleaving.
All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention
All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers.
There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention.
No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention.
layer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to compressed_sparse_attention/heavily_compressed_attention), but all layer types still apply the same sliding_window base cache, and compress-attention layers additionally concatenate long-range compressed KV rather than attending to the genuinely full context, so whether any layer is truly 'full-context' versus all being sliding-window-based cannot be settled from the given files.
Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations reduce the global KV cache footprint by up to 37.5%.
Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations reduce the global KV cache footprint by up to 37.5%.
For attention, we interleave sliding-window and global layers at a 5:1 ratio with 8 KV heads. We find that encoding position with a relative positional embedding
We employ Rotary Positional Encodings (RoPE) [81] to encode positional information. GA layers use 48 Q-heads, θ = 500,000, and partial RoPE applied to the first 50% of the head dimension
Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.
Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window.
The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.
The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.
The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global.