specific method · filed under model architecture
Hybrid Sliding Window Attention
A hybrid attention architecture featuring sliding-window attention; the evidence does not specify the hybrid mechanism.
Also called Hybrid window attention.
- sources
- 4
- models
- 2
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 4core 1
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- MiMo-V2-Flash Technical Report (Xiaomi, 2026) paper arxiv.org§2.2: SWA interleaved with global attention at 5:1, the backbone MiMo-V2.5 inherits
- Sliding Window Attention (SWA) - LLM Architecture Gallery explainer sebastianraschka.comlocal layers interleaved with global ones, and the ratios models use
In the code
Checked as “Interleaved sliding-window and global attention” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | not in its code | not stated | There is no sliding-window layer type in layer_types (only linear_attention and deepseek_sparse_attention), so no interleaving of sliding and global attention exists. |
| DeepSeek-V4.1-Flash | in its code | not stated | The compress_ratios array fixes, per layer, whether it is pure sliding-window (0), full compressed (1), or ratio-2 compressed with sparse attention over the full context (2), interleaving sliding-window-only layers with full/compressed-context layers. attention.py#L300 |
| Hy4-preview | not in its code | not stated | layer_types lists only "deepseek_sparse_attention" for every layer, with no sliding-window layer type interleaved. |
| DeepSeek-V4-Flash-0731 | in its code | not stated | layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, with only sliding_attention layers being pure local-window and the others extending attention to compressed long-range context. modeling_deepseek_v4.py#L777 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | layers_block_type only distinguishes mamba/moe/attention layers with full causal masks; there is no sliding-window layer type interleaved with global attention. |
| MiMo-V2.6-Flash | in its code | not stated | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48 decoder layers. modeling_mimo_v2.py#L400 |
| DeepSeek-V4-Flash | in its code | not stated | layer_types assigns each layer to sliding_attention, compressed_sparse_attention, or heavily_compressed_attention, but all three still apply the same sliding_window cache underneath while only sliding_attention layers lack the additional compressed long-range branch, giving a fixed interleaving of local-only vs local+global-context layers. modeling_deepseek_v4.py#L777 |
| MiMo-V2.5 | in its code | core | hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly. modeling_mimo_v2.py#L400 |
| GLM-5.3 | not in its code | not stated | There is no layer_types pattern distinguishing sliding vs global attention; layer_types is uniformly set to 'indexed_attention' for every layer. configuration_glm_moe_dsa.py#L158 |
| Hy3 | not in its code | not stated | There is only one attention layer type (HYV3Attention) applied uniformly with a full causal mask, no interleaving of local/global layers |
| GLM-5.2 | not in its code | not stated | All layers use the single 'indexed_attention' layer type with the same causal (plus sparse-index) mask; there is no interleaving of sliding-window and global attention layers. configuration_glm_moe_dsa.py#L159 |
| MiniMax-M3 | not in its code | not stated | layer_types alternates between full_attention and the sparse-indexer attention, not between sliding-window and global attention, so this interleaving pattern is absent. configuration_minimax_m3_vl.py#L144 |
| DeepSeek-V3.2 | not in its code | not stated | There is no layer_types pattern mixing sliding and global attention; all layers are uniformly 'indexed_attention'. configuration_deepseek_v32.py#L138 |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | All layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same sliding-window base attention plus optional long-range compressed KV; none of them attend to the full uncompressed context, so there is no interleaving of sliding vs full-global attention layers. modeling_deepseek_v4.py#L751 |
| DeepSeek-V4-Pro | code unclear | not stated | layer_types cycles through sliding_attention/compressed_sparse_attention/heavily_compressed_attention but all three branches apply the same sliding_window cache underneath (CSA/HCA add extra compressed long-range KV on top), so whether any layer truly attends to full context is not settled by the given files, and layer_types itself is not explicitly given in this config (derived by default logic from compress_ratios) |
| DeepSeek-V4-Pro-0813 | in its code | not stated | layer_types (derived from compress_ratios in this config) interleave sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different attention/context scope, per the compress_ratios array mapping to layer types. configuration_deepseek_v4.py#L28 |
| Gemma 4 31B | in its code | not stated | layer_types interleaves five sliding_attention layers followed by one full_attention layer throughout the 60 layers, with separate masks built for each type. modeling_gemma4.py#L1672 |
| Inkling | in its code | not stated | local_layer_ids selects most layers (0,1,2,3,4,6,...) as sliding-window while the remaining layers (5,11,17,23,29,35,41,47,53,59,65) attend globally, an interleaved fixed pattern. model.py#L152 |
| Kimi K3 | not in its code | not stated | There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention. |
| Laguna-S-2.1 | in its code | not stated | layer_types interleaves one full_attention layer followed by three sliding_attention layers repeatedly across all 48 layers. modeling_laguna.py#L560 |
| MiMo-V2.5-Pro | in its code | used | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across 70 layers. modeling_mimo_v2.py#L395 |
| MiMo-V2.6-Pro | in its code | not stated | hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 70 layers. modeling_mimo_v2.py#L400 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | layers_block_type interleaves mamba, moe, and full_attention layers, but none are sliding-window since sliding_window is null. |
| Qwen3.5-397B-A17B | not in its code | not stated | The layer_types interleaving alternates linear_attention and full_attention (global), not sliding-window and global attention. modeling_qwen3_5_moe.py#L1393 |
| Qwen3.6-35B-A3B | not in its code | not stated | The layer_types pattern interleaves full_attention with linear_attention (a recurrent mixer), not with sliding-window attention, so there is no sliding/global interleaving. |
| Qwen3.8-Flash-Next | not in its code | not stated | layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention. configuration_qwen4_exp.py#L190 |
| Step-3.7-Flash | in its code | not stated | layer_types interleaves one full_attention layer every four layers with three sliding_attention layers, and the model builds separate causal masks per type dispatched by layer index. modeling_step3p7.py#L898 |
| gpt-oss-120b | in its code | not stated | layer_types alternates sliding_attention and full_attention across the 36 layers, with separate masks built for each. modeling_gpt_oss.py#L473 |
Evidence
5 spans quoted from the sources, strongest treatment first.
through its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and the MOPD post-training paradigm.
coremodel architecturein MiMo-V2-FlashXiaomi
Equipped with a 729M-param Vision Transformer (ViT) featuring hybrid window attention
usedmodel architecturein MiMo-V2.5Xiaomi
We train a dedicated MiMo ViT that adopts sliding-window attention to enable efficient visual encoding.
usedmodel architecturein MiMo-V2.5Xiaomi
featuring hybrid window attention
usedmodel architecturein MiMo-V2.5Xiaomi
Its language backbone inherits from MiMo-V2-Flash's hybrid sliding-window attention architecture
usedmodel architecturein MiMo-V2.5Xiaomi
Filed alongside
Other methods under model architecture :: token mixer :: softmax attention :: sliding window attention.
Sliding Window Attention512-token Sliding Window AttentionDecoder SWA Bounded ReplayAlternating Row-Major and Column-Major Token SerializationDynamic Attention Window Size TrainingFixed Sliding Window AttentionPer-Head GatingPure Sliding Window AttentionSink-Augmented Sliding Window AttentionSWA-1024SWA-128