general family · filed under model architecture
Sliding Window Attention
Attention restricted to a local sliding window; the evidence does not specify a particular window size.
Also called SWA, Sliding Window Attention (SWA), Sliding Window Attention (SWA) Interleave.
- sources
- 8
- models
- 7
- labs adopt it
- 2
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
mentioned 1evaluated 1used 5core 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Mistral 7B (Jiang et al., 2023) paper arxiv.orgpopularized sliding-window attention in open LLMs
- Sliding Window Attention (SWA) - LLM Architecture Gallery explainer sebastianraschka.com
In the code
Checked as “Sliding-window attention” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | not in its code | not stated | No sliding_window config key or windowed masking mechanism exists anywhere in this model's code; layer types are only linear_attention and deepseek_sparse_attention (indexed_attention). |
| DeepSeek-V4.1-Flash | in its code | core | Every layer maintains a sliding-window KV cache (DeepseekV4SWACache) sized from config.sliding_window=128, and some layers (compress_ratio=0) are pure sliding-window attention. attention.py#L292 |
| Hy4-preview | not in its code | not stated | No sliding_window config key or windowed masking code is present; all layers use the DeepSeek sparse (indexer-selected) attention over the full causal context. |
| DeepSeek-V4-Flash-0731 | in its code | not stated | sliding_window=128 is used by all layer types' cache update (DeepseekV4HCACache/base sliding window) and passed into attention as the local window. modeling_deepseek_v4.py#L784 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | sliding_window is null in the config and the attention code has no window-masking mechanism applied. |
| MiMo-V2.6-Flash | in its code | not stated | Layers flagged 1 in hybrid_layer_pattern are is_swa layers whose sliding_window is set to the non-null config value 128, and a sliding-window causal mask is built for them. modeling_mimo_v2.py#L262 |
| DeepSeek-V4-Flash | in its code | used | sliding_window is set to 128 and is used unconditionally in every attention layer's cache/window mechanism. modeling_deepseek_v4.py#L784 |
| MiMo-V2.5 | in its code | core | sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for those layers. modeling_mimo_v2.py#L262 |
| GLM-5.3 | not in its code | not stated | No sliding_window config key or windowed mask logic appears anywhere in the modeling or configuration code; attention is full causal (with the DSA sparse mask) rather than local-window. |
| Hy3 | not in its code | not stated | The model builds a plain causal mask with no sliding window mechanism, as noted explicitly in the code comment modeling_hy_v3.py#L500 |
| GLM-5.2 | not in its code | evaluated | There is no sliding_window config key or windowed masking logic anywhere in the attention code; masking is a single causal mask ('indexed_attention') applied uniformly. |
| MiniMax-M3 | not in its code | mentioned | No sliding_window config key or windowed-mask code exists; layer_types only distinguish full_attention vs the sparse indexer layers, not a local window. |
| DeepSeek-V3.2 | not in its code | not stated | No sliding_window key or windowed masking mechanism appears anywhere in the config or attention code; every layer is 'indexed_attention' with full causal masking modulated only by the sparse indexer. |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | Every attention layer caches and attends only over the last sliding_window=128 tokens via DynamicSlidingWindowLayer / DeepseekV4HCACache.update. modeling_deepseek_v4.py#L205 |
| DeepSeek-V4-Pro | in its code | used | sliding_window is 128 and every attention layer type (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) uses a sliding-window K/V cache with this window size modeling_deepseek_v4.py#L203 |
| DeepSeek-V4-Pro-0813 | in its code | not stated | Every layer, including compressed layers, uses a sliding-window cache/mask with sliding_window=128 set in config, and the mask is built with create_sliding_window_causal_mask. modeling_deepseek_v4.py#L1350 |
| Gemma 4 31B | in its code | not stated | Most layers are typed 'sliding_attention' with sliding_window=1024 enforced through the sliding-window causal mask and passed to the attention interface. modeling_gemma4.py#L1170 |
| Inkling | in its code | not stated | Layers marked local (per local_layer_ids) use a sliding window of size sliding_window_size (512) via the FA4 window_size tuple. attention.py#L150 |
| Kimi K3 | not in its code | not stated | No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers. |
| Laguna-S-2.1 | in its code | not stated | Layers marked sliding_attention in layer_types use sliding_window=512 to restrict attention to a local window. modeling_laguna.py#L360 |
| MiMo-V2.5-Pro | in its code | used | SWA layers (marked by hybrid_layer_pattern==1) use sliding_window=128 to build a sliding-window causal mask. modeling_mimo_v2.py#L257 |
| MiMo-V2.6-Pro | in its code | not stated | sliding_window is set to 128 and layers flagged 1 in hybrid_layer_pattern use is_swa attention with that window via create_sliding_window_causal_mask. modeling_mimo_v2.py#L262 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | sliding_window is null in this config and no windowed mask is constructed for full_attention layers. |
| Qwen3.5-397B-A17B | not in its code | not stated | There is no sliding_window config key or windowed mask logic in the code; full_attention layers use the standard causal mask. |
| Qwen3.6-35B-A3B | not in its code | not stated | There is no sliding_window config key or masking mechanism for local windows in the text model; full_attention layers use the plain causal mask. modeling_qwen3_5_moe.py#L1393 |
| Qwen3.8-Flash-Next | not in its code | not stated | No sliding_window config key or window-masking code exists; layer_types only distinguish linear_attention and full/indexed attention, none windowed. |
| Step-3.7-Flash | in its code | not stated | layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from config.sliding_window when the layer is local. modeling_step3p7.py#L703 |
| gpt-oss-120b | in its code | not stated | Layers marked sliding_attention use a 128-token sliding window mask. modeling_gpt_oss.py#L292 |
Evidence
10 spans quoted from the sources, strongest treatment first.
DeepSeek-V4.1-Flash, like DeepSeek-V4, uses Sliding-Window Attention (SWA) in every layer.
coremodel architecturein DeepSeek-V4.1-FlashDeepSeek
The first two encoder layers use sliding window attention (SWA); the rest use Compressed Sparse Attention 2 (CSA2)
coremodel architecturein DeepSeek-V4.1-FlashDeepSeek
Sliding Window Attention (SWA)
coremodel architecturein MiMo-V2-FlashXiaomi
we introduce a supplementary attention branch to both CSA and HCA in a sliding window manner
usedmodel architecturein DeepSeek-V4DeepSeek
Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 6:1 ratio
usedmodel architecturein MiMo-V2.5-ProXiaomi
interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio
usedmodel architecturein MiMo-V2.5Xiaomi
Audio Encoder: 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full)
usedmodel architecturein MiMo-V2.5Xiaomi
interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window
usedmodel architecturein MiMo-V2.5Xiaomi
Sliding Window Attention (SWA) Interleave: A fixed alternating pattern of full-attention and windowed-attention layers applied uniformly across the network.
evaluatedmodel architecturein GLM-5Z.ai
fixed-pattern attention keeps softmax attention but imposes a predefined support, including local windows
mentionedother
Filed alongside
Other methods under model architecture :: token mixer :: softmax attention :: sliding window attention.
Hybrid Sliding Window Attention512-token Sliding Window AttentionDecoder SWA Bounded ReplayAlternating Row-Major and Column-Major Token SerializationDynamic Attention Window Size TrainingFixed Sliding Window AttentionPer-Head GatingPure Sliding Window AttentionSink-Augmented Sliding Window AttentionSWA-1024SWA-128