Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionsliding window attention

general family · filed under model architecture

Sliding Window Attention

Sliding Window Attention is an attention variant that restricts each token's attention to a fixed-size window of preceding tokens instead of all preceding tokens.1 It reduces memory usage and computational cost, since the KV cache and attention compute no longer grow with the full sequence length.2

Trade-off It lowers long-context inference memory and compute compared with full attention, at the cost of each token not attending to all preceding tokens.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. SWA (Beltagy, Peters, and Cohan 2020) layers, which restrict attention to a fixed-size window of preceding tokens — tech report
  2. reducing both memory usage and computational cost — tech report
  3. each token attends to all preceding tokens, so the compute grows quadratically with the sequence length in all layers — tech report

Also called fixed sliding window attention, sliding-window baseline, pure sliding window attention, SWA, Sliding Window Attention (SWA), Sliding Window Attention (SWA) Interleave.

sources
12
models
10
labs adopt it
5
strongest
core

In practice

Checked as “Sliding-window attention” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codeadopted
sliding_window=256 · layer_types=["conv", "conv", "full_attention", …

Layers typed sliding_attention set self.sliding_window = config.sliding_window = 256 and a sliding window mask is built for them.

sliding_window = 256layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full…
self.sliding_window = config.sliding_window if self.layer_type == "sliding_attention" else None
modeling_lfm3.py · L237
DeepSeek-V4.1-Flash-1017in its codenot stated
sliding_window=128

sliding_window is set to 128 and the decoder uses a dedicated swa_cache_layer with a bounded window for attention layers, as referenced throughout the model code.

text_config.sliding_window = 128
self.decoder_replay_layers = DecoderReplayLayers(
                config.sliding_window,
                self._run_replay_layers,
model.py · L901–903
Step-5-Previewin its codenot stated
sliding_window=512 · layer_types=["sliding_attention", "sliding_atte…

layer_types alternates and sliding_window=512 is set, enabling local-window attention on layers marked sliding_attention.

text_config.sliding_window = 512text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti…
enable_sliding_window = layer_type == "sliding_attention"
        if yarn_only_types and layer_type not in yarn_only_types:
            rope_scaling = None

        if sliding_window is not None and enable_sliding_window:
model.py · L895–899
Kolibri-1in its codeadopted
sliding_window=513 · use_sliding_window=true

use_sliding_window is true and sliding_window=513, and layers tagged sliding_attention apply this window via create_sliding_window_causal_mask.

use_sliding_window = truesliding_window = 513
        self.sliding_window = config.sliding_window
        layer_type = config.layer_types[layer_idx] if hasattr(config, "layer_types") else None
        self.is_sliding = layer_type == "sliding_attention"
modeling_kolibri1.py · L149–151
MiMo-V2.6-Flashin its codenot stated
sliding_window=128 · hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

Layers flagged 1 in hybrid_layer_pattern are is_swa layers whose sliding_window is set to the non-null config value 128, and a sliding-window causal mask is built for them.

sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L262
causal_mask_mapping["sliding_window_attention"] = create_sliding_window_causal_mask(**mask_kwargs)
modeling_mimo_v2.py · L1657
MiMo-V2.6-Proin its codenot stated
sliding_window=128 · hybrid_layer_pattern=[0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1…

sliding_window is set to 128 and layers flagged 1 in hybrid_layer_pattern use is_swa attention with that window via create_sliding_window_causal_mask.

sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1…
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L262
DeepSeek-V4.1-Flashin its codenot stated
sliding_window=128

text_config.sliding_window is 128 and the model builds an SWA KV cache layer (swa_cache_layer) used across decoder layers and decoder-replay logic.

text_config.sliding_window = 128
            self.decoder_replay_layers = DecoderReplayLayers(
                config.sliding_window,
                self._run_replay_layers,
model.py · L901–903
DeepSeek-V4-Flash-Vision-Expin its codenot stated
sliding_window=128

Every attention layer applies a sliding-window KV cache of size config.sliding_window=128 in addition to any compressor branch.

sliding_window = 128
self.cumulative_length += key_states.shape[-2]
        full = torch.cat([self.keys, key_states], dim=-2)
        self.keys = full[:, :, -self.sliding_window + 1 :, :]
modeling_deepseek_v4.py · L197–199
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L792
DeepSeek-V4-Pro-0813in its codenot stated
sliding_window=128

Every attention layer uses a sliding-window KV cache (DynamicSlidingWindowLayer) with sliding_window=128 passed to the attention interface

sliding_window = 128
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L792
sliding_window=self.sliding_window,
modeling_deepseek_v4.py · L868
DeepSeek-V4-Flash-0731in its codenot stated
sliding_window=128

Every attention layer applies a sliding-window KV cache of size sliding_window=128 regardless of layer type.

sliding_window = 128
self.cumulative_length += key_states.shape[-2]
        full = torch.cat([self.keys, key_states], dim=-2)
        self.keys = full[:, :, -self.sliding_window + 1 :, :]
modeling_deepseek_v4.py · L197–199
Laguna-S-2.1in its codenot stated
sliding_window=512 · layer_types=["full_attention", "sliding_attenti…

Layers whose layer_types entry is sliding_attention set self.sliding_window = config.sliding_window (512), restricting attention to a local window via create_sliding_window_causal_mask.

sliding_window = 512layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
self.is_local_attention = config.layer_types[layer_idx] == "sliding_attention"
        self.sliding_window = config.sliding_window if self.is_local_attention else None
modeling_laguna.py · L349–350
Inklingin its codeadopted
sliding_window_size=512 · local_layer_ids=[0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12,…

Layers marked local (local_layer_ids) use a sliding window of size sliding_window_size=512 implemented via the FA4 window_size and SlidingWindowSpec KV cache.

text_config.sliding_window_size = 512text_config.local_layer_ids = [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 2…
self.window_size: tuple[int, int] = (
            (local_extent - 1, 0) if is_local else (-1, -1)
        )
attention.py · L150–152
Step-3.7-Flashin its codenot stated
sliding_window=512 · layer_types=["full_attention", "sliding_attenti…

layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from config.sliding_window when the layer is local.

text_config.sliding_window = 512text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
self.is_local_attention = config.layer_types[layer_idx] == "sliding_attention"
        self.sliding_window = config.sliding_window if self.is_local_attention else None
modeling_step3p7.py · L693–694
DeepSeek-V4-Flashin its codeadopted
sliding_window=128

Every attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches and the attention call passes sliding_window to the interface.

sliding_window = 128
self.keys = full[:, :, -self.sliding_window + 1 :, :]
        self.values = self.keys
modeling_deepseek_v4.py · L199–200
DeepSeek-V4-Proin its codeadopted
sliding_window=128

Every attention layer (sliding_attention, CSA, HCA) uses a sliding-window cache of size config.sliding_window=128 as its base local K/V window.

sliding_window = 128
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L792
MiMo-V2.5in its codeadopted
sliding_window=128 · hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…

sliding_window is 128 and the hybrid_layer_pattern marks most layers as SWA (value 1), enabling create_sliding_window_causal_mask for those layers.

sliding_window = 128hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1…
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L262
MiMo-V2.5-Proin its codeadopted
sliding_window=128

SWA layers (marked by hybrid_layer_pattern==1) use sliding_window=128 to build a sliding-window causal mask.

sliding_window = 128
self.sliding_window = getattr(config, "sliding_window", None) if is_swa else None
modeling_mimo_v2.py · L257
Gemma 4 31Bin its codenot stated
sliding_window=1024

Sliding-attention layers restrict queries with sliding_window=1024 via create_sliding_window_causal_mask and the sliding_window passed into attention.

text_config.sliding_window = 1024
self.is_sliding = self.layer_type == "sliding_attention"
        self.sliding_window = config.sliding_window if self.is_sliding else None
modeling_gemma4.py · L1158–1159
"full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
modeling_gemma4.py · L1659–1660
gpt-oss-120bin its codenot stated
sliding_window=128 · layer_types=["sliding_attention", "full_attenti…

Layers tagged sliding_attention use sliding_window=128 as the local window size

sliding_window = 128layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
self.sliding_window = config.sliding_window if self.layer_type == "sliding_attention" else None
modeling_gpt_oss.py · L284
Mistral-Large-4not in its codenot used
verdict and lines

No sliding_window key or windowed masking mechanism appears in the config or code; all layers are 'indexed_attention' with full causal masking.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

Layer types are only "full_attention" or the sparse indexer type; there is no sliding-window mechanism or config key enabling a local window.

self.layer_types = [
                "minimax_m3_sparse" if f else "full_attention" for f in sparse_cfg["sparse_attention_freq"]
            ]
configuration_minimax_m3_vl.py · L142–144
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot stated
sliding_window=null

sliding_window is explicitly null in the text_config, so no sliding-window restriction is applied even if supported elsewhere.

llm_config.sliding_window = null
Hy4-previewnot in its codenot stated
layer_types=["deepseek_sparse_attention", "deep…

No sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly 'deepseek_sparse_attention'.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
GLM-5.3-Flashnot in its codenot stated
verdict and lines

No sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either linear-attention or full indexed/sparse attention over the whole context.

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

There is no sliding_window config key or windowed mask logic in the attention/masking code for this model; layer_types only distinguishes linear_attention and indexed_attention (full attention with an indexer), with no local window.

GLM-5.3not in its codenot stated
verdict and lines

No sliding_window config key or windowed mask logic exists in the modeling or config files; attention uses the full causal mask with indexer-based sparsity instead.

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
sliding_window=null

sliding_window is null in this config, so no attention layer restricts tokens to a local window.

sliding_window = null
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No sliding_window config key or windowed-mask code is present for the language model; attention is full causal attention.

Kimi K3not in its codenot stated
verdict and lines

No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers.

Hy3not in its codenot stated
verdict and lines

The model builder explicitly uses a full causal mask with no sliding window mechanism.

# No sliding window opposed to mixtral
        causal_mask = create_causal_mask(
modeling_hy_v3.py · L489–490
GLM-5.2not in its codediscussed
verdict and lines

No sliding_window key or windowed masking mechanism appears anywhere in the config or code; all layers use the full causal (plus indexer top-k sparse) mask.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
sliding_window=null

sliding_window is null in this config and no windowed masking code is used for the full-attention layers.

sliding_window = null
MiniMax-M3not in its codediscussed
verdict and lines

No sliding_window config key or windowed masking code is present; layer_types only distinguishes full_attention from the sparse indexer layers.

MolmoAct 2not in its codenot stated
verdict and lines

No sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask with no windowing logic.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

There is no sliding_window config key or windowed mask logic in the attention code; full_attention layers use the standard causal mask.

Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

There is no sliding_window config key or windowed-mask code; attention is either full or linear (recurrent), not local-window.

DeepSeek-V3.2not in its codenot stated
verdict and lines

No sliding_window config key or windowed masking logic exists anywhere in this model's attention implementation.

X-VLAnot in its codenot stated
verdict and lines

No sliding-window masking mechanism is implemented in the Florence2 text attention or the action transformer attention; attention is either full bidirectional (encoder) or full causal (decoder).

OpenVLA / OpenVLA-OFTnot in its codenot stated
model_type=llama

There is no sliding_window, use_sliding_window, or layer_types key in text_config, and no windowed-attention code is present in the given files.

text_config.model_type = "llama"

Documented in

Aleph Alpha
Liquid AI
Thinking Machines Lab
MiniMax
Z.ai (Zhipu AI)
Mistral AI

Evidence

16 quotes from 12 sources, one card per model, each opening with its strongest.

How sources treat it
mentioned 1evaluated 2used 7core 5not used 1

MiMo-V2.5

Xiaomiadopted3 sources

interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window

usedmodel architecturein MiMo-V2.5Xiaomi
3 more from this model

interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio

usedmodel architecturein MiMo-V2.5Xiaomi

Audio Encoder: 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full)

usedmodel architecturein MiMo-V2.5Xiaomi

Sliding Window Attention (SWA)

coremodel architecturein MiMo-V2-FlashXiaomi

Kolibri-1

Aleph Alphaadopted1 source

restrict attention to a fixed-size window of preceding tokens, reducing both memory usage and computational cost

coremodel architecturein KolibriAleph Alpha
2 more from this model

40 of its 50 blocks use sliding-window attention with RoPE

coreunclearin KolibriAleph Alpha

We therefore use a sliding window of w = 512 in Kolibri

usedunclearin KolibriAleph Alpha

MiniMax-M3

MiniMaxdiscussed1 source

fixed-pattern attention keeps softmax attention but imposes a predefined support, including local windows

mentionedother
1 more from this model

a FLOP-matched sliding-window baseline

evaluatedmodel architecturein MiniMax Sparse AttentionMiniMax

Mistral-Large-4

Mistral AInot used1 source

the remaining layers use sliding-window attention (SWA) with a window size of 512 tokens

not usedmodel architecturein Mistral Large 4 initial architectureMistral AI

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑