Sliding Window Attention is an attention variant that restricts each token's attention to a fixed-size window of preceding tokens instead of all preceding tokens.1 It reduces memory usage and computational cost, since the KV cache and attention compute no longer grow with the full sequence length.2
Trade-off It lowers long-context inference memory and compute compared with full attention, at the cost of each token not attending to all preceding tokens.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
SWA (Beltagy, Peters, and Cohan 2020) layers, which restrict attention to a fixed-size window of preceding tokens — tech report
reducing both memory usage and computational cost — tech report
each token attends to all preceding tokens, so the compute grows quadratically with the sequence length in all layers — tech report
Also called fixed sliding window attention, sliding-window baseline, pure sliding window attention, SWA, Sliding Window Attention (SWA), Sliding Window Attention (SWA) Interleave.
sliding_window is set to 128 and the decoder uses a dedicated swa_cache_layer with a bounded window for attention layers, as referenced throughout the model code.
enable_sliding_window = layer_type == "sliding_attention"
if yarn_only_types and layer_type not in yarn_only_types:
rope_scaling = None
if sliding_window is not None and enable_sliding_window:
Layers flagged 1 in hybrid_layer_pattern are is_swa layers whose sliding_window is set to the non-null config value 128, and a sliding-window causal mask is built for them.
Layers whose layer_types entry is sliding_attention set self.sliding_window = config.sliding_window (512), restricting attention to a local window via create_sliding_window_causal_mask.
Layers marked local (local_layer_ids) use a sliding window of size sliding_window_size=512 implemented via the FA4 window_size and SlidingWindowSpec KV cache.
layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from config.sliding_window when the layer is local.
Every attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches and the attention call passes sliding_window to the interface.
No sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly 'deepseek_sparse_attention'.
No sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either linear-attention or full indexed/sparse attention over the whole context.
There is no sliding_window config key or windowed mask logic in the attention/masking code for this model; layer_types only distinguishes linear_attention and indexed_attention (full attention with an indexer), with no local window.
No sliding_window config key or windowed mask logic exists in the modeling or config files; attention uses the full causal mask with indexer-based sparsity instead.
No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers.
No sliding_window key or windowed masking mechanism appears anywhere in the config or code; all layers use the full causal (plus indexer top-k sparse) mask.
No sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask with no windowing logic.
No sliding-window masking mechanism is implemented in the Florence2 text attention or the action transformer attention; attention is either full bidirectional (encoder) or full causal (decoder).
the sliding-window-heavy architecture helps. Of the 66 decoder layers, 55 use local attention with a small 512-token window. A learned relative-position bias may provide enough positional information within these windows.
Sliding Window Attention (SWA) Interleave: A fixed alternating pattern of full-attention and windowed-attention layers applied uniformly across the network.