Model techniques map
Taxonomymodel architecturetoken mixersoftmax attentionsliding window attention

taxonomy node · level 4

sliding window attention

12 methods filed at this node or below it, from the sources of 12 models.

model architecture :: token mixer :: softmax attention :: sliding window attention

Matching aids for the classifier: SWA; local window; window attention; hybrid window attention.

In this branch 12

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 12

Sliding Window Attention core · 8 sources · 10 quotes
Hybrid Sliding Window Attention core · 4 sources · 5 quotes
Alternating Row-Major and Column-Major Token Serialization core · 1 source · 1 quote
Sink-Augmented Sliding Window Attention core · 1 source · 1 quote
512-token Sliding Window Attention used · 2 sources · 2 quotes
Decoder SWA Bounded Replay used · 1 source · 2 quotes
Dynamic Attention Window Size Training used · 1 source · 1 quote
Pure Sliding Window Attention used · 1 source · 1 quote
SWA-128 used · 1 source · 1 quote
Fixed Sliding Window Attention evaluated · 1 source · 1 quote
Per-Head Gating evaluated · 1 source · 1 quote
SWA-1024 evaluated · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modeltechniques
DeepSeek-V4.1-Flash coreSliding Window Attention coreDecoder SWA Bounded Replay usedPure Sliding Window Attention used—
MiMo-V2.6-Flash coreAlternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—
DeepSeek-V4-Flash usedSliding Window Attention used—
MiMo-V2.5 coreHybrid Sliding Window Attention coreSliding Window Attention coreSWA-128 used—
GLM-5.2 evaluatedSliding Window Attention evaluated—
MiniMax-M3 evaluatedFixed Sliding Window Attention evaluatedSliding Window Attention mentioned—
DeepSeek-V4-Pro usedSliding Window Attention used—
Inkling used512-token Sliding Window Attention used—
Laguna-S-2.1 evaluated512-token Sliding Window Attention evaluatedPer-Head Gating evaluatedSWA-1024 evaluated—
MiMo-V2.5-Pro usedHybrid Sliding Window Attention usedSliding Window Attention used—
MiMo-V2.6-Pro coreAlternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core—
Qwen3.5-397B-A17B usedDynamic Attention Window Size Training used—