Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionsliding window attention

methods · softmax attention

sliding window attention

11 methods filed at this node or below it, from the sources of 17 models.

In practice

2 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelSliding-window attentionInterleaved sliding-window and global attention
LFM3-1.7Bin its codein its code
DeepSeek-V4.1-Flash-1017in its codein its code
Step-5-Previewin its codein its code
Kolibri-1in its codein its code
MiMo-V2.6-Flashin its codein its code
MiMo-V2.6-Proin its codein its code
DeepSeek-V4.1-Flashin its codein its code
DeepSeek-V4-Flash-Vision-Expin its codein its code
Laguna-S-2.1in its codein its code
Inklingin its codein its code
Step-3.7-Flashin its codein its code
DeepSeek-V4-Flashin its codein its code
MiMo-V2.5in its codein its code
MiMo-V2.5-Proin its codein its code
Gemma 4 31Bin its codein its code
gpt-oss-120bin its codein its code
DeepSeek-V4-Pro-0813in its codenot in its code
DeepSeek-V4-Flash-0731in its codenot in its code
DeepSeek-V4-Proin its codecode unclear
Mistral-Large-4not in its codenot in its code
MiniMax-M3.1-Flashnot in its codenot in its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bnot in its codenot in its code
Hy4-previewnot in its codenot in its code
GLM-5.3-Flashnot in its codenot in its code
Qwen3.8-Flash-Nextnot in its codenot in its code
GLM-5.3not in its codenot in its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot in its code
Xiaomi-Robotics-1not in its codenot in its code
Kimi K3not in its codenot in its code
Hy3not in its codenot in its code
GLM-5.2not in its codenot in its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot in its code
MiniMax-M3not in its codenot in its code
MolmoAct 2not in its codenot in its code
Qwen3.6-35B-A3Bnot in its codenot in its code
Qwen3.5-397B-A17Bnot in its codenot in its code
DeepSeek-V3.2not in its codenot in its code
X-VLAnot in its codenot in its code
OpenVLA / OpenVLA-OFTnot in its codenot in its code

In this branch 11

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 11

Sliding Window Attention core · 12 sources · 16 quotes
Hybrid Sliding Window Attention core · 4 sources · 5 quotes
9 single mentions
Sink-Augmented Sliding Window Attention core · 1 source · 1 quote
Decoder SWA Bounded Replay used · 1 source · 1 quote
Dynamic Attention Window Size Training used · 1 source · 1 quote
SWA-128 used · 1 source · 1 quote
Per-Head Gating evaluated · 1 source · 1 quote
SWA-1024 (3:1 Interleaving) evaluated · 1 source · 1 quote
SWA-512 evaluated · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑