Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

methods · token mixer

softmax attention

48 methods filed at this node or below it, from the sources of 32 models.

In practice

5 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelGrouped-query attentionMulti-head latent attentionSliding-window attentionInterleaved sliding-window and global attentionLearnable attention sink
DeepSeek-V4.1-Flash-1017not in its codein its codein its codein its codein its code
MiMo-V2.6-Flashin its codenot in its codein its codein its codein its code
MiMo-V2.6-Proin its codenot in its codein its codein its codein its code
DeepSeek-V4.1-Flashnot in its codein its codein its codein its codein its code
DeepSeek-V4-Flash-Vision-Expin its codenot in its codein its codein its codein its code
MiMo-V2.5in its codenot in its codein its codein its codein its code
MiMo-V2.5-Proin its codenot in its codein its codein its codein its code
gpt-oss-120bin its codenot in its codein its codein its codein its code
LFM3-1.7Bin its codenot in its codein its codein its codenot in its code
Step-5-Previewin its codenot in its codein its codein its codenot in its code
Kolibri-1in its codenot in its codein its codein its codenot in its code
DeepSeek-V4-Pro-0813not in its codein its codein its codenot in its codein its code
Laguna-S-2.1in its codenot in its codein its codein its codenot in its code
Inklingin its codenot in its codein its codein its codenot in its code
Step-3.7-Flashin its codenot in its codein its codein its codenot in its code
DeepSeek-V4-Flashnot in its codecode unclearin its codein its codein its code
Gemma 4 31Bin its codenot in its codein its codein its codenot in its code
Hy4-previewnot in its codein its codenot in its codenot in its codein its code
DeepSeek-V4-Flash-0731not in its codenot in its codein its codenot in its codein its code
DeepSeek-V4-Pronot in its codenot in its codein its codecode unclearin its code
Mistral-Large-4not in its codein its codenot in its codenot in its codenot in its code
MiniMax-M3.1-Flashin its codenot in its codenot in its codenot in its codenot in its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codecode unclearnot in its codenot in its codecode unclear
GLM-5.3-Flashnot in its codein its codenot in its codenot in its codenot in its code
Qwen3.8-Flash-Nextin its codenot in its codenot in its codenot in its codenot in its code
GLM-5.3not in its codein its codenot in its codenot in its codenot in its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot in its codenot in its codenot in its codenot in its code
Xiaomi-Robotics-1in its codenot in its codenot in its codenot in its codenot in its code
Kimi K3not in its codein its codenot in its codenot in its codenot in its code
Hy3in its codenot in its codenot in its codenot in its codenot in its code
GLM-5.2not in its codein its codenot in its codenot in its codenot in its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot in its codenot in its codenot in its codenot in its code
MiniMax-M3in its codenot in its codenot in its codenot in its codenot in its code
MolmoAct 2in its codenot in its codenot in its codenot in its codenot in its code
Qwen3.6-35B-A3Bin its codenot in its codenot in its codenot in its codenot in its code
Qwen3.5-397B-A17Bin its codenot in its codenot in its codenot in its codenot in its code
DeepSeek-V3.2not in its codein its codenot in its codenot in its codenot in its code
X-VLAnot in its codenot in its codenot in its codenot in its codenot in its code
OpenVLA / OpenVLA-OFTcode unclearnot in its codenot in its codenot in its codenot in its code

In this branch 48

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 18

Bidirectional attention core · 4 sources · 4 quotes
Blockwise causal attention core · 2 sources · 2 quotes
Causal self-attention core · 2 sources · 2 quotes
Causal attention used · 2 sources · 2 quotes
Scaled Dot-Product Attention (SDPA) used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Attention layers core · 1 source · 1 quote
DFlash attention core · 1 source · 1 quote
Shared self-attention across modalities core · 1 source · 1 quote
Softplus attention gating core · 1 source · 1 quote
Three-block causal attention mask core · 1 source · 1 quote
Off-by-one attention used · 1 source · 1 quote
SSMax used · 1 source · 1 quote
Strict action padding mask optional · 1 source · 1 quote
Strict vision-language padding mask optional · 1 source · 1 quote
Cascade attention not used · 2 sources · 2 quotes

global attention 5

Global attention core · 6 sources · 6 quotes
Unified Keys and Values core · 2 sources · 2 quotes
Full attention used · 2 sources · 2 quotes
2 single mentions
Full attention without positional encoding core · 1 source · 1 quote

sliding window attention 11

Sliding Window Attention core · 12 sources · 16 quotes
Hybrid Sliding Window Attention core · 4 sources · 5 quotes
9 single mentions
Sink-Augmented Sliding Window Attention core · 1 source · 1 quote
Decoder SWA Bounded Replay used · 1 source · 1 quote
Dynamic Attention Window Size Training used · 1 source · 1 quote
SWA-128 used · 1 source · 1 quote
Per-Head Gating evaluated · 1 source · 1 quote
SWA-1024 (3:1 Interleaving) evaluated · 1 source · 1 quote
SWA-512 evaluated · 1 source · 1 quote

grouped-query attention 5

Grouped-query attention core · 12 sources · 15 quotes
Multi-query attention used · 2 sources · 2 quotes
3 single mentions
Block-sparse grouped-query attention core · 1 source · 1 quote
Softplus-based per-head attention gating used · 1 source · 1 quote

multi-head latent attention 5

Gated Multi-head Latent Attention core · 3 sources · 3 quotes
Multi-head Latent Attention core · 3 sources · 4 quotes
MQA mode of MLA core · 2 sources · 2 quotes
2 from one source, or only mentioned as not used
MHA and MQA modes of MLA used · 1 source · 1 quote
Ragged Multi-head Latent Attention not used · 1 source · 1 quote

attention sink 4

Learnable attention sink bias used · 4 sources · 4 quotes
Learnable attention sink token used · 2 sources · 2 quotes
2 single mentions
Attention sink used · 1 source · 1 quote
RoPE with attention sink unclear · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled hereglobal attentionsliding window attentiongrouped-query attentionmulti-head latent attentionattention sink
DeepSeek-V4.1-Flash used——Decoder SWA Bounded Replay used————
Hy3 core———Grouped-query attention core———
MiMo-V2.6-Flash core——Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core————
NVIDIA-Nemotron-3-Ultra-550B-A55B coreAttention layers core——————
DeepSeek-V4-Flash used——Sliding Window Attention used—Multi-query attention used——Attention sink used—
DeepSeek-V3.2 core————MQA mode of MLA coreMHA and MQA modes of MLA used——
DeepSeek-V4-Flash-Vision-Exp coreDFlash attention core——————
DeepSeek-V4-Pro used——Sliding Window Attention used—Multi-query attention used——Attention sink used—
GLM-5.2 core——Sliding Window Attention evaluated—Multi-query attention mentioned—Multi-head Latent Attention core——
GR00T N1.7 optionalStrict action padding mask optionalStrict vision-language padding mask optional——————
Gemma 4 31B core—Unified Keys and Values core—————
Inkling used——Sliding Window Attention used—Grouped-query attention used———
Kimi K3 core————Gated Multi-head Latent Attention core——
Kolibri-1 core—Full attention without positional encoding core—Sliding Window Attention core————
LFM3-1.7B core—Global attention core—Sliding Window Attention core—Grouped-query attention coreSparse use of grouped-query attention blocks for global context core———
Laguna-S-2.1 coreSoftplus attention gating core——Per-Head Gating evaluatedSWA-1024 (3:1 Interleaving) evaluatedSWA-512 evaluated—Grouped-query attention coreSoftplus-based per-head attention gating used———
LingBot-VLA 2.0 coreBlockwise causal attention coreShared self-attention across modalities core——————
MiMo-V2.5 core—Global attention coreFull attention used—Hybrid Sliding Window Attention coreSliding Window Attention coreSWA-128 used—Grouped-query attention used——Learnable attention sink bias used—
MiMo-V2.5-Pro used—Global attention used—Hybrid Sliding Window Attention usedSliding Window Attention used———Learnable attention sink bias used—
MiMo-V2.6-Pro core——Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core————
MiniMax-M3 core—Full attention evaluated—Sliding Window Attention evaluated—Block-sparse grouped-query attention coreGrouped-query attention core—Multi-head Latent Attention not used—Learnable attention sink token not usedRoPE with attention sink unclear—
Mistral-Large-4 coreSSMax used——Sliding Window Attention not used——Multi-head Latent Attention coreRagged Multi-head Latent Attention not used——
MolmoAct 2 coreAttention-based pooling of local visual windows coreScaled Dot-Product Attention (SDPA) used——————
NVIDIA-Nemotron-3.5-Lightning-30B-A3B used——Sliding Window Attention with an 8,192-Position Window used————
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B used——Sliding Window Attention with an 8,192-Position Window used————
OpenVLA / OpenVLA-OFT coreBidirectional attention core——————
Qwen3.5-397B-A17B used—Full attention with multi-head attention for reliability used—Dynamic Attention Window Size Training used————
SmolVLA coreCausal self-attention coreCausal attention usedBidirectional attention evaluated——————
Step-3.7-Flash not usedCascade attention not used——————
Xiaomi-Robotics-1 usedCausal attention usedShape attention mask for action-prefix conditioning used—————Learnable attention sink token used—
gpt-oss-120b usedOff-by-one attention used———Grouped-query attention used———
π0.5 coreBidirectional attention coreThree-block causal attention mask corePrefix-conditioned autoregressive action-token attention used——————
↑