methods · model architecture
token mixer
153 methods filed at this node or below it, from the sources of 37 models.
In practice
9 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.
adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
| Model | Grouped-query attention | Multi-head latent attention | Sliding-window attention | Interleaved sliding-window and global attention | Indexer-selected sparse attention (DeepSeek Sparse Attention) | Linear-attention or state-space layers alongside full attention | Gated DeltaNet layers | Mamba-2 layers | Learnable attention sink |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | not in its code | in its code | in its code | in its code | in its code | not in its code | not in its code | not in its code | in its code |
| DeepSeek-V4.1-Flash | not in its code | in its code | in its code | in its code | in its code | not in its code | not in its code | not in its code | in its code |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not in its code | in its code | in its code | in its code | not in its code | not in its code | not in its code | in its code |
| Step-5-Preview | in its code | not in its code | in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| MiMo-V2.6-Flash | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | in its code |
| MiMo-V2.6-Pro | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | in its code |
| GLM-5.3-Flash | not in its code | in its code | not in its code | not in its code | in its code | in its code | in its code | not in its code | not in its code |
| Qwen3.8-Flash-Next | in its code | not in its code | not in its code | not in its code | in its code | in its code | in its code | not in its code | not in its code |
| DeepSeek-V4-Pro-0813 | not in its code | in its code | in its code | not in its code | in its code | not in its code | not in its code | not in its code | in its code |
| DeepSeek-V4-Flash | not in its code | code unclear | in its code | in its code | in its code | not in its code | not in its code | not in its code | in its code |
| MiMo-V2.5 | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | in its code |
| MiMo-V2.5-Pro | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | in its code |
| gpt-oss-120b | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | in its code |
| LFM3-1.7B | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| Kolibri-1 | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| Hy4-preview | not in its code | in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | in its code |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not in its code | not in its code | not in its code | not in its code | in its code | not in its code | in its code | not in its code |
| DeepSeek-V4-Flash-0731 | not in its code | not in its code | in its code | not in its code | in its code | not in its code | not in its code | not in its code | in its code |
| Laguna-S-2.1 | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| Kimi K3 | not in its code | in its code | not in its code | not in its code | not in its code | in its code | in its code | not in its code | not in its code |
| Inkling | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not in its code | not in its code | not in its code | not in its code | in its code | not in its code | in its code | not in its code |
| Step-3.7-Flash | in its code | not in its code | in its code | in its code | code unclear | not in its code | not in its code | not in its code | not in its code |
| DeepSeek-V4-Pro | not in its code | not in its code | in its code | code unclear | in its code | not in its code | not in its code | not in its code | in its code |
| Qwen3.6-35B-A3B | in its code | not in its code | not in its code | not in its code | not in its code | in its code | in its code | not in its code | not in its code |
| Gemma 4 31B | in its code | not in its code | in its code | in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| Qwen3.5-397B-A17B | in its code | not in its code | not in its code | not in its code | not in its code | in its code | in its code | not in its code | not in its code |
| Mistral-Large-4 | not in its code | in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| MiniMax-M3.1-Flash | in its code | not in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | code unclear | not in its code | not in its code | not in its code | code unclear | code unclear | in its code | code unclear |
| GLM-5.3 | not in its code | in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| GLM-5.2 | not in its code | in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| MiniMax-M3 | in its code | not in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| DeepSeek-V3.2 | not in its code | in its code | not in its code | not in its code | in its code | not in its code | not in its code | not in its code | not in its code |
| Xiaomi-Robotics-1 | in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| Hy3 | in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| MolmoAct 2 | in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| X-VLA | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
| OpenVLA / OpenVLA-OFT | code unclear | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code | not in its code |
In this branch 153
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 15
13 from one source, or only mentioned as not used
softmax attention 48
13 from one source, or only mentioned as not used
global attention 5
sliding window attention 11
grouped-query attention 5
multi-head latent attention 5
2 from one source, or only mentioned as not used
attention sink 4
sparse attention 60
15 from one source, or only mentioned as not used
block selection 9
sparse attention indexer 24
17 from one source, or only mentioned as not used
fixed-pattern sparse attention 3
all mentioned once or only as not used
linear attention & state space 18
1 only mentioned as not used
gated delta network 12
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
Proposed children
Paths the corpus wanted and the taxonomy does not have. A human promotes them into the outline; the extraction cannot.
short convolution