methods · token mixer
softmax attention
48 methods filed at this node or below it, from the sources of 32 models.
Explainer · T1 · 25 minFull attention on a memory budgetMHA, MQA, GQA and MLA trade capacity or compute for a smaller KV cacheExplainer · T6 · 32 minAttention sinks and output gatesA softmax head's attention always adds up to one: a learned sink logit lets it add up to less for one number per head, an output gate scales the head's result for a small projection, and the few direct comparisons are small and mixed
In practice
5 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.
adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
| Model | Grouped-query attention | Multi-head latent attention | Sliding-window attention | Interleaved sliding-window and global attention | Learnable attention sink |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | not in its code | in its code | in its code | in its code | in its code |
| MiMo-V2.6-Flash | in its code | not in its code | in its code | in its code | in its code |
| MiMo-V2.6-Pro | in its code | not in its code | in its code | in its code | in its code |
| DeepSeek-V4.1-Flash | not in its code | in its code | in its code | in its code | in its code |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not in its code | in its code | in its code | in its code |
| MiMo-V2.5 | in its code | not in its code | in its code | in its code | in its code |
| MiMo-V2.5-Pro | in its code | not in its code | in its code | in its code | in its code |
| gpt-oss-120b | in its code | not in its code | in its code | in its code | in its code |
| LFM3-1.7B | in its code | not in its code | in its code | in its code | not in its code |
| Step-5-Preview | in its code | not in its code | in its code | in its code | not in its code |
| Kolibri-1 | in its code | not in its code | in its code | in its code | not in its code |
| DeepSeek-V4-Pro-0813 | not in its code | in its code | in its code | not in its code | in its code |
| Laguna-S-2.1 | in its code | not in its code | in its code | in its code | not in its code |
| Inkling | in its code | not in its code | in its code | in its code | not in its code |
| Step-3.7-Flash | in its code | not in its code | in its code | in its code | not in its code |
| DeepSeek-V4-Flash | not in its code | code unclear | in its code | in its code | in its code |
| Gemma 4 31B | in its code | not in its code | in its code | in its code | not in its code |
| Hy4-preview | not in its code | in its code | not in its code | not in its code | in its code |
| DeepSeek-V4-Flash-0731 | not in its code | not in its code | in its code | not in its code | in its code |
| DeepSeek-V4-Pro | not in its code | not in its code | in its code | code unclear | in its code |
| Mistral-Large-4 | not in its code | in its code | not in its code | not in its code | not in its code |
| MiniMax-M3.1-Flash | in its code | not in its code | not in its code | not in its code | not in its code |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | code unclear | not in its code | not in its code | code unclear |
| GLM-5.3-Flash | not in its code | in its code | not in its code | not in its code | not in its code |
| Qwen3.8-Flash-Next | in its code | not in its code | not in its code | not in its code | not in its code |
| GLM-5.3 | not in its code | in its code | not in its code | not in its code | not in its code |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | not in its code | not in its code | not in its code | not in its code |
| Xiaomi-Robotics-1 | in its code | not in its code | not in its code | not in its code | not in its code |
| Kimi K3 | not in its code | in its code | not in its code | not in its code | not in its code |
| Hy3 | in its code | not in its code | not in its code | not in its code | not in its code |
| GLM-5.2 | not in its code | in its code | not in its code | not in its code | not in its code |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | not in its code | not in its code | not in its code | not in its code |
| MiniMax-M3 | in its code | not in its code | not in its code | not in its code | not in its code |
| MolmoAct 2 | in its code | not in its code | not in its code | not in its code | not in its code |
| Qwen3.6-35B-A3B | in its code | not in its code | not in its code | not in its code | not in its code |
| Qwen3.5-397B-A17B | in its code | not in its code | not in its code | not in its code | not in its code |
| DeepSeek-V3.2 | not in its code | in its code | not in its code | not in its code | not in its code |
| X-VLA | not in its code | not in its code | not in its code | not in its code | not in its code |
| OpenVLA / OpenVLA-OFT | code unclear | not in its code | not in its code | not in its code | not in its code |
In this branch 48
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 18
13 from one source, or only mentioned as not used
multi-head latent attention 5
2 from one source, or only mentioned as not used
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.