Open Models Almanac
Methodsmodel architecturetoken mixer

methods · model architecture

token mixer

153 methods filed at this node or below it, from the sources of 37 models.

In practice

9 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelGrouped-query attentionMulti-head latent attentionSliding-window attentionInterleaved sliding-window and global attentionIndexer-selected sparse attention (DeepSeek Sparse Attention)Linear-attention or state-space layers alongside full attentionGated DeltaNet layersMamba-2 layersLearnable attention sink
DeepSeek-V4.1-Flash-1017not in its codein its codein its codein its codein its codenot in its codenot in its codenot in its codein its code
DeepSeek-V4.1-Flashnot in its codein its codein its codein its codein its codenot in its codenot in its codenot in its codein its code
DeepSeek-V4-Flash-Vision-Expin its codenot in its codein its codein its codein its codenot in its codenot in its codenot in its codein its code
Step-5-Previewin its codenot in its codein its codein its codein its codenot in its codenot in its codenot in its codenot in its code
MiMo-V2.6-Flashin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its code
MiMo-V2.6-Proin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its code
GLM-5.3-Flashnot in its codein its codenot in its codenot in its codein its codein its codein its codenot in its codenot in its code
Qwen3.8-Flash-Nextin its codenot in its codenot in its codenot in its codein its codein its codein its codenot in its codenot in its code
DeepSeek-V4-Pro-0813not in its codein its codein its codenot in its codein its codenot in its codenot in its codenot in its codein its code
DeepSeek-V4-Flashnot in its codecode unclearin its codein its codein its codenot in its codenot in its codenot in its codein its code
MiMo-V2.5in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its code
MiMo-V2.5-Proin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its code
gpt-oss-120bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its code
LFM3-1.7Bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its code
Kolibri-1in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its code
Hy4-previewnot in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codein its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its code
DeepSeek-V4-Flash-0731not in its codenot in its codein its codenot in its codein its codenot in its codenot in its codenot in its codein its code
Laguna-S-2.1in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its code
Kimi K3not in its codein its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its code
Inklingin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its code
Step-3.7-Flashin its codenot in its codein its codein its codecode unclearnot in its codenot in its codenot in its codenot in its code
DeepSeek-V4-Pronot in its codenot in its codein its codecode unclearin its codenot in its codenot in its codenot in its codein its code
Qwen3.6-35B-A3Bin its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its code
Gemma 4 31Bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its code
Qwen3.5-397B-A17Bin its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its code
Mistral-Large-4not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
MiniMax-M3.1-Flashin its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codecode unclearnot in its codenot in its codenot in its codecode unclearcode unclearin its codecode unclear
GLM-5.3not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
GLM-5.2not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
MiniMax-M3in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
DeepSeek-V3.2not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its code
Xiaomi-Robotics-1in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
Hy3in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
MolmoAct 2in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
X-VLAnot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
OpenVLA / OpenVLA-OFTcode unclearnot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code

In this branch 153

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 15

Cross-attention core · 3 sources · 3 quotes
Gated short convolution core · 3 sources · 4 quotes
13 from one source, or only mentioned as not used
Forward attention among visual tokens core · 1 source · 1 quote
Grouped dynamic convolution core · 1 source · 1 quote
Kernel-size-7 convolution core · 1 source · 1 quote
Noncausal transformer action expert core · 1 source · 1 quote
Windowed multi-head attention pooling core · 1 source · 1 quote
Factorized spatial-temporal attention used · 1 source · 1 quote
Kernel-4 convolutions in decoder blocks used · 1 source · 1 quote
Cross-attention-only action expert evaluated · 1 source · 1 quote
Cross-layer attention sharing not used · 1 source · 1 quote

softmax attention 48

Bidirectional attention core · 4 sources · 4 quotes
Blockwise causal attention core · 2 sources · 2 quotes
Causal self-attention core · 2 sources · 2 quotes
Causal attention used · 2 sources · 2 quotes
Scaled Dot-Product Attention (SDPA) used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Attention layers core · 1 source · 1 quote
DFlash attention core · 1 source · 1 quote
Shared self-attention across modalities core · 1 source · 1 quote
Softplus attention gating core · 1 source · 1 quote
Three-block causal attention mask core · 1 source · 1 quote
Off-by-one attention used · 1 source · 1 quote
SSMax used · 1 source · 1 quote
Strict action padding mask optional · 1 source · 1 quote
Strict vision-language padding mask optional · 1 source · 1 quote
Cascade attention not used · 2 sources · 2 quotes

global attention 5

Global attention core · 6 sources · 6 quotes
Unified Keys and Values core · 2 sources · 2 quotes
Full attention used · 2 sources · 2 quotes
2 single mentions
Full attention without positional encoding core · 1 source · 1 quote

sliding window attention 11

Sliding Window Attention core · 12 sources · 16 quotes
Hybrid Sliding Window Attention core · 4 sources · 5 quotes
9 single mentions
Sink-Augmented Sliding Window Attention core · 1 source · 1 quote
Decoder SWA Bounded Replay used · 1 source · 1 quote
Dynamic Attention Window Size Training used · 1 source · 1 quote
SWA-128 used · 1 source · 1 quote
Per-Head Gating evaluated · 1 source · 1 quote
SWA-1024 (3:1 Interleaving) evaluated · 1 source · 1 quote
SWA-512 evaluated · 1 source · 1 quote

grouped-query attention 5

Grouped-query attention core · 12 sources · 15 quotes
Multi-query attention used · 2 sources · 2 quotes
3 single mentions
Block-sparse grouped-query attention core · 1 source · 1 quote
Softplus-based per-head attention gating used · 1 source · 1 quote

multi-head latent attention 5

Gated Multi-head Latent Attention core · 3 sources · 3 quotes
Multi-head Latent Attention core · 3 sources · 4 quotes
MQA mode of MLA core · 2 sources · 2 quotes
2 from one source, or only mentioned as not used
MHA and MQA modes of MLA used · 1 source · 1 quote
Ragged Multi-head Latent Attention not used · 1 source · 1 quote

attention sink 4

Learnable attention sink bias used · 4 sources · 4 quotes
Learnable attention sink token used · 2 sources · 2 quotes
2 single mentions
Attention sink used · 1 source · 1 quote
RoPE with attention sink unclear · 1 source · 1 quote

sparse attention 60

DeepSeek Sparse Attention core · 16 sources · 18 quotes
Sparse attention core · 8 sources · 8 quotes
Compressed Sparse Attention core · 6 sources · 6 quotes
Qwen Sparse Attention core · 6 sources · 6 quotes
Heavily Compressed Attention core · 5 sources · 5 quotes
Gated DeepSeek Sparse Attention core · 3 sources · 3 quotes
Compressed Sparse Attention 2 core · 2 sources · 3 quotes
KV-outer sparse attention used · 2 sources · 2 quotes
15 from one source, or only mentioned as not used
CSA2 Full, Reindex, and Reuse modes core · 1 source · 1 quote
Fixed-budget sparse-attention selection core · 1 source · 1 quote
NoPE sparse multi-head latent attention core · 1 source · 1 quote
QSA micro-block compression at ratio 4 core · 1 source · 1 quote
Token-wise compression core · 1 source · 1 quote
FlashMLA sparse attention default · 1 source · 1 quote
Natively trained sparsity used · 1 source · 1 quote
Sparse softmax attention used · 1 source · 1 quote
Two-stage sparse-attention formulation used · 1 source · 1 quote
Two-stage sparse-attention training used · 1 source · 1 quote
Native Sparse Attention not used · 1 source · 1 quote

block selection 9

MiniMax Sparse Attention core · 7 sources · 8 quotes
Top-k block selection used · 2 sources · 2 quotes
7 single mentions
Block-level selection core · 1 source · 1 quote
Dynamic sparse selection core · 1 source · 1 quote
Main Branch core · 1 source · 2 quotes
Top-512 block selection core · 1 source · 1 quote
Block Max Pooling used · 1 source · 1 quote
Top-scoring compressed-block selection used · 1 source · 1 quote
Mixture of Block Attention evaluated · 1 source · 1 quote

sparse attention indexer 24

IndexCache core · 3 sources · 3 quotes
IndexShare core · 3 sources · 3 quotes
Lightning Indexer core · 3 sources · 3 quotes
Fine-grained token selection mechanism core · 2 sources · 2 quotes
Hierarchical Sparse Indexer core · 2 sources · 3 quotes
17 from one source, or only mentioned as not used
Compressed lightweight indexer core · 1 source · 1 quote
CSA2 compressor and indexer design core · 1 source · 1 quote
DeepSeek Sparse Attention indexer core · 1 source · 1 quote
Index Branch core · 1 source · 2 quotes
IndexPool core · 1 source · 1 quote
MQA indexer core · 1 source · 1 quote
Average pooling used · 1 source · 1 quote
Block-causal scoring used · 1 source · 1 quote
Detached indexer-input optimization used · 1 source · 1 quote
Indexer warmup used · 1 source · 2 quotes
KL-distillation indexer warm-up used · 1 source · 1 quote
Single-head index key used · 1 source · 1 quote
Top-2,048 key-token selection per query used · 1 source · 1 quote
Index Branch output not used · 1 source · 1 quote
Index value head not used · 1 source · 1 quote

fixed-pattern sparse attention 3

all mentioned once or only as not used
Sparse Global Attention Anchors core · 1 source · 1 quote
Local Block used · 1 source · 1 quote
Forced Sink and Local Window not used · 1 source · 1 quote

linear attention & state space 18

Linear attention core · 2 sources · 2 quotes
1 only mentioned as not used
Lightning Attention not used · 1 source · 1 quote

Mamba 4

Mamba-2 core · 5 sources · 6 quotes
Mamba core · 2 sources · 2 quotes
2 single mentions
Mamba checkpointing core · 1 source · 1 quote
Mamba-2 SSM cache optional · 1 source · 1 quote

gated delta network 12

Gated DeltaNet core · 12 sources · 12 quotes
Kimi Delta Attention core · 6 sources · 6 quotes
KDA core · 2 sources · 2 quotes
8 single mentions
Gated attention with reduced KV heads core · 1 source · 1 quote
Gated DeltaNet hybrid attention core · 1 source · 1 quote
Hybrid linear attention mechanism core · 1 source · 1 quote
Bounded sigmoid output gate used · 1 source · 1 quote
Input-dependent full-rank output gate used · 1 source · 1 quote
Lower-bounded scaled-sigmoid log-decay used · 1 source · 1 quote
SimpleGDN evaluated · 1 source · 1 quote

hybrid layer stacking 12

Hybrid Sliding-Window and Global Attention core · 21 sources · 23 quotes
Gated DeltaNet and full attention core · 7 sources · 9 quotes
Hybrid Mamba-Transformer core · 6 sources · 9 quotes
Hybrid Mamba-Attention core · 3 sources · 4 quotes
Hybrid Sparse and Linear Attention core · 3 sources · 3 quotes
Hybrid compressed attention (CSA and HCA) used · 3 sources · 3 quotes
4 single mentions
Search-Based Sliding-Window Attention Pattern evaluated · 1 source · 1 quote
Dense Attention Fallback unclear · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled heresoftmax attentionsparse attentionlinear attention & state spacehybrid layer stacking
GLM-5.3-Flash core——IndexPool coreNoPE sparse multi-head latent attention core—Kimi Delta Attention core—Hybrid Sparse and Linear Attention core—
DeepSeek-V4.1-Flash core—Decoder SWA Bounded Replay used—Compressed Sparse Attention 2 coreCSA2 compressor and indexer design coreCSA2 Full, Reindex, and Reuse modes coreHierarchical Sparse Indexer coreProgressive context-length extension for sparse attention usedSparse attention training from scratch without dense-attention warmup used———
Hy4-preview core——Gated DeepSeek Sparse Attention coreIndexCache core———
MiniMax-M3.1-Flash core——MiniMax Sparse Attention coreSparse attention coreAggressive sparsity for inference efficiency used———
Hy3 core—Grouped-query attention core————
MiMo-V2.6-Flash core—Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core———Hybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
NVIDIA-Nemotron-3-Ultra-550B-A55B core—Attention layers core—Sparse Global Attention Anchors core—Mamba coreMamba-2 coreMamba-2 SSM cache optional—Hybrid Mamba-Attention coreHybrid Mamba-Transformer core—
DeepSeek-V4-Flash core—Attention sink usedMulti-query attention usedSliding Window Attention used—Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage sparse-attention training used——Hybrid compressed attention (CSA and HCA) used—
DeepSeek-V3.2 core—MQA mode of MLA coreMHA and MQA modes of MLA used—DeepSeek Sparse Attention coreFine-grained token selection mechanism coreLightning Indexer coreDense attention warm-up with a frozen lightning indexer usedDetached indexer-input optimization usedSparse-attention continued pretraining with joint model and indexer optimization used———
DeepSeek-V4-Flash-Vision-Exp core—DFlash attention core————
DeepSeek-V4-Pro core—Attention sink usedMulti-query attention usedSliding Window Attention used—Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreTwo-stage sparse-attention training used——Hybrid compressed attention (CSA and HCA) used—
GLM-5.2 core—Multi-head Latent Attention coreSliding Window Attention evaluatedMulti-query attention mentioned—DeepSeek Sparse Attention coreIndexShare coreLightning Indexer usedSparse attention used—Gated DeltaNet evaluatedSimpleGDN evaluated—Search-Based Sliding-Window Attention Pattern evaluated—
GR00T N1.7 coreCross-attention core—Strict action padding mask optionalStrict vision-language padding mask optional————
Gemma 4 31B core—Unified Keys and Values core———Hybrid Sliding-Window and Global Attention core—
Inkling coreKernel-4 convolutions in decoder blocks used—Grouped-query attention usedSliding Window Attention used———Hybrid Sliding-Window and Global Attention core—
Kimi K3 coreFactorized spatial-temporal attention used—Gated Multi-head Latent Attention core——Hybrid linear attention mechanism coreKDA coreKimi Delta Attention coreKimi Delta Attention–Attention Residuals architecture coreGated DeltaNet usedInput-dependent full-rank output gate usedLower-bounded scaled-sigmoid log-decay used—Layerwise Hybrid Linear and Global Attention core—
Kolibri-1 core—Full attention without positional encoding coreSliding Window Attention core———Hybrid Sliding-Window and Global Attention core—
LFM3-1.7B coreGated short convolution coreInput-dependent multiplicative gating around a depthwise short convolution coreKernel-size-7 convolution core—Global attention coreGrouped-query attention coreSliding Window Attention coreSparse use of grouped-query attention blocks for global context core————
Laguna-S-2.1 coreDense gated attention with full RoPE and full gating evaluated—Grouped-query attention coreSoftplus attention gating coreSoftplus-based per-head attention gating usedPer-Head Gating evaluatedSWA-1024 (3:1 Interleaving) evaluatedSWA-512 evaluated———Hybrid Sliding-Window and Global Attention coreGated DeltaNet and full attention used—
LingBot-VLA 2.0 core—Blockwise causal attention coreShared self-attention across modalities core————
MiMo-V2.5 core—Global attention coreHybrid Sliding Window Attention coreSliding Window Attention coreFull attention usedGrouped-query attention usedLearnable attention sink bias usedSWA-128 used———Hybrid Sliding-Window and Global Attention core—
MiMo-V2.5-Pro core—Global attention usedHybrid Sliding Window Attention usedLearnable attention sink bias usedSliding Window Attention used———Hybrid Sliding-Window and Global Attention core—
MiMo-V2.6-Pro core—Alternating Row-Major and Column-Major Token Serialization coreSink-Augmented Sliding Window Attention core———Hybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture core—
MiniMax-M3 core—Block-sparse grouped-query attention coreGrouped-query attention coreFull attention evaluatedSliding Window Attention evaluatedLearnable attention sink token not usedMulti-head Latent Attention not usedRoPE with attention sink unclear—Block-level selection coreDynamic sparse selection coreIndex Branch coreMain Branch coreMiniMax Sparse Attention coreSparse attention coreBlock Max Pooling usedIndexer warmup usedKV-outer sparse attention usedLocal Block usedNatively trained sparsity usedSingle-head index key usedSparse softmax attention usedTop-k block selection usedTwo-stage sparse-attention formulation usedDeepSeek Sparse Attention evaluatedMixture of Block Attention evaluatedCompressed Sparse Attention not usedForced Sink and Local Window not usedIndex Branch output not usedIndex value head not usedNative Sparse Attention not used—Linear attention mentionedLightning Attention not used—Dense Attention Fallback unclear—
Mistral-Large-4 coreCross-layer attention sharing not used—Multi-head Latent Attention coreSSMax usedRagged Multi-head Latent Attention not usedSliding Window Attention not used—DeepSeek Sparse Attention indexer coreMulti-head latent attention with a DSA sparse indexer coreSparse attention coreFlashMLA sparse attention defaultDeepSeek Sparse Attention usedKL-distillation indexer warm-up usedShared DSA indexer across sliding-window layers usedTop-2,048 key-token selection per query used———
MolmoAct 2 coreBidirectional self-attention with VLM cross-attention coreForward attention among visual tokens coreNoncausal transformer action expert coreWindowed multi-head attention pooling core—Attention-based pooling of local visual windows coreScaled Dot-Product Attention (SDPA) used————
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreGrouped dynamic convolution core—Sliding Window Attention with an 8,192-Position Window used——Mamba-2 core—Hybrid Mamba-2 and Mixture-of-Experts Architecture coreHybrid Mamba-Transformer core—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreGrouped dynamic convolution core—Sliding Window Attention with an 8,192-Position Window used——Mamba checkpointing core—Hybrid Mamba-Transformer core—
OpenVLA / OpenVLA-OFT core—Bidirectional attention core————
Qwen3.5-397B-A17B core—Dynamic Attention Window Size Training usedFull attention with multi-head attention for reliability used—DeepSeek Sparse Attention core—Gated attention with reduced KV heads coreGated DeltaNet coreGated DeltaNet–sparse Mixture-of-Experts hybrid coreLinear attention core—Gated DeltaNet and full attention core—
Qwen3.6-35B-A3B core———Gated DeltaNet coreGated DeltaNet hybrid attention core—Gated DeltaNet and full attention core—
Qwen3.8-Flash-Next core——Compressed lightweight indexer coreFixed-budget sparse-attention selection coreMQA indexer coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreReuse QSA index selection across speculative decoding steps coreTop-512 block selection coreAverage pooling usedBlock-causal scoring usedJoint backbone and indexer training under sparse attention usedTop-scoring compressed-block selection used—Gated DeltaNet coreBounded sigmoid output gate used—Gated DeltaNet and full attention core—
SmolVLA coreCross-attention coreCross-attention-only action expert evaluated—Causal self-attention coreCausal attention usedBidirectional attention evaluated———Interleaved Self-Attention and Cross-Attention core—
Step-3.7-Flash not used—Cascade attention not used————
Xiaomi-Robotics-1 used—Causal attention usedLearnable attention sink token usedShape attention mask for action-prefix conditioning used————
gpt-oss-120b used—Grouped-query attention usedOff-by-one attention used———Hybrid Sliding-Window and Global Attention used—
π0.5 coreUnidirectional VLM-to-action-expert information flow used—Bidirectional attention coreThree-block causal attention mask corePrefix-conditioned autoregressive action-token attention used————

Proposed children

Paths the corpus wanted and the taxonomy does not have. A human promotes them into the outline; the extraction cannot.

short convolutionshort convolution, sconv, gated short convolution
↑