Open Models Almanac
Methodsmodel architecture

methods · pipeline stage 3 of 6

model architecture

513 methods filed at this node or below it, from the sources of 45 models.

In practice

18 features checked against each model’s own code and config.json, apart from its documents. A column’s page has the verdict and the lines.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
ModelGrouped-query attentionMulti-head latent attentionSliding-window attentionInterleaved sliding-window and global attentionIndexer-selected sparse attention (DeepSeek Sparse Attention)Linear-attention or state-space layers alongside full attentionGated DeltaNet layersMamba-2 layersLearnable attention sinkQK normalizationPartial RoPEYaRN RoPE scalingLayers without positional encoding (NoPE)Mixture of expertsShared expertAuxiliary-loss-free load balancing (selection bias)Multi-token prediction layersSwiGLU feed-forward
DeepSeek-V4-Flash-Vision-Expin its codenot in its codein its codein its codein its codenot in its codenot in its codenot in its codein its codein its codein its codein its codenot in its codein its codein its codein its codein its codein its code
DeepSeek-V4.1-Flash-1017not in its codein its codein its codein its codein its codenot in its codenot in its codenot in its codein its codecode unclearin its codein its codecode unclearin its codein its codein its codein its codein its code
DeepSeek-V4.1-Flashnot in its codein its codein its codein its codein its codenot in its codenot in its codenot in its codein its codecode unclearin its codein its codecode unclearin its codein its codein its codein its codein its code
DeepSeek-V4-Pro-0813not in its codein its codein its codenot in its codein its codenot in its codenot in its codenot in its codein its codein its codein its codein its codenot in its codein its codein its codein its codein its codein its code
DeepSeek-V4-Flashnot in its codecode unclearin its codein its codein its codenot in its codenot in its codenot in its codein its codein its codein its codein its codenot in its codein its codein its codein its codein its codein its code
Step-5-Previewin its codenot in its codein its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codecode unclearin its codein its codein its codein its codein its code
DeepSeek-V4-Flash-0731not in its codenot in its codein its codenot in its codein its codenot in its codenot in its codenot in its codein its codein its codein its codein its codenot in its codein its codein its codein its codein its codein its code
DeepSeek-V4-Pronot in its codenot in its codein its codecode unclearin its codenot in its codenot in its codenot in its codein its codein its codein its codein its codenot in its codein its codein its codein its codein its codein its code
Qwen3.8-Flash-Nextin its codenot in its codenot in its codenot in its codein its codein its codein its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codenot in its codein its codein its code
Laguna-S-2.1in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codein its codenot in its codein its codein its codein its codenot in its codein its code
Step-3.7-Flashin its codenot in its codein its codein its codecode unclearnot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codein its codein its codein its code
Kolibri-1in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codecode unclearin its codein its codein its codein its codenot in its codein its code
MiMo-V2.6-Flashin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codein its codenot in its codein its codein its codein its code
Hy4-previewnot in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codein its codein its codein its codein its codein its code
GLM-5.3-Flashnot in its codein its codenot in its codenot in its codein its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codein its codein its codein its code
Inklingin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codecode unclearin its codein its codein its codein its codein its code
MiniMax-M3in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codein its codein its codecode unclearnot in its codein its codein its codein its codein its codein its code
Qwen3.5-397B-A17Bin its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codenot in its codein its codein its code
Mistral-Large-4not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codein its codein its codein its codenot in its codenot in its codein its code
MiniMax-M3.1-Flashin its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codein its codein its codecode unclearnot in its codein its codein its codein its codecode unclearin its code
MiMo-V2.6-Proin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codein its codenot in its codein its codecode unclearin its code
GLM-5.3not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codein its codein its codein its codein its codein its code
MiMo-V2.5in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codein its codenot in its codein its codecode unclearin its code
MiMo-V2.5-Proin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codein its codenot in its codein its codecode unclearin its code
Qwen3.6-35B-A3Bin its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codenot in its codenot in its codein its codein its codenot in its codecode unclearin its code
DeepSeek-V3.2not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codein its codein its codein its codecode unclearin its code
Kimi K3not in its codein its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codenot in its codein its codecode unclearnot in its codein its codein its codein its codenot in its codenot in its code
GLM-5.2not in its codein its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codein its codein its codein its codecode unclearin its code
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codenot in its codecode unclearcode unclearin its codein its codein its codein its codenot in its code
gpt-oss-120bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codenot in its codein its code
LFM3-1.7Bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its code
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codein its codenot in its codenot in its codecode unclearnot in its codecode unclearin its codein its codein its codenot in its codenot in its code
Hy3in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codein its codein its codein its codenot in its codein its code
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codecode unclearnot in its codenot in its codenot in its codecode unclearcode unclearin its codecode unclearcode unclearcode unclearnot in its codecode unclearin its codein its codecode unclearin its codenot in its code
Gemma 4 31Bin its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
Xiaomi-Robotics-1in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its code
MolmoAct 2in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codein its code
X-VLAnot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its code
OpenVLA / OpenVLA-OFTcode unclearnot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codenot in its codecode unclearnot in its codenot in its codenot in its codenot in its codenot in its codenot in its codecode unclear

In this branch 513

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 19

Causal Encoder-Decoder (CED) core · 5 sources · 6 quotes
Diffusion Transformer (DiT) core · 4 sources · 4 quotes
Flow-based vision-language-action model core · 4 sources · 4 quotes
Per-Layer Embeddings (PLE) core · 4 sources · 4 quotes
Flow-matching action expert core · 3 sources · 3 quotes
Mixture-of-Transformers (MoT) core · 3 sources · 3 quotes
Hybrid architecture core · 2 sources · 2 quotes
12 single mentions
Asymmetric architecture core · 1 source · 1 quote
Decoder-only Transformer architecture core · 1 source · 1 quote
Flow-matching transformer core · 1 source · 1 quote
Regular Transformer decoder core · 1 source · 1 quote
Dense model architecture used · 1 source · 1 quote
Downsized action expert used · 1 source · 1 quote
Residual-stream width selection evaluated · 1 source · 1 quote

token mixer 153

Cross-attention core · 3 sources · 3 quotes
Gated short convolution core · 3 sources · 4 quotes
13 from one source, or only mentioned as not used
Forward attention among visual tokens core · 1 source · 1 quote
Grouped dynamic convolution core · 1 source · 1 quote
Kernel-size-7 convolution core · 1 source · 1 quote
Noncausal transformer action expert core · 1 source · 1 quote
Windowed multi-head attention pooling core · 1 source · 1 quote
Factorized spatial-temporal attention used · 1 source · 1 quote
Kernel-4 convolutions in decoder blocks used · 1 source · 1 quote
Cross-attention-only action expert evaluated · 1 source · 1 quote
Cross-layer attention sharing not used · 1 source · 1 quote

softmax attention 48

Bidirectional attention core · 4 sources · 4 quotes
Blockwise causal attention core · 2 sources · 2 quotes
Causal self-attention core · 2 sources · 2 quotes
Causal attention used · 2 sources · 2 quotes
Scaled Dot-Product Attention (SDPA) used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Attention layers core · 1 source · 1 quote
DFlash attention core · 1 source · 1 quote
Shared self-attention across modalities core · 1 source · 1 quote
Softplus attention gating core · 1 source · 1 quote
Three-block causal attention mask core · 1 source · 1 quote
Off-by-one attention used · 1 source · 1 quote
SSMax used · 1 source · 1 quote
Strict action padding mask optional · 1 source · 1 quote
Strict vision-language padding mask optional · 1 source · 1 quote
Cascade attention not used · 2 sources · 2 quotes

global attention 5

Global attention core · 6 sources · 6 quotes
Unified Keys and Values core · 2 sources · 2 quotes
Full attention used · 2 sources · 2 quotes
2 single mentions
Full attention without positional encoding core · 1 source · 1 quote

sliding window attention 11

Sliding Window Attention core · 12 sources · 16 quotes
Hybrid Sliding Window Attention core · 4 sources · 5 quotes
9 single mentions
Sink-Augmented Sliding Window Attention core · 1 source · 1 quote
Decoder SWA Bounded Replay used · 1 source · 1 quote
Dynamic Attention Window Size Training used · 1 source · 1 quote
SWA-128 used · 1 source · 1 quote
Per-Head Gating evaluated · 1 source · 1 quote
SWA-1024 (3:1 Interleaving) evaluated · 1 source · 1 quote
SWA-512 evaluated · 1 source · 1 quote

grouped-query attention 5

Grouped-query attention core · 12 sources · 15 quotes
Multi-query attention used · 2 sources · 2 quotes
3 single mentions
Block-sparse grouped-query attention core · 1 source · 1 quote
Softplus-based per-head attention gating used · 1 source · 1 quote

multi-head latent attention 5

Gated Multi-head Latent Attention core · 3 sources · 3 quotes
Multi-head Latent Attention core · 3 sources · 4 quotes
MQA mode of MLA core · 2 sources · 2 quotes
2 from one source, or only mentioned as not used
MHA and MQA modes of MLA used · 1 source · 1 quote
Ragged Multi-head Latent Attention not used · 1 source · 1 quote

attention sink 4

Learnable attention sink bias used · 4 sources · 4 quotes
Learnable attention sink token used · 2 sources · 2 quotes
2 single mentions
Attention sink used · 1 source · 1 quote
RoPE with attention sink unclear · 1 source · 1 quote

sparse attention 60

DeepSeek Sparse Attention core · 16 sources · 18 quotes
Sparse attention core · 8 sources · 8 quotes
Compressed Sparse Attention core · 6 sources · 6 quotes
Qwen Sparse Attention core · 6 sources · 6 quotes
Heavily Compressed Attention core · 5 sources · 5 quotes
Gated DeepSeek Sparse Attention core · 3 sources · 3 quotes
Compressed Sparse Attention 2 core · 2 sources · 3 quotes
KV-outer sparse attention used · 2 sources · 2 quotes
15 from one source, or only mentioned as not used
CSA2 Full, Reindex, and Reuse modes core · 1 source · 1 quote
Fixed-budget sparse-attention selection core · 1 source · 1 quote
NoPE sparse multi-head latent attention core · 1 source · 1 quote
QSA micro-block compression at ratio 4 core · 1 source · 1 quote
Token-wise compression core · 1 source · 1 quote
FlashMLA sparse attention default · 1 source · 1 quote
Natively trained sparsity used · 1 source · 1 quote
Sparse softmax attention used · 1 source · 1 quote
Two-stage sparse-attention formulation used · 1 source · 1 quote
Two-stage sparse-attention training used · 1 source · 1 quote
Native Sparse Attention not used · 1 source · 1 quote

block selection 9

MiniMax Sparse Attention core · 7 sources · 8 quotes
Top-k block selection used · 2 sources · 2 quotes
7 single mentions
Block-level selection core · 1 source · 1 quote
Dynamic sparse selection core · 1 source · 1 quote
Main Branch core · 1 source · 2 quotes
Top-512 block selection core · 1 source · 1 quote
Block Max Pooling used · 1 source · 1 quote
Top-scoring compressed-block selection used · 1 source · 1 quote
Mixture of Block Attention evaluated · 1 source · 1 quote

sparse attention indexer 24

IndexCache core · 3 sources · 3 quotes
IndexShare core · 3 sources · 3 quotes
Lightning Indexer core · 3 sources · 3 quotes
Fine-grained token selection mechanism core · 2 sources · 2 quotes
Hierarchical Sparse Indexer core · 2 sources · 3 quotes
17 from one source, or only mentioned as not used
Compressed lightweight indexer core · 1 source · 1 quote
CSA2 compressor and indexer design core · 1 source · 1 quote
DeepSeek Sparse Attention indexer core · 1 source · 1 quote
Index Branch core · 1 source · 2 quotes
IndexPool core · 1 source · 1 quote
MQA indexer core · 1 source · 1 quote
Average pooling used · 1 source · 1 quote
Block-causal scoring used · 1 source · 1 quote
Detached indexer-input optimization used · 1 source · 1 quote
Indexer warmup used · 1 source · 2 quotes
KL-distillation indexer warm-up used · 1 source · 1 quote
Single-head index key used · 1 source · 1 quote
Top-2,048 key-token selection per query used · 1 source · 1 quote
Index Branch output not used · 1 source · 1 quote
Index value head not used · 1 source · 1 quote

fixed-pattern sparse attention 3

all mentioned once or only as not used
Sparse Global Attention Anchors core · 1 source · 1 quote
Local Block used · 1 source · 1 quote
Forced Sink and Local Window not used · 1 source · 1 quote

linear attention & state space 18

Linear attention core · 2 sources · 2 quotes
1 only mentioned as not used
Lightning Attention not used · 1 source · 1 quote

Mamba 4

Mamba-2 core · 5 sources · 6 quotes
Mamba core · 2 sources · 2 quotes
2 single mentions
Mamba checkpointing core · 1 source · 1 quote
Mamba-2 SSM cache optional · 1 source · 1 quote

gated delta network 12

Gated DeltaNet core · 12 sources · 12 quotes
Kimi Delta Attention core · 6 sources · 6 quotes
KDA core · 2 sources · 2 quotes
8 single mentions
Gated attention with reduced KV heads core · 1 source · 1 quote
Gated DeltaNet hybrid attention core · 1 source · 1 quote
Hybrid linear attention mechanism core · 1 source · 1 quote
Bounded sigmoid output gate used · 1 source · 1 quote
Input-dependent full-rank output gate used · 1 source · 1 quote
Lower-bounded scaled-sigmoid log-decay used · 1 source · 1 quote
SimpleGDN evaluated · 1 source · 1 quote

hybrid layer stacking 12

Hybrid Sliding-Window and Global Attention core · 21 sources · 23 quotes
Gated DeltaNet and full attention core · 7 sources · 9 quotes
Hybrid Mamba-Transformer core · 6 sources · 9 quotes
Hybrid Mamba-Attention core · 3 sources · 4 quotes
Hybrid Sparse and Linear Attention core · 3 sources · 3 quotes
Hybrid compressed attention (CSA and HCA) used · 3 sources · 3 quotes
4 single mentions
Search-Based Sliding-Window Attention Pattern evaluated · 1 source · 1 quote
Dense Attention Fallback unclear · 1 source · 1 quote

channel mixer 86

3 single mentions
Feature-wise Linear Modulation (FiLM) optional · 1 source · 2 quotes
Parallel MLP-Mixer-based action decoder evaluated · 1 source · 1 quote

dense feed-forward network 7

SiTU-GLU core · 2 sources · 3 quotes
SwiGLU used · 2 sources · 2 quotes
SwiGLU clamping used · 2 sources · 2 quotes
4 from one source, or only mentioned as not used
Dense feed-forward network used · 1 source · 1 quote
Sigmoid Tanh Unit (SiTU) used · 1 source · 1 quote
SwiGLU-clip not used · 1 source · 1 quote

mixture of experts 76

Mixture of experts core · 113 sources · 135 quotes
Sparse expert activation core · 9 sources · 9 quotes
Action expert core · 2 sources · 2 quotes
Asymmetric input/output parameter activation core · 2 sources · 2 quotes
DeepSeekMoE core · 2 sources · 2 quotes
12 from one source, or only mentioned as not used
Dispatch recomputation core · 1 source · 1 quote
Gated DeltaNet MoE core · 1 source · 1 quote
Hybrid instruction-and-reasoning MoE core · 1 source · 1 quote
Hybrid Mixture of Experts core · 1 source · 1 quote
MegaMoE used · 1 source · 1 quote
Routed expert modulation used · 1 source · 1 quote
Routed-expert silencing test evaluated · 1 source · 1 quote
MoE layer densification by expert concatenation not used · 1 source · 1 quote

expert routing 15

Top-k expert routing core · 11 sources · 12 quotes
Anticipatory Routing used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Keep Routing core · 1 source · 1 quote
Sigmoid-based expert routing core · 1 source · 1 quote
Token-choice routing with softplus gating core · 1 source · 1 quote
Top-k routing with frozen bias default · 1 source · 1 quote
DP-aware routing used · 1 source · 1 quote
Hash routing used · 1 source · 1 quote
Latent-space routing used · 1 source · 1 quote
Token-choice routing used · 1 source · 1 quote
Loss-spike-triggered Anticipatory Routing optional · 1 source · 1 quote
Random routed-expert selection evaluated · 1 source · 1 quote
Heterogeneity-guided MoE routing not used · 1 source · 1 quote

expert load balancing 25

Auxiliary-loss-free load balancing core · 3 sources · 4 quotes
Quantile Balancing core · 3 sources · 5 quotes
Exact Quantile Balancing core · 2 sources · 3 quotes
Load-error gradient injection core · 2 sources · 3 quotes
Load balancing used · 2 sources · 2 quotes
20 from one source, or only mentioned as not used
Modality-specific load balancing core · 1 source · 2 quotes
Per-expert routing correction bias core · 1 source · 1 quote
Auxiliary-loss load balancing used · 1 source · 1 quote
Disabled router-bias updates used · 1 source · 1 quote
Exact coordinate minimization used · 1 source · 1 quote
Expert bias update factor used · 1 source · 1 quote
Expert Parallelism Load Balancing used · 1 source · 1 quote
Expert-parallel-group load-balancing loss used · 1 source · 1 quote
Histogram-based global quantile estimation used · 1 source · 1 quote
Mixture-of-experts load balancing used · 1 source · 2 quotes
Persistent load balancing used · 1 source · 1 quote
Pooled-global-batch quantile estimation used · 1 source · 1 quote
Recentered global-quantile target update used · 1 source · 1 quote
Two-pass radix selection used · 1 source · 1 quote
Uniform-bin histogram quantile estimation used · 1 source · 1 quote
Round-robin load balancing optional · 1 source · 1 quote
MaxVio evaluated · 1 source · 1 quote
Load-balancing regularization not used · 1 source · 1 quote

shared experts 6

Shared expert core · 13 sources · 13 quotes
MoE without shared experts core · 3 sources · 3 quotes
2 single mentions
MoE with expert sinks core · 1 source · 1 quote
Shared expert isolation core · 1 source · 1 quote

fine-grained experts 8

Expert granularity evaluated · 3 sources · 3 quotes
7 single mentions
Fine-grained expert segmentation core · 1 source · 1 quote
Many small experts used · 1 source · 1 quote
MoE granularity 4 used · 1 source · 1 quote
Coarse-grained MoE configuration evaluated · 1 source · 1 quote
Fine-grained MoE configuration evaluated · 1 source · 1 quote

latent mixture of experts 5

LatentMoE core · 7 sources · 9 quotes
Stable LatentMoE core · 6 sources · 7 quotes
3 single mentions
Hybrid LatentMoE core · 1 source · 1 quote

positional encoding 24

Rotary Position Embedding core · 6 sources · 6 quotes
No Position Encoding core · 3 sources · 4 quotes
Partial rotary position embeddings core · 3 sources · 3 quotes
Learned relative attention core · 2 sources · 2 quotes
LeRoPE core · 2 sources · 2 quotes
YaRN used · 11 sources · 11 quotes
Static YaRN used · 6 sources · 6 quotes
2D rotary position embeddings used · 2 sources · 2 quotes
Proportional Rotary Position Embedding used · 2 sources · 2 quotes
RoPE scaling optional · 2 sources · 2 quotes
YaRN scaling-factor selection optional · 2 sources · 3 quotes
13 from one source, or only mentioned as not used
Alternating RoPE/NoPE layers core · 1 source · 1 quote
Per-layer-type rotary position scales core · 1 source · 1 quote
UV positional encodings core · 1 source · 1 quote
2D coordinate-based positional embeddings used · 1 source · 1 quote
Axial 2D-RoPE used · 1 source · 1 quote
RoPE positional-index offset used · 1 source · 1 quote
RoPE scaling for context extension (YaRN) optional · 1 source · 1 quote
YaRN configuration optional · 1 source · 1 quote
Gated attention with partial RoPE evaluated · 1 source · 1 quote
Temporal-Modality Rotary Position Embedding not used · 1 source · 1 quote

normalization & residual 36

Manifold-Constrained Hyper-Connections core · 9 sources · 9 quotes
Attention Residuals core · 7 sources · 7 quotes
Gated Residual core · 6 sources · 8 quotes
Adaptive Layer Normalization (AdaLN) core · 3 sources · 3 quotes
Identity Hyper-Connections core · 3 sources · 3 quotes
Head-wise gated attention core · 2 sources · 2 quotes
Hyper-Connections core · 2 sources · 2 quotes
Per-branch scalar residual write gate core · 2 sources · 2 quotes
QK Normalization core · 2 sources · 3 quotes
Sandwich normalization core · 2 sources · 2 quotes
Single-Pass mHC core · 2 sources · 2 quotes
RMSNorm used · 2 sources · 2 quotes
QK-Clip evaluated · 3 sources · 3 quotes
23 from one source, or only mentioned as not used
Block Attention Residuals core · 1 source · 2 quotes
Four-branch residual stream core · 1 source · 1 quote
GatedNorm core · 1 source · 3 quotes
Learned per-layer depth gate core · 1 source · 2 quotes
Two-phase Block AttnRes schedule core · 1 source · 1 quote
Widened residual stream core · 1 source · 1 quote
Adaptive RMSNorm used · 1 source · 1 quote
Bounded positive gates used · 1 source · 1 quote
Depth-conditioning gate initialization used · 1 source · 1 quote
Learned depth gate on expert conditioning used · 1 source · 1 quote
Post-embedding RMSNorm used · 1 source · 1 quote
Pre-LN used · 1 source · 1 quote
Zero-centered RMSNorm used · 1 source · 1 quote
Simplified AltUp variant evaluated · 1 source · 1 quote
Full Attention Residuals mentioned · 1 source · 1 quote
Sparse gated residual writes not used · 1 source · 1 quote

prediction head 27

Multi-Token Prediction core · 33 sources · 40 quotes
MTP speculative decoding core · 11 sources · 23 quotes
Flow-matching action head core · 3 sources · 3 quotes
Frozen-backbone MTP-head-only fine-tuning core · 3 sources · 5 quotes
Shared Multi-Token Prediction head core · 3 sources · 3 quotes
Diffusion-policy action head used · 2 sources · 2 quotes
20 single mentions
Diffusion-based trajectory decoder core · 1 source · 1 quote
Discrete autoregressive robot policy core · 1 source · 1 quote
Parallel action decoding core · 1 source · 2 quotes
Diffusion-Transformer-style action expert used · 1 source · 1 quote
MTP-head warm-up with a frozen backbone used · 1 source · 1 quote
MTP-only training used · 1 source · 1 quote
Nemotron Hybrid Multi-Token Prediction optional · 1 source · 1 quote
Autoregressive discrete action-tokenization head mentioned · 1 source · 1 quote
Regression-head action prediction mentioned · 1 source · 1 quote

multimodal architecture 154

Encoder-free multimodal architecture core · 6 sources · 6 quotes
Early-fusion multimodal training core · 5 sources · 5 quotes
Native multimodal architecture core · 5 sources · 5 quotes
Discrete audio token encoding core · 4 sources · 4 quotes
Hierarchical patch encoder core · 4 sources · 4 quotes
Mixed-modality training from the start core · 4 sources · 4 quotes
Separate flow-matching action expert core · 3 sources · 3 quotes
C-RADIO vision encoder core · 2 sources · 2 quotes
Multimodal input core · 2 sources · 2 quotes
Native omnimodal architecture core · 2 sources · 2 quotes
Per-layer key-value conditioning core · 2 sources · 5 quotes
Reduced action-expert hidden size core · 2 sources · 2 quotes
Small pretrained VLM backbone core · 2 sources · 2 quotes
Visual token reduction core · 2 sources · 2 quotes
3×3 PixelUnshuffle downsampling used · 2 sources · 2 quotes
Raw audio projection into LLM embedding space used · 2 sources · 2 quotes
Vision Transformer (ViT) used · 2 sources · 2 quotes
Configurable visual token budget optional · 3 sources · 3 quotes
134 single mentions
Continuous action representation core · 1 source · 1 quote
Continuous DiT-style action expert core · 1 source · 1 quote
Convolution-attention vision encoder core · 1 source · 1 quote
ConvStack core · 1 source · 1 quote
Cross-embodiment modeling core · 1 source · 1 quote
Depth-aware perception tokens core · 1 source · 2 quotes
Dual proprioception conditioning core · 1 source · 1 quote
Dual-system compositional architecture core · 1 source · 1 quote
Eight-codebook Mimi RVQ codes core · 1 source · 1 quote
Embodiment-indexed proprioception MLP core · 1 source · 1 quote
Embodiment-specific action heads core · 1 source · 1 quote
Fused SigLIP-DINOv2 vision encoder core · 1 source · 1 quote
Hierarchical convolutional decoder core · 1 source · 1 quote
Intermediate VLM-layer features core · 1 source · 1 quote
Joint multimodal input understanding core · 1 source · 1 quote
Language backbone core · 1 source · 1 quote
Last-four-layer action-head conditioning core · 1 source · 1 quote
Learnable image queries core · 1 source · 1 quote
Multi-layer ViT feature concatenation core · 1 source · 1 quote
Multi-scale image cropping core · 1 source · 1 quote
Multi-view image input core · 1 source · 1 quote
Multimodal input understanding core · 1 source · 1 quote
Native visual and audio understanding core · 1 source · 1 quote
Pairwise sampled-frame embedding core · 1 source · 1 quote
Per-embodiment action MLPs core · 1 source · 1 quote
Perceptual-query token appending core · 1 source · 1 quote
Pixtral vision encoder core · 1 source · 1 quote
Robot-specific input and output layers core · 1 source · 1 quote
Separate RGB and depth patch embeddings core · 1 source · 1 quote
Shared action trunk core · 1 source · 1 quote
SigLIP2 pretrained vision transformer core · 1 source · 1 quote
Single resized crop per image core · 1 source · 2 quotes
Spatial patch-merge projector core · 1 source · 1 quote
T5 pretrained text encoder core · 1 source · 1 quote
Thinker–Talker architecture core · 1 source · 1 quote
Three-stage multimodal training protocol core · 1 source · 1 quote
Time-conditioned DiT modulation core · 1 source · 1 quote
Vision encoder core · 1 source · 1 quote
Vision encoder and multimodal aligner core · 1 source · 1 quote
Vision-language connector core · 1 source · 1 quote
Vision-language model backbone core · 1 source · 1 quote
Global-image-only runtime input default · 1 source · 1 quote
ASR-based connector alignment used · 1 source · 1 quote
Audio Transformer (AuT) used · 1 source · 1 quote
Causal streaming ConvNet codec decoder used · 1 source · 1 quote
Cross-attention-based resampler used · 1 source · 2 quotes
Data-parallel-first multimodal encoding used · 1 source · 1 quote
Decoupled Encoder Process (DEP) used · 1 source · 1 quote
Domain-specific resamplers and action heads used · 1 source · 2 quotes
Dual-format coordinate supervision used · 1 source · 1 quote
Dynamic image tiling used · 1 source · 1 quote
End-to-end autonomous driving used · 1 source · 1 quote
Explicit text-string timestamps used · 1 source · 1 quote
FiLM used · 1 source · 1 quote
Frozen encoders during pre-training used · 1 source · 1 quote
Grid quantization of depth maps used · 1 source · 1 quote
Index-token marking for multi-image inputs used · 1 source · 1 quote
Joint full-model multimodal training used · 1 source · 1 quote
Learned image-padding-status embeddings used · 1 source · 1 quote
Lightweight vision embedding module used · 1 source · 1 quote
Masking missing robot image slots used · 1 source · 1 quote
MiMo-Audio-initialized audio encoder used · 1 source · 1 quote
Molmo2-ER backbone used · 1 source · 1 quote
MoonViT-V2 used · 1 source · 1 quote
Multi-resolution visual feature fusion used · 1 source · 1 quote
Parallel action prediction used · 1 source · 1 quote
PixelUnshuffle visual-token downsampling used · 1 source · 1 quote
Pre-trained vision and language encoders used · 1 source · 1 quote
Pretrained vision-language encoder used · 1 source · 1 quote
Projector warmup used · 1 source · 1 quote
Repurposed navigation vocabulary tokens used · 1 source · 1 quote
Residual-quantized trajectory primitives used · 1 source · 1 quote
Reusing image tokens for video frames used · 1 source · 1 quote
RVQ-based speech representation used · 1 source · 1 quote
Separate robotics-specific token weights used · 1 source · 1 quote
Shared frozen encoders used · 1 source · 1 quote
Shared vision backbone used · 1 source · 1 quote
Single-matmul vision projection used · 1 source · 1 quote
Static third-person camera input used · 1 source · 1 quote
Two-layer MLP vision projector used · 1 source · 1 quote
Universal Speech Model-based audio encoder used · 1 source · 1 quote
Variable aspect-ratio image handling used · 1 source · 1 quote
Video-frame pruning used · 1 source · 1 quote
Autoregressive depth-token prefix conditioning optional · 1 source · 1 quote
Depth injection optional · 1 source · 1 quote
Global thumbnail alongside tiled image inputs optional · 1 source · 1 quote
Video preprocessor longest-edge configuration optional · 1 source · 1 quote
Action-modality isolation evaluated · 1 source · 1 quote
Hidden-state conditioning evaluated · 1 source · 1 quote
HPT-style domain-specific input projection evaluated · 1 source · 1 quote
Multimodal Diffusion Transformer decoder evaluated · 1 source · 1 quote
Natural-language hardware-configuration prompts evaluated · 1 source · 1 quote
Per-head per-layer key-value conditioning evaluated · 1 source · 1 quote
Single SigLIP vision encoder evaluated · 1 source · 1 quote
Triplane-based multi-camera tokenization evaluated · 1 source · 1 quote
Vision-and-language navigation evaluated · 1 source · 1 quote

context capacity 14

1M-token context window core · 5 sources · 5 quotes
N-gram embedding lookup memory core · 5 sources · 5 quotes
Engram core · 3 sources · 3 quotes
Host-memory offload for N-gram embeddings used · 3 sources · 3 quotes
Progressive context extension used · 2 sources · 2 quotes
8 single mentions
256K context window core · 1 source · 1 quote
Single-layer N-gram embedding placement core · 1 source · 1 quote
Enabling 1M context in Step Plan optional · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled heretoken mixerchannel mixerpositional encodingnormalization & residualprediction headmultimodal architecturecontext capacity
DeepSeek-V4.1-Flash-1017 coreCausal Encoder-Decoder (CED) core——Asymmetric input/output parameter activation coreMixture of experts core————Joint vision-language pre-training from the start core——
GLM-5.3-Flash core—Hybrid Sparse and Linear Attention coreIndexPool coreKimi Delta Attention coreNoPE sparse multi-head latent attention core—Mixture of experts coreTop-k expert routing core——Manifold-Constrained Hyper-Connections core——Reusing image tokens for video frames used——
DeepSeek-V4.1-Flash coreAsymmetric architecture coreCausal Encoder-Decoder (CED) core—Compressed Sparse Attention 2 coreCSA2 compressor and indexer design coreCSA2 Full, Reindex, and Reuse modes coreHierarchical Sparse Indexer coreDecoder SWA Bounded Replay usedProgressive context-length extension for sparse attention usedSparse attention training from scratch without dense-attention warmup used—Asymmetric input/output parameter activation coreMixture of experts coreModality-specific load balancing core—2D rotary position embeddings used—Single-Pass mHC coreManifold-Constrained Hyper-Connections used——Joint vision-language pre-training from the start core3×3 PixelUnshuffle downsampling usedTwo-layer MLP vision projector used—Engram core—
Hy4-preview core—Gated DeepSeek Sparse Attention coreIndexCache core—Mixture of experts coreShared expert coreTop-k expert routing core——Identity Hyper-Connections core—MTP speculative decoding core———
Step-5-Preview core——Mixture of experts core————Multimodal input understanding core—1M-token context window coreEnabling 1M context in Step Plan optional—
MiniMax-M3.1-Flash core—MiniMax Sparse Attention coreSparse attention coreAggressive sparsity for inference efficiency used—Mixture of experts core——————
DeepSeek-V4-Flash-0731 core——————Native multimodal architecture core——
Hy3 core—Grouped-query attention core—Mixture of experts coreShared expert coreSparse expert activation coreTop-k expert routing core———Multi-Token Prediction core———
MiMo-V2.6-Flash core—Alternating Row-Major and Column-Major Token Serialization coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core—Mixture of experts coreMoE without shared experts core———Multi-Token Prediction core—Four-frame audio patches with within-patch bidirectional attention coreJoint multimodal input understanding coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used—1M-token context window core—
NVIDIA-Nemotron-3-Ultra-550B-A55B core—Attention layers coreHybrid Mamba-Attention coreHybrid Mamba-Transformer coreMamba coreMamba-2 coreSparse Global Attention Anchors coreMamba-2 SSM cache optional—Hybrid LatentMoE coreLatentMoE coreMixture of experts coreExpert Parallelism Load Balancing usedShared expert usedMaxVio evaluatedExpert granularity not used———MTP speculative decoding coreMulti-Token Prediction coreHidden-state mixing during recursive MTP training usedShared Multi-Token Prediction head usedNemotron Hybrid Multi-Token Prediction optional———
DeepSeek-V4-Flash core—Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid compressed attention (CSA and HCA) usedMulti-query attention usedSliding Window Attention usedTwo-stage sparse-attention training used—DeepSeekMoE coreMixture of experts coreShared expert coreAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedAnticipatory Routing optionalLoss-spike-triggered Anticipatory Routing optional—Rotary Position Embedding used—Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—Multi-Token Prediction core——1M-token context window default—
Alpamayo 1.5 / 2 Super core—————Diffusion-based trajectory decoder core—Chain-of-thought reasoning vision-language-action model coreSeparate flow-matching action expert coreEnd-to-end autonomous driving usedTriplane-based multi-camera tokenization evaluated——
DeepSeek-V3.2 core—DeepSeek Sparse Attention coreFine-grained token selection mechanism coreLightning Indexer coreMQA mode of MLA coreDense attention warm-up with a frozen lightning indexer usedDetached indexer-input optimization usedMHA and MQA modes of MLA usedSparse-attention continued pretraining with joint model and indexer optimization used—Keep Routing core——————
DeepSeek-V4-Flash-Vision-Exp core—DFlash attention core—Mixture of experts core——Hyper-Connections core——Native multimodal architecture coreVision encoder and multimodal aligner coreVisual modules for multimodal understanding core——
DeepSeek-V4-Pro core—Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid compressed attention (CSA and HCA) usedMulti-query attention usedSliding Window Attention usedTwo-stage sparse-attention training used—DeepSeekMoE coreMixture of experts coreShared expert coreAnticipatory Routing usedAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedLoss-spike-triggered Anticipatory Routing optional—Rotary Position Embedding used—Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—Multi-Token Prediction core——1M-token context window defaultEngram not used—
DeepSeek-V4-Pro-0813 core——————Native multimodal architecture core——
FLUX 3 Action coreDiffusion Transformer (DiT) core——————Embodiment-specific action heads coreRobot-specific input and output layers coreShared action trunk coreShared frozen encoders used——
GLM-5.2 core—DeepSeek Sparse Attention coreIndexShare coreMulti-head Latent Attention coreLightning Indexer usedSparse attention usedGated DeltaNet evaluatedSearch-Based Sliding-Window Attention Pattern evaluatedSimpleGDN evaluatedSliding Window Attention evaluatedMulti-query attention mentioned—Mixture of experts coreDP-aware routing used———MTP speculative decoding usedMulti-Token Prediction used———
GR00T N1.7 coreDiffusion Transformer (DiT) coreFlow-matching transformer coreSemantically partitioned state and action vectors with field metadata used—Cross-attention coreStrict action padding mask optionalStrict vision-language padding mask optional———Adaptive Layer Normalization (AdaLN) core—Diffusion Transformer IDM using SigLIP-2 vision embeddings used—Cross-embodiment modeling coreDual-system compositional architecture coreEmbodiment-indexed proprioception MLP coreEmbodiment-specific state and action projectors corePer-embodiment action MLPs coreSigLIP2 pretrained vision transformer coreT5 pretrained text encoder coreVision-language model with diffusion-transformer action head coreMiddle-layer VLM representations for policy conditioning usedPre-trained vision and language encoders used——
Gemma 4 31B coreDecoder-only Transformer architecture corePer-Layer Embeddings (PLE) coreDense model architecture used—Hybrid Sliding-Window and Global Attention coreUnified Keys and Values core—Mixture of experts coreSparse expert activation core—2D coordinate-based positional embeddings usedAxial 2D-RoPE usedProportional Rotary Position Embedding used——MTP speculative decoding core—Encoder-free multimodal architecture coreDedicated multimodal encoders before the LLM usedFrozen encoders during pre-training usedLightweight vision embedding module usedRaw audio projection into LLM embedding space usedSingle-matmul vision projection usedUniversal Speech Model-based audio encoder usedVariable aspect-ratio image handling usedConfigurable visual token budget optional——
Inkling coreRegular Transformer decoder coreShort convolutions in attention and residual branches used—Hybrid Sliding-Window and Global Attention coreGrouped-query attention usedKernel-4 convolutions in decoder blocks usedSliding Window Attention used—Mixture of experts coreMoE with expert sinks coreShared expert coreTop-k expert routing coreSigmoid router with auxiliary-loss-free load balancing used—Learned relative attention coreLearned input-dependent relative-position bias used—Post-embedding RMSNorm used——Discrete audio token encoding coreHierarchical patch encoder coreJoint multimodal decoding in a shared hidden space coreNative multimodal architecture coreEncoder-free multimodal architecture used——
Kimi K3 core—Gated Multi-head Latent Attention coreHybrid linear attention mechanism coreKDA coreKimi Delta Attention coreKimi Delta Attention–Attention Residuals architecture coreLayerwise Hybrid Linear and Global Attention coreFactorized spatial-temporal attention usedGated DeltaNet usedInput-dependent full-rank output gate usedLower-bounded scaled-sigmoid log-decay used—Auxiliary-loss-free load balancing coreDispatch recomputation coreLatentMoE coreMixture of experts coreOutput-independent MoE gradient reformulation coreQuantile Balancing coreSharded latent weights with fused all-gather GEMM epilogue coreSiTU-GLU coreSparse expert activation coreStable LatentMoE coreTop-k routing with frozen bias defaultExact coordinate minimization usedExponential moving average of estimated quantiles usedHistogram-based global quantile estimation usedLatent-space routing usedPooled-global-batch quantile estimation usedSigmoid Tanh Unit (SiTU) usedUniform-bin histogram quantile estimation used—No Position Encoding core—Attention Residuals coreBlock Attention Residuals coreTwo-phase Block AttnRes schedule coreFull Attention Residuals mentioned——Native multimodal architecture coreDecoupled Encoder Process (DEP) usedDual-format coordinate supervision usedFrom-scratch MoonViT-V2 training with next-token prediction usedMoonViT-V2 usedMultimodal input usedPixelShuffle 2×2 visual-token downsampling used—Progressive context extension used—
Kolibri-1 coreResidual-stream width selection evaluated—Full attention without positional encoding coreHybrid Sliding-Window and Global Attention coreSliding Window Attention core—Exact Quantile Balancing coreLoad-error gradient injection coreMixture of experts coreShared expert coreTop-k expert routing coreMany small experts usedMixture-of-experts load balancing usedRecentered global-quantile target update usedTwo-pass radix selection usedCoarse-grained MoE configuration evaluatedExpert granularity evaluatedFine-grained MoE configuration evaluatedRandom routed-expert selection evaluatedRouted-expert silencing test evaluatedLeading dense feed-forward layers before MoE layers not usedMoE layer densification by expert concatenation not used—No Position Encoding coreStatic YaRN used—QK Normalization coreQK-Clip evaluated———Context-length extrapolation without positional rescaling used—
LFM3-1.7B coreHybrid architecture core—Gated short convolution coreGlobal attention coreGrouped-query attention coreInput-dependent multiplicative gating around a depthwise short convolution coreKernel-size-7 convolution coreSliding Window Attention coreSparse use of grouped-query attention blocks for global context core—Mixture of experts coreTop-k routing with normalized sigmoid and adaptive load-balancing bias coreDense-first-layer, sparse-MoE-later-layer placement used—Full RoPE in sliding-window attention layers corePartial rotary position embeddings core—Head-wise gated attention coreSandwich normalization core—Residual-MLP action head for single-pass prediction of eight-step action chunks core—Causal audio encoder with sliding-window attention and short convolutions coreConvolution-attention vision encoder coreDiscrete Residual Vector Quantization audio generation coreDual proprioception conditioning coreEight-codebook Mimi RVQ codes corePairwise sampled-frame embedding coreThree-stage multimodal training protocol coreVision-language extension with feature connector coreASR-based connector alignment usedConnector pre-training with frozen encoders usedDynamic image tiling usedFixed 6:12 text-to-audio token interleaving usedParallel action prediction usedPixelUnshuffle visual-token downsampling usedRepurposed navigation vocabulary tokens usedResidual-quantized trajectory primitives usedSigLIP2 vision encoder with dynamic tiling and token reduction usedSingle-pass continuous action chunk prediction usedVisual route following with target and waypoint prediction usedGlobal thumbnail alongside tiled image inputs optionalVision-and-language navigation evaluated——
Laguna-S-2.1 core—Grouped-query attention coreHybrid Sliding-Window and Global Attention coreSoftplus attention gating coreGated DeltaNet and full attention usedSoftplus-based per-head attention gating usedDense gated attention with full RoPE and full gating evaluatedPer-Head Gating evaluatedSWA-1024 (3:1 Interleaving) evaluatedSWA-512 evaluated—Mixture of experts coreSparse expert activation coreToken-choice routing with softplus gating coreAuxiliary-loss load balancing usedRouted expert modulation usedShared expert usedToken-choice routing used—Per-layer-type rotary position scales coreRotary Position Embedding usedGated attention with partial RoPE evaluated—————
LingBot-VLA 2.0 coreMixture-of-Transformers (MoT) core—Blockwise causal attention coreShared self-attention across modalities core—Auxiliary-loss-free load balancing coreFine-grained expert segmentation coreFine-grained routed experts with a shared expert coreMixture of experts corePer-expert routing correction bias coreShared expert isolation coreSigmoid-based expert routing coreSign-based expert-load correction-bias updates core—Shared spatial and modality-specific positional embeddings coreUV positional encodings core——Diffusion-policy action head used—ConvStack coreHierarchical convolutional decoder coreLearnable image queries corePerceptual-query token appending coreSeparate RGB and depth patch embeddings coreDINOv2 and Point Transformer grasp-policy conditioning usedDepth injection optional——
MiMo-V2.5 core—Global attention coreHybrid Sliding Window Attention coreHybrid Sliding-Window and Global Attention coreSliding Window Attention coreFull attention usedGrouped-query attention usedLearnable attention sink bias usedSWA-128 used—Mixture of experts coreDense feed-forward network usedExpert bias update factor usedLoad balancing usedTop-k expert routing usedRound-robin load balancing optionalMoE without shared experts not used—Rotary Position Embedding used——Multi-Token Prediction coreMTP speculative decoding used—Native omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedMiMo-Audio-initialized audio encoder usedProjector warmup usedVision Transformer (ViT) used—1M-token context window usedProgressive context extension used—
MiMo-V2.5-Pro core—Hybrid Sliding-Window and Global Attention coreGlobal attention usedHybrid Sliding Window Attention usedLearnable attention sink bias usedSliding Window Attention used—Mixture of experts core———MTP speculative decoding usedMulti-Token Prediction used—Native multimodal architecture coreNative omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedProjector warmup used—1M-token context window usedProgressive context extension used—
MiMo-V2.6-Pro core—Alternating Row-Major and Column-Major Token Serialization coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core—Mixture of experts coreMoE without shared experts coreSparse expert activation core———Multi-Token Prediction core—Four-frame audio patches with within-patch bidirectional attention coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used——
MiniMax-M3 core—Block-level selection coreBlock-sparse grouped-query attention coreDynamic sparse selection coreGrouped-query attention coreIndex Branch coreMain Branch coreMiniMax Sparse Attention coreSparse attention coreBlock Max Pooling usedIndexer warmup usedKV-outer sparse attention usedLocal Block usedNatively trained sparsity usedSingle-head index key usedSparse softmax attention usedTop-k block selection usedTwo-stage sparse-attention formulation usedDeepSeek Sparse Attention evaluatedFull attention evaluatedMixture of Block Attention evaluatedSliding Window Attention evaluatedLinear attention mentionedCompressed Sparse Attention not usedDense Attention Fallback unclearForced Sink and Local Window not usedIndex Branch output not usedIndex value head not usedLearnable attention sink token not usedLightning Attention not usedMulti-head Latent Attention not usedNative Sparse Attention not usedRoPE with attention sink unclear—Mixture of experts corePersistent load balancing usedShared expert usedTop-k expert routing used—Rotary Position Embedding used———Mixed-modality training from the start core——
Mistral-Large-4 core—DeepSeek Sparse Attention indexer coreMulti-head Latent Attention coreMulti-head latent attention with a DSA sparse indexer coreSparse attention coreFlashMLA sparse attention defaultDeepSeek Sparse Attention usedKL-distillation indexer warm-up usedShared DSA indexer across sliding-window layers usedSSMax usedTop-2,048 key-token selection per query usedCross-layer attention sharing not usedRagged Multi-head Latent Attention not usedSliding Window Attention not used—Granular Mixture-of-Experts with routed and shared experts coreHybrid instruction-and-reasoning MoE coreMixture of experts coreShared expert coreTop-k expert routing coreExpert-parallel-group load-balancing loss usedMoE granularity 4 usedLoad balancing mentioned—Alternating RoPE/NoPE layers coreLeRoPE coreYaRN evaluated———Multimodal input coreNative multimodal architecture corePixtral vision encoder corePixtral-style variable-resolution vision encoder coreSpatial patch-merge projector core——
MolmoAct 2 coreFlow-matching action expert core—Attention-based pooling of local visual windows coreBidirectional self-attention with VLM cross-attention coreForward attention among visual tokens coreNoncausal transformer action expert coreWindowed multi-head attention pooling coreScaled Dot-Product Attention (SDPA) used——Rotary Position Embedding core—Learned per-layer depth gate coreDepth-conditioning gate initialization usedLearned depth gate on expert conditioning used—Discrete autoregressive robot policy corePredict discrete depth tokens before action generation corePredicting end-effector trajectory traces as intermediate visual plans coreSequential depth, trajectory, and action token generation coreDiffusion-Transformer-style action expert used—Continuous DiT-style action expert coreDepth-aware perception tokens coreMasking action tokens from expert conditioning coreMulti-layer ViT feature concatenation coreMulti-scale image cropping corePer-layer key-value conditioning coreSingle resized crop per image coreTime-conditioned DiT modulation coreGrid quantization of depth maps usedIndex-token marking for multi-image inputs usedLearned image-padding-status embeddings usedMolmo2-ER backbone usedNear-zero action-expert output initialization usedAutoregressive depth-token prefix conditioning optionalHidden-state conditioning evaluatedPer-head per-layer key-value conditioning evaluated——
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core—Grouped dynamic convolution coreHybrid Mamba-2 and Mixture-of-Experts Architecture coreHybrid Mamba-Transformer coreMamba-2 coreSliding Window Attention with an 8,192-Position Window used—LatentMoE coreMixture of experts coreSparse expert activation coreDisabled router-bias updates usedLatent-dimension reduction with increased expert count and active experts used—No Position Encoding core——Frozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used———
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreHybrid architecture core—Grouped dynamic convolution coreHybrid Mamba-Transformer coreMamba checkpointing coreSliding Window Attention with an 8,192-Position Window used—Mixture of experts coreDisabled router-bias updates used———Frozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreShared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used—C-RADIO vision encoder coreVision-language connector coreVideo-frame pruning used——
OpenVLA / OpenVLA-OFT coreSpatially shared FiLM modulation across vision-transformer patch embeddings used—Bidirectional attention core—Feature-wise Linear Modulation (FiLM) optional———Empty action embeddings with distinct positional encodings coreParallel action decoding core—Continuous action representation coreFused SigLIP-DINOv2 vision encoder coreMulti-view image input coreRepresenting robot actions as language-model tokens coreRobot proprioceptive-state projection into language embedding space coreFiLM usedMulti-resolution visual feature fusion usedPatch-as-token vision-language architecture usedShared-sequence multimodal input projection usedStatic third-person camera input usedSingle SigLIP vision encoder evaluated——
Qwen3.5-397B-A17B core—DeepSeek Sparse Attention coreGated attention with reduced KV heads coreGated DeltaNet coreGated DeltaNet and full attention coreGated DeltaNet–sparse Mixture-of-Experts hybrid coreLinear attention coreDynamic Attention Window Size Training usedFull attention with multi-head attention for reliability used—Hybrid Mixture of Experts coreMixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreMoE with 8 routed and 1 shared expert activated per token coreSparse expert activation core—Static YaRN usedRoPE scaling optionalRoPE scaling for context extension (YaRN) optionalYaRN optionalYaRN configuration optionalYaRN scaling-factor selection optionalTemporal-Modality Rotary Position Embedding not used——Multi-Token Prediction usedMulti-Token Prediction module for residual codebooks used—Early-fusion multimodal training coreThinker–Talker architecture core128-channel mel-spectrogram audio front end usedAudio Transformer (AuT) usedCausal streaming ConvNet codec decoder usedExplicit text-string timestamps usedRVQ-based speech representation usedVideo preprocessor longest-edge configuration optional——
Qwen3.6-35B-A3B core—Gated DeltaNet coreGated DeltaNet and full attention coreGated DeltaNet hybrid attention core—Gated DeltaNet MoE coreMixture of experts core—RoPE scaling optionalYaRN optionalYaRN scaling-factor selection optional———Early-fusion multimodal training core——
Qwen3.8-Flash-Next core—Compressed lightweight indexer coreFixed-budget sparse-attention selection coreGated DeltaNet coreGated DeltaNet and full attention coreMQA indexer coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreReuse QSA index selection across speculative decoding steps coreTop-512 block selection coreAverage pooling usedBlock-causal scoring usedBounded sigmoid output gate usedJoint backbone and indexer training under sparse attention usedTop-scoring compressed-block selection used—Mixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreContextual gating for N-gram embedding injection usedFrequency-based partitioning of N-gram embedding slots evaluatedSwiGLU-clip not used—Partial rotary position embeddings usedStatic YaRN usedYaRN optional—All-branch, elementwise read gating in Gated Residual coreDynamic gating of residual reads and writes coreElement-wise data-dependent residual read gate coreFour-branch residual stream coreFour-stream Hyper-Connection Combine update coreGated Residual coreGatedNorm coreIndependent per-branch normalization in Gated Residual coreLow-rank gated Mix for Hyper-Connection residual streams corePer-branch scalar residual write gate coreWidened residual stream coreBounded positive gates usedData-dependent residual read and write operators usedZero-centered RMSNorm usedHyper-Connections evaluatedManifold-Constrained Hyper-Connections evaluatedSimplified AltUp variant evaluatedQK-Clip not usedSparse gated residual writes not used—Multi-Token Prediction core——Host-memory-prefetched N-gram embedding tables coreN-gram embedding lookup memory coreN-gram embedding table for parameter scaling coreSingle-layer N-gram embedding placement coreAsynchronous prefetching of offloaded N-gram embeddings usedHost-memory offload for N-gram embeddings usedMulti-head hashing for N-gram embedding lookup usedToken normalization for N-gram vocabulary compression evaluated—
SmolVLA coreFlow-matching action expert core—Causal self-attention coreCross-attention coreInterleaved Self-Attention and Cross-Attention coreCausal attention usedBidirectional attention evaluatedCross-attention-only action expert evaluated—Action expert core————Intermediate VLM-layer features coreReduced action-expert hidden size coreSmall pretrained VLM backbone coreVision-language model backbone coreVisual token reduction coreGlobal-image-only runtime input defaultIntermediate-layer-limited action-expert conditioning defaultSensorimotor-state projection into VLM token space evaluated——
Step-3.7-Flash core—Cascade attention not used—Mixture of experts core———MTP speculative decoding optionalMulti-Token Prediction optional—Language backbone coreVision encoder coreVision Transformer (ViT) used—256K context window core—
VLA Foundry (Foundry-VLA-1.7B) core—————Flow-matching action head coreDiffusion-policy action head optionalAutoregressive discrete action-tokenization head mentioned—Last-four-layer action-head conditioning corePretrained Qwen3-VL backbone initialization used——
X-VLA coreFlow-based vision-language-action model core——Parallel MLP-Mixer-based action decoder evaluatedHeterogeneity-guided MoE routing not usedLoad-balancing regularization not used———Separate embodiment-specific action projection heads evaluated—Per-hardware-configuration learnable prompt embeddings coreStreamlined encoding pipeline with soft prompts coreCross-attention-based resampler usedDomain-specific resamplers and action heads usedEmbodiment-aware language prompt concatenation usedPretrained vision-language encoder usedShared vision backbone usedAction-modality isolation evaluatedHPT-style domain-specific input projection evaluatedMultimodal Diffusion Transformer decoder evaluatedNatural-language hardware-configuration prompts evaluated——
Xiaomi-Robotics-1 coreMixture-of-Transformers (MoT) core—Causal attention usedLearnable attention sink token usedShape attention mask for action-prefix conditioning used——RoPE positional-index offset used—Adaptive Layer Normalization (AdaLN) used——Masking action-related tokens from DiT attention used——
gpt-oss-120b core—Grouped-query attention usedHybrid Sliding-Window and Global Attention usedOff-by-one attention used—Mixture of experts coreTop-k routing with softmax over selected experts coreSwiGLU used—Rotary Position Embedding usedYaRN used—Pre-LN usedRMSNorm used————
π0.5 coreFlow-based vision-language-action model coreUnified model for high-level and low-level inference coreDownsized action expert usedSemantic action representations using language sub-tasks mentioned—Bidirectional attention coreThree-block causal attention mask corePrefix-conditioned autoregressive action-token attention usedUnidirectional VLM-to-action-expert information flow used—Action expert core——Adaptive RMSNorm used—Flow-matching action head defaultDiffusion decoding for vision-language-action policies evaluatedRegression-head action prediction mentioned—Routing image and language tokens to the VLM backbone coreRouting robotics-specific inputs to an action expert coreSeparate flow-matching action expert coreMasking missing robot image slots usedSeparate robotics-specific token weights used——
↑