46 open-weight models · 233 sources · every claim quoted
What’s actually inside today’s open‑weight models.
We read the technical reports, model cards and repos behind the most-used open models and pulled out every method they describe — each one linked to the exact sentence that says so.
The map
Where every method lives
One block, one pipeline; each box opens its part of the map.
- data curation 532Benchmark decontamination · 3/24 labsSynthetic data generation · 3/24 labsCorpus deduplication · 2/24 labs01D1D2D3D4
- training objective 84Multi-Token Prediction · 3/24 labsKnowledge insulation · 3/24 labsQuantization-aware distillation · 2/24 labs01C3P1S1S6
- model architecture 513Mixture of experts · 17/24 labsMulti-Token Prediction · 7/24 labsGrouped-query attention · 7/24 labs00C1C2C3D4M1P1R1R2T1T2T3T4T5T6X1X2
- positional encoding 24No Position Encoding · 3/24 labsT4
- normalization & residual 36R1R2
- token mixer 153Grouped-query attention · 7/24 labsT1T2T3T5T6
- softmax attention 48Grouped-query attention · 7/24 labsT1T2T6
- sparse attention 60DeepSeek Sparse Attention · 4/24 labsT3
- linear attention & state space 18Gated DeltaNet · 2/24 labsT2
- hybrid layer stacking 12Hybrid Sliding-Window and Global Attention · 6/24 labsT5
- channel mixer 86Mixture of experts · 17/24 labsC1C2C3
- dense feed-forward network 7SwiGLU · 2/24 labsC1
- mixture of experts 76Mixture of experts · 17/24 labsC2C3
- prediction head 27Multi-Token Prediction · 7/24 labsP1
- context capacity 14M1
- multimodal architecture 154D4X1X2
- optimization 242Muon · 7/24 labsExpert parallelism · 6/24 labsQuantization-Aware Training · 5/24 labs01O1O2S1
- post-training 506Supervised fine-tuning · 8/24 labsAsynchronous reinforcement learning · 7/24 labsMulti-Teacher On-Policy Distillation · 6/24 labs01D4S2S3S4S5S6
- inference & serving 521Thinking mode · 14/24 labsMulti-Token Prediction · 8/24 labsFP8 · 7/24 labs02I1I2I3I4I5
Who uses what
The field at a glance
17 of 24 labs ship Mixture of experts; the labs part ways on inference & serving and optimization.
adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
| Method | DeepSeek-V4.1-Flash | Kimi K3 | Kolibri-1 | LFM3-1.7B | Laguna-S-2.1 | LingBot-VLA 2.0 | MiMo-V2.6-Pro | MiniMax-M3 | Mistral-Large-4 | MolmoAct 2 | NVIDIA-Nemotron-3.5-Lightning-30B-A3B | Qwen3.8-Flash-Next | gpt-oss-120b | π0.5 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| model architecture | ||||||||||||||
| Grouped-query attention | DeepSeek-V4.1-Flash: not in its documents; not in its code | Kimi K3: not in its documents; not in its code | Kolibri-1: not in its documents; in its code | LFM3-1.7B: core; in its code | Laguna-S-2.1: core; in its code | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents; in its code | MiniMax-M3: core; in its code | Mistral-Large-4: not in its documents; not in its code | MolmoAct 2: not in its documents; in its code | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; in its code | Qwen3.8-Flash-Next: not in its documents; in its code | gpt-oss-120b: used; in its code | π0.5: not in its documents |
| Hybrid Sliding-Window and Global Attention | DeepSeek-V4.1-Flash: not in its documents; in its code | Kimi K3: not in its documents; not in its code | Kolibri-1: core; in its code | LFM3-1.7B: not in its documents; in its code | Laguna-S-2.1: core; in its code | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: core; in its code | MiniMax-M3: not in its documents; not in its code | Mistral-Large-4: not in its documents; not in its code | MolmoAct 2: not in its documents; not in its code | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; not in its code | Qwen3.8-Flash-Next: not in its documents; not in its code | gpt-oss-120b: used; in its code | π0.5: not in its documents |
| Shared expert | DeepSeek-V4.1-Flash: not in its documents; in its code | Kimi K3: not in its documents; in its code | Kolibri-1: core; in its code | LFM3-1.7B: not in its documents; not in its code | Laguna-S-2.1: used; in its code | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents; not in its code | MiniMax-M3: used; in its code | Mistral-Large-4: core; in its code | MolmoAct 2: not in its documents; not in its code | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; in its code | Qwen3.8-Flash-Next: not in its documents; in its code | gpt-oss-120b: not in its documents; not in its code | π0.5: not in its documents |
| Linear attention & state spaceany method under it | DeepSeek-V4.1-Flash: not in its documents; not in its code | Kimi K3: core; in its code | Kolibri-1: not in its documents; not in its code | LFM3-1.7B: not in its documents; not in its code | Laguna-S-2.1: not in its documents; not in its code | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents; not in its code | MiniMax-M3: mentioned; not in its code | Mistral-Large-4: not in its documents; not in its code | MolmoAct 2: not in its documents; not in its code | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: core; in its code | Qwen3.8-Flash-Next: core; in its code | gpt-oss-120b: not in its documents; not in its code | π0.5: not in its documents |
| optimization | ||||||||||||||
| Muon | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: used | Kolibri-1: used | LFM3-1.7B: not in its documents | Laguna-S-2.1: core | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: not in its documents | Mistral-Large-4: used | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents | Qwen3.8-Flash-Next: core | gpt-oss-120b: not in its documents | π0.5: not in its documents |
| Global-norm gradient clipping | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: not in its documents | Kolibri-1: used | LFM3-1.7B: not in its documents | Laguna-S-2.1: not in its documents | LingBot-VLA 2.0: used | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: not in its documents | Mistral-Large-4: not in its documents | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: used | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: not in its documents | π0.5: used |
| post-training | ||||||||||||||
| Asynchronous reinforcement learning | DeepSeek-V4.1-Flash: core | Kimi K3: not in its documents | Kolibri-1: core | LFM3-1.7B: not in its documents | Laguna-S-2.1: not in its documents | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: core | MiniMax-M3: not in its documents | Mistral-Large-4: core | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: used | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: not in its documents | π0.5: not in its documents |
| Supervised fine-tuning | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: used | Kolibri-1: used | LFM3-1.7B: core | Laguna-S-2.1: not in its documents | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: not in its documents | Mistral-Large-4: used | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: used | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: not in its documents | π0.5: not in its documents |
| Long-context extension via continued pretraining | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: not in its documents | Kolibri-1: used | LFM3-1.7B: used | Laguna-S-2.1: used | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: not in its documents | Mistral-Large-4: not in its documents | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: core | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: not in its documents | π0.5: not in its documents |
| Multi-Teacher On-Policy Distillation | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: core | Kolibri-1: not in its documents | LFM3-1.7B: used | Laguna-S-2.1: not in its documents | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: not in its documents | Mistral-Large-4: used | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: used | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: not in its documents | π0.5: not in its documents |
| inference & serving | ||||||||||||||
| Thinking mode | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: default | Kolibri-1: core | LFM3-1.7B: not in its documents | Laguna-S-2.1: default | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: optional | Mistral-Large-4: default | MolmoAct 2: not in its documents | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents | Qwen3.8-Flash-Next: not in its documents | gpt-oss-120b: used | π0.5: not in its documents |
| CUDA Graph capture and replay | DeepSeek-V4.1-Flash: not in its documents | Kimi K3: not in its documents | Kolibri-1: not in its documents | LFM3-1.7B: not in its documents | Laguna-S-2.1: not in its documents | LingBot-VLA 2.0: not in its documents | MiMo-V2.6-Pro: not in its documents | MiniMax-M3: used | Mistral-Large-4: not in its documents | MolmoAct 2: used | NVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents | Qwen3.8-Flash-Next: used | gpt-oss-120b: used | π0.5: not in its documents |
One column per lab, its most thoroughly documented model. Rows are where the labs differ most; each needs three labs whose documents and code agree, or four whose documents state it.
Every method by model →This run
What changed since 2026-11-12
Built 2026-11-12, ranking week 2026-10-26.
The models
Newest models
| Model | Lab | Released | Attention | Params / active |
|---|---|---|---|---|
| Mistral-Large-4 | Mistral AI | 2026-10-29 | sparse (indexer) | ~1T / ~52B |
| LFM3-1.7B | Liquid AI | 2026-10-27 | conv hybrid | 1.73B / 1.73B |
| MiniMax-M3.1-Flash | MiniMax | 2026-10-19 | sparse (indexer) | ~428B / ~23B |
| DeepSeek-V4.1-Flash-1017 | DeepSeek | 2026-10-17 | sparse (indexer) | 552B / 16B |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | NVIDIA | 2026-10-15 | Mamba hybrid | 120B / 12B |
Reading paths