Open Models Almanac

46 open-weight models · 233 sources · every claim quoted

What’s actually inside today’s open‑weight models.

We read the technical reports, model cards and repos behind the most-used open models and pulled out every method they describe — each one linked to the exact sentence that says so.

The map

Where every method lives

One block, one pipeline; each box opens its part of the map.

data curation: Benchmark decontamination (3/24 labs); Synthetic data generation (3/24 labs); Corpus deduplication (2/24 labs) Data curation 532 methods Benchmarkdecontamination3/24 labsSynthetic datageneration3/24 labsCorpus deduplication2/24 labs +3D2, D3, D4D1 Filtering and deduplication: from a crawl to a corpusD101 The anatomy of a training recipe01 training objective: Multi-Token Prediction (3/24 labs); Knowledge insulation (3/24 labs); Quantization-aware distillation (2/24 labs) Training objective 84 methods Multi-TokenPrediction3/24 labsKnowledge insulation3/24 labsQuantization-awaredistillation2/24 labs +3P1, S1, S6C3 Expert load balancingC301 The anatomy of a training recipe01 model architecture: Mixture of experts (17/24 labs); Multi-Token Prediction (7/24 labs); Grouped-query attention (7/24 labs) Model architecture 513 methods Mixture of experts17/24 labsMulti-TokenPrediction7/24 labsGrouped-queryattention · 7/24 labs +16C1, C2, C3, D4, M1, P1, R1, R2, T1, T2, T3, T4, T5, T6, X1, X200 The early modern LLM and what it costs00 optimization: Muon (7/24 labs); Expert parallelism (6/24 labs); Quantization-Aware Training (5/24 labs) Optimization 242 methods Muon · 7/24 labsExpert parallelism6/24 labsQuantization-AwareTraining · 5/24 labs +2O2, S1O1 Optimizers: AdamW, Muon, and what each one boundsO101 The anatomy of a training recipe01 post-training: Supervised fine-tuning (8/24 labs); Asynchronous reinforcement learning (7/24 labs); Multi-Teacher On-Policy Distillation (6/24 labs) Post-training 506 methods Supervised fine-tuning · 8/24 labsAsynchronousreinforcementlearning · 7/24 labsMulti-Teacher On-Policy Distillation6/24 labs +5S2, S3, S4, S5, S6D4 Multimodal training data: when pixels and sound enter the recipeD401 The anatomy of a training recipe01 inference & serving: Thinking mode (14/24 labs); Multi-Token Prediction (8/24 labs); FP8 (7/24 labs) Inference & serving 521 methods Thinking mode14/24 labsMulti-TokenPrediction8/24 labsFP8 · 7/24 labs +4I2, I3, I4, I5I1 Quantization for serving: which bits to remove, and what each removal savesI102 Serving a large reasoning model at low cost02 One decoder block, opened up repeated for every layer positional encoding: No Position Encoding (3/24 labs); YaRN (2/24 labs); Partial rotary position embeddings (2/24 labs) Positional encoding 24 methods No Position Encoding · 3/24 labs T4 Position encoding and length extensionT4 normalization & residual: no method here is adopted by two labs and checkable Normalization & residual 36 methods no shared pick R2 Rethinking the residual streamR2R1 Normalization and logit controlR1 token mixer: Grouped-query attention (7/24 labs); Hybrid Sliding-Window and Global Attention (6/24 labs); DeepSeek Sparse Attention (4/24 labs) Token mixer 153 methods Grouped-query attention · 7/24 labs +1T6T5 Hybrid layer stacking and cross-layer sharingT5T3 Sparse attention and indexingT3T2 Fixed-state token mixersT2T1 Full attention on a memory budgetT1 softmax attention: Grouped-query attention (7/24 labs); Sliding Window Attention (5/24 labs); Multi-head Latent Attention (2/24 labs) Softmax attention Grouped-queryattention · 7/24 labs 48 methods +1T6T2 Fixed-state token mixersT2T1 Full attention on a memory budgetT1 sparse attention: DeepSeek Sparse Attention (4/24 labs); Lightning Indexer (2/24 labs) Sparse attention DeepSeek SparseAttention · 4/24 labs 60 methods T3 Sparse attention and indexingT3 linear attention & state space: Gated DeltaNet (2/24 labs) Linear attention & state space Gated DeltaNet2/24 labs 18 methods T2 Fixed-state token mixersT2 hybrid layer stacking: Hybrid Sliding-Window and Global Attention (6/24 labs) Hybrid layer stacking Hybrid Sliding-Windowand Global Attention6/24 labs 12 methods T5 Hybrid layer stacking and cross-layer sharingT5 channel mixer: Mixture of experts (17/24 labs); Shared expert (8/24 labs); SwiGLU (2/24 labs) Channel mixer 86 methods Mixture of experts · 17/24 labs C3 Expert load balancingC3C2 The mixture-of-experts design spaceC2C1 Gated feed-forward units: SwiGLU and its bounded descendantsC1 dense feed-forward network: SwiGLU (2/24 labs) Dense feed-forward network SwiGLU · 2/24 labs 7 methods C1 Gated feed-forward units: SwiGLU and its bounded descendantsC1 mixture of experts: Mixture of experts (17/24 labs); Shared expert (8/24 labs); Auxiliary-loss-free load balancing (3/24 labs) Mixture of experts Mixture of experts · 17/24 labs 76 methods C3 Expert load balancingC3C2 The mixture-of-experts design spaceC2 prediction head: Multi-Token Prediction (7/24 labs) Prediction head 27 methods Multi-Token Prediction · 7/24 labs P1 Multi-token prediction: a training signal and a built-in drafterP1 multimodal architecture: no method here is adopted by two labs and checkable Multimodal architecture no shared pick 154 methods +2X1, X2D4 Multimodal training data: when pixels and sound enter the recipeD4 context capacity: no method here is adopted by two labs and checkable Context capacity no shared pick 14 methods M1 Conditional memory: lookup tables beside the expertsM1 Each box names the method the most labs adopt there. an article explains it an article is planned Architecture lists only methods the models’ source code can be checked for.

Who uses what

The field at a glance

17 of 24 labs ship Mixture of experts; the labs part ways on inference & serving and optimization.

adopteddiscussednot usedblank: documents silent✓in its code✕not in its code
MethodDeepSeek-V4.1-FlashKimi K3Kolibri-1LFM3-1.7BLaguna-S-2.1LingBot-VLA 2.0MiMo-V2.6-ProMiniMax-M3Mistral-Large-4MolmoAct 2NVIDIA-Nemotron-3.5-Lightning-30B-A3BQwen3.8-Flash-Nextgpt-oss-120bπ0.5
model architecture
Grouped-query attentionDeepSeek-V4.1-Flash: not in its documents; not in its codeKimi K3: not in its documents; not in its codeKolibri-1: not in its documents; in its codeLFM3-1.7B: core; in its codeLaguna-S-2.1: core; in its codeLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documents; in its codeMiniMax-M3: core; in its codeMistral-Large-4: not in its documents; not in its codeMolmoAct 2: not in its documents; in its codeNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; in its codeQwen3.8-Flash-Next: not in its documents; in its codegpt-oss-120b: used; in its codeπ0.5: not in its documents
Hybrid Sliding-Window and Global AttentionDeepSeek-V4.1-Flash: not in its documents; in its codeKimi K3: not in its documents; not in its codeKolibri-1: core; in its codeLFM3-1.7B: not in its documents; in its codeLaguna-S-2.1: core; in its codeLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: core; in its codeMiniMax-M3: not in its documents; not in its codeMistral-Large-4: not in its documents; not in its codeMolmoAct 2: not in its documents; not in its codeNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; not in its codeQwen3.8-Flash-Next: not in its documents; not in its codegpt-oss-120b: used; in its codeπ0.5: not in its documents
Shared expertDeepSeek-V4.1-Flash: not in its documents; in its codeKimi K3: not in its documents; in its codeKolibri-1: core; in its codeLFM3-1.7B: not in its documents; not in its codeLaguna-S-2.1: used; in its codeLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documents; not in its codeMiniMax-M3: used; in its codeMistral-Large-4: core; in its codeMolmoAct 2: not in its documents; not in its codeNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documents; in its codeQwen3.8-Flash-Next: not in its documents; in its codegpt-oss-120b: not in its documents; not in its codeπ0.5: not in its documents
Linear attention & state spaceany method under itDeepSeek-V4.1-Flash: not in its documents; not in its codeKimi K3: core; in its codeKolibri-1: not in its documents; not in its codeLFM3-1.7B: not in its documents; not in its codeLaguna-S-2.1: not in its documents; not in its codeLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documents; not in its codeMiniMax-M3: mentioned; not in its codeMistral-Large-4: not in its documents; not in its codeMolmoAct 2: not in its documents; not in its codeNVIDIA-Nemotron-3.5-Lightning-30B-A3B: core; in its codeQwen3.8-Flash-Next: core; in its codegpt-oss-120b: not in its documents; not in its codeπ0.5: not in its documents
optimization
MuonDeepSeek-V4.1-Flash: not in its documentsKimi K3: usedKolibri-1: usedLFM3-1.7B: not in its documentsLaguna-S-2.1: coreLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: not in its documentsMistral-Large-4: usedMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documentsQwen3.8-Flash-Next: coregpt-oss-120b: not in its documentsπ0.5: not in its documents
Global-norm gradient clippingDeepSeek-V4.1-Flash: not in its documentsKimi K3: not in its documentsKolibri-1: usedLFM3-1.7B: not in its documentsLaguna-S-2.1: not in its documentsLingBot-VLA 2.0: usedMiMo-V2.6-Pro: not in its documentsMiniMax-M3: not in its documentsMistral-Large-4: not in its documentsMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: usedQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: not in its documentsπ0.5: used
post-training
Asynchronous reinforcement learningDeepSeek-V4.1-Flash: coreKimi K3: not in its documentsKolibri-1: coreLFM3-1.7B: not in its documentsLaguna-S-2.1: not in its documentsLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: coreMiniMax-M3: not in its documentsMistral-Large-4: coreMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: usedQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: not in its documentsπ0.5: not in its documents
Supervised fine-tuningDeepSeek-V4.1-Flash: not in its documentsKimi K3: usedKolibri-1: usedLFM3-1.7B: coreLaguna-S-2.1: not in its documentsLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: not in its documentsMistral-Large-4: usedMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: usedQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: not in its documentsπ0.5: not in its documents
Long-context extension via continued pretrainingDeepSeek-V4.1-Flash: not in its documentsKimi K3: not in its documentsKolibri-1: usedLFM3-1.7B: usedLaguna-S-2.1: usedLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: not in its documentsMistral-Large-4: not in its documentsMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: coreQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: not in its documentsπ0.5: not in its documents
Multi-Teacher On-Policy DistillationDeepSeek-V4.1-Flash: not in its documentsKimi K3: coreKolibri-1: not in its documentsLFM3-1.7B: usedLaguna-S-2.1: not in its documentsLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: not in its documentsMistral-Large-4: usedMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: usedQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: not in its documentsπ0.5: not in its documents
inference & serving
Thinking modeDeepSeek-V4.1-Flash: not in its documentsKimi K3: defaultKolibri-1: coreLFM3-1.7B: not in its documentsLaguna-S-2.1: defaultLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: optionalMistral-Large-4: defaultMolmoAct 2: not in its documentsNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documentsQwen3.8-Flash-Next: not in its documentsgpt-oss-120b: usedπ0.5: not in its documents
CUDA Graph capture and replayDeepSeek-V4.1-Flash: not in its documentsKimi K3: not in its documentsKolibri-1: not in its documentsLFM3-1.7B: not in its documentsLaguna-S-2.1: not in its documentsLingBot-VLA 2.0: not in its documentsMiMo-V2.6-Pro: not in its documentsMiniMax-M3: usedMistral-Large-4: not in its documentsMolmoAct 2: usedNVIDIA-Nemotron-3.5-Lightning-30B-A3B: not in its documentsQwen3.8-Flash-Next: usedgpt-oss-120b: usedπ0.5: not in its documents

One column per lab, its most thoroughly documented model. Rows are where the labs differ most; each needs three labs whose documents and code agree, or four whose documents state it.

Every method by model →

This run

What changed since 2026-11-12

Built 2026-11-12, ranking week 2026-10-26.

The models

Newest models

ModelLabReleasedAttentionParams / active
Mistral-Large-4Mistral AI2026-10-29sparse (indexer)~1T / ~52B
LFM3-1.7BLiquid AI2026-10-27conv hybrid1.73B / 1.73B
MiniMax-M3.1-FlashMiniMax2026-10-19sparse (indexer)~428B / ~23B
DeepSeek-V4.1-Flash-1017DeepSeek2026-10-17sparse (indexer)552B / 16B
NVIDIA-Nemotron-3.5-Super-VL-120B-A12BNVIDIA2026-10-15Mamba hybrid120B / 12B

All 46 models →

Reading paths

Three ways through