Open Models Almanac

Methods · 3 299 in all · 495 from two sources or more

All methods, listed.

3 299 methods from 233 sources, by the area they are filed under; the map browses the same outline. Within each group the strongest treatment comes first.

adopteddiscussednot used

filed at the root 19

token mixer 153

Cross-attention core · 3 sources · 3 quotes
Gated short convolution core · 3 sources · 4 quotes
13 from one source, or only mentioned as not used

softmax attention 48

Bidirectional attention core · 4 sources · 4 quotes
Blockwise causal attention core · 2 sources · 2 quotes
Causal self-attention core · 2 sources · 2 quotes
Causal attention used · 2 sources · 2 quotes
Scaled Dot-Product Attention (SDPA) used · 2 sources · 2 quotes
13 from one source, or only mentioned as not used
Cascade attention not used · 2 sources · 2 quotes

global attention 5

Global attention core · 6 sources · 6 quotes
Unified Keys and Values core · 2 sources · 2 quotes
Full attention used · 2 sources · 2 quotes
2 single mentions

multi-head latent attention 5

Gated Multi-head Latent Attention core · 3 sources · 3 quotes
Multi-head Latent Attention core · 3 sources · 4 quotes
MQA mode of MLA core · 2 sources · 2 quotes
2 from one source, or only mentioned as not used

attention sink 4

Learnable attention sink bias used · 4 sources · 4 quotes
Learnable attention sink token used · 2 sources · 2 quotes
2 single mentions

sparse attention 60

DeepSeek Sparse Attention core · 16 sources · 18 quotes
Sparse attention core · 8 sources · 8 quotes
Compressed Sparse Attention core · 6 sources · 6 quotes
Qwen Sparse Attention core · 6 sources · 6 quotes
Heavily Compressed Attention core · 5 sources · 5 quotes
Gated DeepSeek Sparse Attention core · 3 sources · 3 quotes
Compressed Sparse Attention 2 core · 2 sources · 3 quotes
KV-outer sparse attention used · 2 sources · 2 quotes
show all 9show fewer
15 from one source, or only mentioned as not used

fixed-pattern sparse attention 3

all mentioned once or only as not used

linear attention & state space 18

Linear attention core · 2 sources · 2 quotes
1 only mentioned as not used

Mamba 4

Mamba-2 core · 5 sources · 6 quotes
Mamba core · 2 sources · 2 quotes
2 single mentions

channel mixer 86

3 single mentions

dense feed-forward network 7

SiTU-GLU core · 2 sources · 3 quotes
SwiGLU used · 2 sources · 2 quotes
SwiGLU clamping used · 2 sources · 2 quotes
4 from one source, or only mentioned as not used

mixture of experts 76

Mixture of experts core · 113 sources · 135 quotes
Sparse expert activation core · 9 sources · 9 quotes
Action expert core · 2 sources · 2 quotes
Asymmetric input/output parameter activation core · 2 sources · 2 quotes
DeepSeekMoE core · 2 sources · 2 quotes
12 from one source, or only mentioned as not used

shared experts 6

Shared expert core · 13 sources · 13 quotes
MoE without shared experts core · 3 sources · 3 quotes
2 single mentions

positional encoding 24

normalization & residual 36

prediction head 27

multimodal architecture 154

Encoder-free multimodal architecture core · 6 sources · 6 quotes
Early-fusion multimodal training core · 5 sources · 5 quotes
Native multimodal architecture core · 5 sources · 5 quotes
Discrete audio token encoding core · 4 sources · 4 quotes
Hierarchical patch encoder core · 4 sources · 4 quotes
Mixed-modality training from the start core · 4 sources · 4 quotes
show all 20show fewer
Separate flow-matching action expert core · 3 sources · 3 quotes
C-RADIO vision encoder core · 2 sources · 2 quotes
Multimodal input core · 2 sources · 2 quotes
Native omnimodal architecture core · 2 sources · 2 quotes
Per-layer key-value conditioning core · 2 sources · 5 quotes
Reduced action-expert hidden size core · 2 sources · 2 quotes
Small pretrained VLM backbone core · 2 sources · 2 quotes
Visual token reduction core · 2 sources · 2 quotes
3×3 PixelUnshuffle downsampling used · 2 sources · 2 quotes
Raw audio projection into LLM embedding space used · 2 sources · 2 quotes
Vision Transformer (ViT) used · 2 sources · 2 quotes
Configurable visual token budget optional · 3 sources · 3 quotes
134 single mentions

context capacity 14

filed at the root 11

language modelling objective 15

multi-token prediction objective 7

distillation objective 25

auxiliary loss 26

filed at the root 16

optimizer 29

learning-rate schedule 16

training precision 30

quantization-aware training 17

training stability 23

training parallelism 47

Expert parallelism used · 12 sources · 15 quotes
Tensor parallelism used · 6 sources · 7 quotes
Fully Sharded Data Parallel used · 5 sources · 6 quotes
Context parallelism used · 2 sources · 2 quotes
Pipeline parallelism used · 2 sources · 2 quotes
Tensor and expert parallelism used · 2 sources · 2 quotes
41 single mentions

training runtime 64

Automatic training recovery after failures core · 3 sources · 3 quotes
Checkpointing complete data-loader state core · 2 sources · 2 quotes
Checkpoint merging used · 4 sources · 5 quotes
Asynchronous checkpointing used · 3 sources · 3 quotes
59 single mentions

filed at the root 11

data sourcing 37

data filtering 145

Benchmark decontamination core · 4 sources · 8 quotes
Heuristic filtering used · 4 sources · 4 quotes
CSAM filtering used · 3 sources · 3 quotes
Sensitive data filtering used · 3 sources · 3 quotes
CBRN safety filtering of pre-training data used · 2 sources · 2 quotes
Keyword- and regex-based content filtering used · 2 sources · 2 quotes
show all 16show fewer
LLM-based refusal and inability filtering used · 2 sources · 2 quotes
Manual camera-viewpoint normalization used · 2 sources · 2 quotes
N-gram repetition filtering used · 2 sources · 2 quotes
Pass@100 filtering of synthesized RL tasks used · 2 sources · 2 quotes
Permissive-license filtering used · 2 sources · 2 quotes
Personally identifiable information redaction used · 2 sources · 2 quotes
PII-redaction-threshold document filtering used · 2 sources · 2 quotes
Unified data filtering pipeline used · 2 sources · 2 quotes
129 single mentions

deduplication 15

synthetic data 138

Large-scale agentic task synthesis pipeline core · 3 sources · 3 quotes
Synthetic data generation used · 3 sources · 3 quotes
show all 10show fewer
Counterfactual data augmentation mentioned · 2 sources · 2 quotes
128 single mentions

data mixture & curriculum 117

Heterogeneous training mixture core · 2 sources · 2 quotes
Heterogeneous-data co-training core · 2 sources · 4 quotes
Curated higher-quality mid-training data used · 3 sources · 4 quotes
Cross-embodiment robot training used · 2 sources · 2 quotes
KL-regularized data-mixture optimization used · 2 sources · 3 quotes
MergeMix used · 2 sources · 4 quotes
MergeMix expert-checkpoint merging used · 2 sources · 3 quotes
Vulnerability-discovery training data used · 2 sources · 2 quotes
109 from one source, or only mentioned as not used

sequence packing 27

tokenization 42

Quantile normalization of robot actions core · 2 sources · 2 quotes
UniBPE used · 3 sources · 4 quotes
Quick Instruction tokens used · 2 sources · 2 quotes
38 single mentions

filed at the root 28

supervised fine-tuning 110

Supervised fine-tuning core · 23 sources · 27 quotes
Low-Rank Adaptation core · 11 sources · 12 quotes
Two-stage supervised fine-tuning core · 4 sources · 5 quotes
Full-model fine-tuning core · 3 sources · 3 quotes
Embodiment-specific fine-tuning core · 2 sources · 3 quotes
German-language reasoning post-training core · 2 sources · 2 quotes
Fine-tuning used · 6 sources · 6 quotes
Assistant-only loss used · 2 sources · 2 quotes
show all 17show fewer
End-to-end fine-tuning used · 2 sources · 2 quotes
Imitation learning used · 2 sources · 2 quotes
Multitask fine-tuning used · 2 sources · 2 quotes
Parameter-efficient fine-tuning used · 2 sources · 2 quotes
Rejection sampling used · 2 sources · 2 quotes
Self-correction cold start used · 2 sources · 2 quotes
SFT bootstrapping with synthetic data used · 2 sources · 2 quotes
Single-GPU fine-tuning optional · 2 sources · 2 quotes
93 from one source, or only mentioned as not used

reinforcement learning algorithm 75

Group Relative Policy Optimization core · 16 sources · 18 quotes
Mixed-domain reinforcement learning core · 5 sources · 5 quotes
Groupwise Advantage Redistribution core · 3 sources · 4 quotes
Chain-of-thought reinforcement learning core · 2 sources · 2 quotes
Group-relative advantage estimation core · 2 sources · 2 quotes
Dropping zero-advantage rollout groups default · 2 sources · 2 quotes
Reinforcement learning used · 20 sources · 23 quotes
show all 16show fewer
Abstention training used · 2 sources · 2 quotes
IcePop used · 2 sources · 3 quotes
Keep Sampling Mask used · 2 sources · 2 quotes
Off-Policy Sequence Masking used · 2 sources · 3 quotes
Rollout Routing Replay used · 2 sources · 3 quotes
59 single mentions

reward modelling 68

Groupwise Agentic Grading core · 3 sources · 4 quotes
Groupwise Reward Synthesis core · 3 sources · 4 quotes
Generative Reward Model used · 6 sources · 7 quotes
Adversarial-agent reward-hacking audit used · 4 sources · 4 quotes
Length penalty used · 4 sources · 4 quotes
Language-consistency reward used · 3 sources · 3 quotes
Reasoning-length penalty used · 2 sources · 2 quotes
show all 9show fewer
Verifier cross-checking used · 2 sources · 2 quotes
59 single mentions

preference optimization 8

policy distillation 32

rollout & RL infrastructure 98

Asynchronous reinforcement learning core · 14 sources · 18 quotes
Partial rollout used · 3 sources · 5 quotes
Adaptive Rollout Concurrency used · 2 sources · 2 quotes
Co-located rollout and training used · 2 sources · 2 quotes
On-policy rollout generation used · 2 sources · 2 quotes
Seamless Rollout Engine used · 2 sources · 2 quotes
show all 10show fewer
SLIME used · 2 sources · 3 quotes
Token-in-Token-out (TITO) used · 2 sources · 3 quotes
88 from one source, or only mentioned as not used

agentic post-training 69

Agentic reinforcement learning core · 8 sources · 8 quotes
Merlin–Arthur protocol core · 2 sources · 4 quotes
Multi-environment reinforcement learning core · 2 sources · 2 quotes
Agentic post-training used · 2 sources · 2 quotes
Agentic tool-use post-training used · 2 sources · 2 quotes
62 single mentions

mid-training & continual pretraining 18

filed at the root 9

decoding strategy 52

Multi-Token Prediction core · 19 sources · 19 quotes
Speculative decoding core · 15 sources · 15 quotes
DSpark core · 14 sources · 14 quotes
DFlash core · 6 sources · 7 quotes
Five-layer SWA MTP drafter core · 2 sources · 2 quotes
Three-token speculative draft depth core · 2 sources · 2 quotes
Greedy decoding default · 4 sources · 4 quotes
Standardized sampling configuration default · 3 sources · 3 quotes
show all 20show fewer
EAGLE used · 10 sources · 10 quotes
Multi-layer EAGLE used · 5 sources · 5 quotes
NEXTN speculative decoding used · 4 sources · 4 quotes
Best-of-N selection used · 3 sources · 4 quotes
Action-chunk execution used · 2 sources · 2 quotes
Multi-stage candidate filtering used · 2 sources · 2 quotes
Presence penalty tuning used · 2 sources · 3 quotes
Standalone parallel speculative drafters used · 2 sources · 3 quotes
Speculative sampling optional · 3 sources · 3 quotes
Task-specific sampling-parameter presets optional · 3 sources · 3 quotes
LiLiCorr optional · 2 sources · 2 quotes
32 from one source, or only mentioned as not used

reasoning control 49

Thinking mode core · 49 sources · 55 quotes
Continuously controllable reasoning effort core · 3 sources · 6 quotes
Interleaved thinking core · 3 sources · 3 quotes
Adaptive reasoning core · 2 sources · 2 quotes
Always-on thinking mode core · 2 sources · 2 quotes
Full chain-of-thought output core · 2 sources · 2 quotes
Chain-of-thought reasoning default · 6 sources · 6 quotes
Reasoning enabled by default default · 4 sources · 4 quotes
show all 18show fewer
clear_thinking chat-template flag default · 2 sources · 2 quotes
Max thinking mode with self-set compute budget default · 2 sources · 3 quotes
System-prompt reasoning-effort conditioning default · 2 sources · 2 quotes
Reasoning token-budget control used · 7 sources · 8 quotes
Persistent reasoning history used · 3 sources · 3 quotes
Generate-verify-refine used · 2 sources · 2 quotes
Reasoning parser used · 2 sources · 2 quotes
Disable reasoning via enable_thinking optional · 5 sources · 5 quotes
Parallel test-time compute scaling evaluated · 2 sources · 2 quotes
Parallel-fewest-step sampling evaluated · 2 sources · 2 quotes
31 from one source, or only mentioned as not used
Qwen3 soft thinking switch not used · 2 sources · 2 quotes

KV cache management 53

Prefix caching core · 10 sources · 10 quotes
Compressed KV caching core · 3 sources · 3 quotes
Cross-layer KV-cache sharing core · 2 sources · 2 quotes
Hybrid-state caching core · 2 sources · 2 quotes
SWA Bounded Replay core · 2 sources · 2 quotes
Chunked prefill default · 6 sources · 6 quotes
Prompt caching used · 4 sources · 4 quotes
Language-model-only serving mode used · 3 sources · 3 quotes
show all 13show fewer
Internal prefill checkpointing used · 2 sources · 2 quotes
RadixCache used · 2 sources · 2 quotes
Periodic cache checkpointing optional · 3 sources · 3 quotes
HiCache mentioned · 2 sources · 2 quotes
40 single mentions

inference quantization 112

Quantization core · 4 sources · 4 quotes
FP4 global KV-cache quantization core · 2 sources · 2 quotes
Mobile quantization core · 2 sources · 2 quotes
Per-tensor FP8 quantization core · 2 sources · 2 quotes
FP8 default · 18 sources · 18 quotes
BF16 inference default · 2 sources · 2 quotes
Mixed-precision NVFP4 routed-expert quantization default · 2 sources · 2 quotes
FP8 KV-cache quantization used · 12 sources · 13 quotes
show all 20show fewer
Post-training quantization used · 5 sources · 5 quotes
NVFP4 quantization used · 4 sources · 4 quotes
MSE-based weight-scale selection used · 3 sources · 4 quotes
MXFP4 weight quantization used · 3 sources · 3 quotes
Four-Over-Six used · 2 sources · 3 quotes
GGUF used · 2 sources · 2 quotes
Mamba SSM-cache quantization used · 2 sources · 2 quotes
ModelOpt NVFP4 re-quantization used · 2 sources · 2 quotes
MXFP8 activation quantization used · 2 sources · 2 quotes
FP8 E4M3 quantization evaluated · 2 sources · 2 quotes
Max-based weight scaling evaluated · 2 sources · 2 quotes
92 from one source, or only mentioned as not used

serving parallelism 20

inference scheduling 35

inference kernel 59

Batch-invariant deterministic kernels core · 2 sources · 3 quotes
CUDA Graph capture and replay used · 4 sources · 6 quotes
CUDA graph capture size reduction used · 2 sources · 2 quotes
FlashAttention used · 2 sources · 2 quotes
FlashAttention-2 used · 2 sources · 2 quotes
FlashAttention-3 used · 2 sources · 2 quotes
Kernel fusion used · 2 sources · 2 quotes
MoE-side chunking used · 2 sources · 2 quotes
show all 9show fewer
TensorRT-LLM multi-head attention backend used · 2 sources · 2 quotes
50 from one source, or only mentioned as not used

context management 24

agentic scaffolding 108

Function calling core · 3 sources · 3 quotes
Harmony format core · 3 sources · 3 quotes
Role-based instruction hierarchy core · 2 sources · 2 quotes
Automatic tool choice used · 4 sources · 4 quotes
Browser tool used · 3 sources · 4 quotes
Python tool for visual tasks used · 3 sources · 3 quotes
Robot control through tool orchestration used · 2 sources · 2 quotes
show all 15show fewer
Thinking with tools used · 2 sources · 3 quotes
Tool-call parser used · 2 sources · 2 quotes
Visual Search Tool used · 2 sources · 2 quotes
XML-based tool-call schema with DSML token used · 2 sources · 2 quotes
Advisor strategy optional · 2 sources · 4 quotes
MCP configuration for tool definition optional · 2 sources · 2 quotes
Tool calling with the qwen3_coder parser optional · 2 sources · 2 quotes
93 single mentions

filed at the root 15

inference engine 19

training framework 13

kernel & quantization library 25

agent product 11

OpenCode used · 4 sources · 4 quotes
Claude Code used · 3 sources · 3 quotes
OpenHands used · 2 sources · 2 quotes
Qwen-Agent used · 2 sources · 2 quotes
7 single mentions

infrastructure service 33

filed at the root 69

Prompt-based output standardization used · 5 sources · 5 quotes
Evaluation without safety filters used · 3 sources · 3 quotes
Refusal-suppressed capability evaluation used · 3 sources · 3 quotes
Chain-of-thought few-shot prompting used · 2 sources · 3 quotes
Controlled evaluation with fixed settings used · 2 sources · 2 quotes
Open-loop evaluation used · 2 sources · 2 quotes
63 single mentions

cybersecurity - vulnerability discovery; exploit development 4

all mentioned once

benchmark 33

all mentioned once

evaluation harness 21

judge 30

averaging - mean@k; avg@k; pass@k; pass^k 5

all mentioned once

human & real-world evaluation 33

filed at the root 60

Hardware-in-the-loop architecture search core · 3 sources · 4 quotes
Presence penalty adjustment optional · 2 sources · 2 quotes
58 from one source, or only mentioned as not used
530 methods not yet filed

unfiled 530

Flow matching core · 8 sources · 10 quotes
Action chunking core · 4 sources · 5 quotes
Gated Attention core · 3 sources · 3 quotes
Tri-hybrid model architecture core · 2 sources · 2 quotes
World-action modeling core · 2 sources · 2 quotes
OpenAI-compatible API default · 3 sources · 3 quotes
show all 11show fewer
Terminus 2 used · 2 sources · 2 quotes
cross-provider failover routing optional · 2 sources · 2 quotes
519 from one source, or only mentioned as not used