Open Models Almanac

Learn

Articles

Explainers for groups of methods that compete or combine: the problem they solve, the trade-off to remember, and where the models in the map sit. Unlike the method pages, these are written, not extracted.

3 of 3 published

Foundations

The cost model of a decoder layer and the stages of a training recipe; read these first.

00 · 35 min

The early modern LLM and what it costs

LLaMA 1 as the baseline: prefill pays in FLOPs, decode pays in bytes, and nearly every later design moves one of the two

01 · 35 min

The anatomy of a training recipe

Labs agree on the rough order of stages and a few habits inside them. Each stage spends tokens on what the last one could not teach, and each can lose some of what came before

02 · 35 min

Serving a large reasoning model at low cost

A token gets cheaper as more requests share each forward pass, until the latency promise or the KV-cache budget stops the batch from growing

6 of 6 published

Token mixer

How a layer reads the rest of the sequence, and what that costs in compute and KV-cache memory.

T1 · 25 min

Full attention on a memory budget

MHA, MQA, GQA and MLA trade capacity or compute for a smaller KV cache

T2 · 30 min

Fixed-state token mixers

Sliding windows, linear attention, Mamba-2, Gated DeltaNet and KDA trade exact recall for constant decode memory

T3 · 32 min

Sparse attention and indexing

Selecting a few past entries per query cuts attention work, but the selector still reads every entry, misses some of what dense attention would use, and leaves the KV cache full unless entries are compressed

T4 · 32 min

Position encoding and length extension

Rotary position gives attention order and locality, but its slow pairs go out of range past the trained length. Rescaling and training recover length; layers without position work beside layers that carry it

T5 · 32 min

Hybrid layer stacking and cross-layer sharing

Most layers can be cheap if a few global layers handle long-range retrieval: the growing cache shrinks with the ratio of cheap to global layers, and sharing caches across layers shrinks it further, with quality losses one lab reported

T6 · 32 min

Attention sinks and output gates

A softmax head's attention always adds up to one: a learned sink logit lets it add up to less for one number per head, an output gate scales the head's result for a small projection, and the few direct comparisons are small and mixed

3 of 3 published

Channel mixer

What each token does on its own: gated feed-forward units and mixtures of experts.

C1 · 26 min

Gated feed-forward units: SwiGLU and its bounded descendants

Multiplying two projections buys quality per parameter and lets a few outputs grow with the square of the weights behind them; a bound gives each unit a known ceiling and pays for it in shape and gradient

C2 · 32 min

The mixture-of-experts design space

At fixed k, more experts raise quality but cost memory, large-batch reads and balance; finer experts cost traffic; LatentMoE narrows the vector to pay for both, and confines the routed path

C3 · 30 min

Expert load balancing

Balance over the whole batch lets experts specialize and leaves GPUs waiting; balance inside each micro-batch is faster and costs quality; a selection bias drops the loss and can quietly override the router

2 of 2 published

Residual & normalization

What keeps activations and logits in range, and what can replace the plain residual stream.

R1 · 30 min

Normalization and logit control

Most norms and clips in a modern decoder cost almost nothing to run; what each costs is a constraint on the architecture, and what each bounds is narrower than its name suggests

R2 · 35 min

Rethinking the residual stream

Wider residual streams and attention over depth lower the loss with little extra arithmetic in most designs; they cost activation bytes wherever many tokens run at once, and they differ in how they keep a learned weight on the identity path from compounding

1 of 1 published

Prediction head

Predicting more than the next token, as a training objective and as a speed-up at decode time.

P1 · 34 min

Multi-token prediction: a training signal and a built-in drafter

An MTP module takes one or two percent of a checkpoint's parameters and has two uses: a training signal whose measured gains are modest, and a drafter that loses value at each further token and can fall behind the backbone as training goes on

1 of 1 published

Memory

Capacity beyond the attention window: conditional memory and embedding-side storage.

M1 · 35 min

Conditional memory: lookup tables beside the experts

Tables addressed by the last few tokens add parameters for almost no arithmetic and can live outside GPU memory; they hold static patterns rather than computation, and their gains in loss carry to benchmarks unevenly

4 of 4 published

Data

How the corpus is filtered, mixed, synthesized and extended to other modalities.

D1 · 30 min

Filtering and deduplication: from a crawl to a corpus

A good filter raises average quality and leaves less unique text to fill a budget. Reports count what each step removes, and rarely what removing it bought

D2 · 35 min

Data mixture and curriculum: choosing what to read, and when

Fitting a data mix to an objective makes gains according to that objective, but risks losing what you don't measure, and is still affected by your chosen prior

D3 · 30 min

Synthetic data: what models write for other models

A generator writes what you can't crawl. It passes on its errors and biases, so what counts is whether anything can catch them

D4 · 25 min

Multimodal training data: when pixels and sound enter the recipe

Images from the first step train both modalities together, at a cost in text tokens; images added later keep a finished text model, which then has to be protected

2 of 2 published

Optimization

Optimizers such as Muon, and the precision a model is trained and quantized at.

O1 · 40 min

Optimizers: AdamW, Muon, and what each one bounds

Two limits on a training step, one on each weight and one on a layer's output, and what the difference costs in memory, communication and fine-tuning

O2 · 34 min

Training precision: when the model learns about its deployment format

Low-precision arithmetic in training buys speed, and training against the serving format buys accuracy at serving time; the earlier quantization-aware training starts, the longer the model adapts and the more of the recipe runs on a quantized model

6 of 6 published

Training stages

From pre-training through fine-tuning and reinforcement learning to distillation, and what each stage buys.

S1 · 35 min

Pre-training: one long run, chosen on small ones

A run that can't be rerun is set up on proxies, and its learning-rate schedule decides whether the end is fixed in advance or left open

S2 · 30 min

Mid-training and long-context extension: one word for several operations

A stage on a finished base adds skills and context length, the model loses some of what it knew, and both sides are mostly measured through stand-ins

S3 · 25 min

Supervised fine-tuning: a cold start for what comes next

SFT has to put in place every behaviour RL will later improve, and each token of it teaches the model to imitate a teacher; the measured effects are in what is kept, masked and averaged

S4 · 25 min

RLHF/RLVR: reward-based learning, from human preferences to verifiers

A program that checks the answer is cheap to run and gives the same verdict every time, but exists only for checkable tasks; a learned judge reaches open-ended ones and gets exploited by the policy it trains

S5 · 25 min

RL at scale: GRPO and the gap between sampler and trainer

Asynchrony, low-precision rollouts and a separate inference engine make RL affordable, and each makes the samples come from a slightly different policy than the one being trained

S6 · 25 min

On-policy distillation: folding specialists back into one model

Distilling RL-trained specialists on the student's own samples gives every token a signal and ships one model, and the student keeps only part of each teacher, unevenly by task

2 of 2 published

Multimodal

How images, audio and video get into a decoder built for text.

X1 · 30 min

Getting images and video into a decoder: encoders, projectors, and native fusion

How the mapped models turn pixels into decoder positions, when the vision part trains, and how many positions one image takes

X2 · 32 min

Hearing, speaking and keeping time: audio, omni models and multimodal positions

A separate speaking model keeps a large model's steps and context for text at the price of a second stack; one backbone is a single model whose every speech frame takes a step and a position; and time can travel in position ids, which cost no tokens, or in text, which is read like any other token

5 of 5 published

Inference & serving

Running a finished model cheaply: number formats, kernels, parallelism, the KV-cache system and speculative decoding.

I1 · 36 min

Quantization for serving: which bits to remove, and what each removal saves

Fewer bits in the weights save memory and bytes read; fewer in the activations raise peak arithmetic where the matrix units multiply that format and shrink what crosses between GPUs; fewer in the KV cache let a GPU hold more sessions; and when values share a scale, one large value decides the grid for the rest

I2 · 34 min

Kernels and graph compilation: where the cost model meets the GPU

A kernel gets close to the roofline by keeping intermediate results on chip, streaming what the phase needs at full bandwidth and hiding launches and communication behind work, and each of those gains is cut for one shape, one format and one GPU generation

I3 · 36 min

Parallelism for serving large MoE models: what each layout copies, and what it sends

Every layout copies something onto every GPU: tensor parallelism the single-head KV cache, data-parallel attention the non-expert weights. The batch and the context decide which copy costs more, and memory often decides before speed does

I4 · 31 min

Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory

Reading a cached prefix back from another tier usually costs far less than recomputing it; the price is in keeping it: memory held for hours, a route back to it, and for hybrid models a saved state at the right boundary

I5 · 32 min

Speculative decoding in production: drafters, acceptance and load

Acceptance length sets the ceiling on what a drafter can save; what each check costs at the load you serve decides how much of it you keep

Where to start

Reading paths