Learn
Articles
Explainers for groups of methods that compete or combine: the problem they solve, the trade-off to remember, and where the models in the map sit. Unlike the method pages, these are written, not extracted.
3 of 3 published
Foundations
The cost model of a decoder layer and the stages of a training recipe; read these first.
6 of 6 published
Token mixer
How a layer reads the rest of the sequence, and what that costs in compute and KV-cache memory.
3 of 3 published
Channel mixer
What each token does on its own: gated feed-forward units and mixtures of experts.
2 of 2 published
Residual & normalization
What keeps activations and logits in range, and what can replace the plain residual stream.
1 of 1 published
Prediction head
Predicting more than the next token, as a training objective and as a speed-up at decode time.
1 of 1 published
Memory
Capacity beyond the attention window: conditional memory and embedding-side storage.
4 of 4 published
Data
How the corpus is filtered, mixed, synthesized and extended to other modalities.
2 of 2 published
Optimization
Optimizers such as Muon, and the precision a model is trained and quantized at.
6 of 6 published
Training stages
From pre-training through fine-tuning and reinforcement learning to distillation, and what each stage buys.
2 of 2 published
Multimodal
How images, audio and video get into a decoder built for text.
5 of 5 published
Inference & serving
Running a finished model cheaply: number formats, kernels, parallelism, the KV-cache system and speculative decoding.
Where to start


































