Open Models Almanac
Methodstraining objectivedistillation objective

methods · training objective

distillation objective

25 methods filed at this node or below it, from the sources of 12 models.

In this branch 25

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 25

Full-vocabulary logit distillation core · 3 sources · 3 quotes
Reverse KL divergence core · 2 sources · 2 quotes
Quantization-aware distillation used · 4 sources · 5 quotes
Skew reverse-KL distillation used · 2 sources · 2 quotes
Temperature-scaled forward-KL distillation used · 2 sources · 2 quotes
Logit matching evaluated · 2 sources · 3 quotes
19 single mentions
Chain-rule decomposition of Top-K KL core · 1 source · 1 quote
Conditional Top-K-only temperature scaling core · 1 source · 3 quotes
Decoupled Top-K knowledge distillation core · 1 source · 3 quotes
Forward-KL distillation with a sink token used · 1 source · 1 quote
Full next-token distribution distillation used · 1 source · 1 quote
Hidden-state caching for KL distillation used · 1 source · 1 quote
KL-divergence distillation objective used · 1 source · 2 quotes
Depth distillation optional · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
NVIDIA-Nemotron-3-Ultra-550B-A55B usedTemperature-scaled forward-KL distillation usedLogit matching evaluated—
DeepSeek-V4-Flash usedFull-vocabulary logit distillation used—
DeepSeek-V4-Pro usedFull-vocabulary logit distillation used—
LFM3-1.7B coreChain-rule decomposition of Top-K KL coreConditional Top-K-only temperature scaling coreDecoupled Top-K knowledge distillation coreCross-encoder-to-ColBERT knowledge distillation usedMean squared error distillation of relevance scores usedMin-max normalization of teacher relevance scores usedTeacher-weighted reverse-KL surrogate for Top-K distillation evaluated—
Laguna-S-2.1 usedHidden-state caching for KL distillation usedQuantization-aware distillation used—
LingBot-VLA 2.0 coreDepth-representation distillation for geometric supervision coreDistillation-loss alignment of VLM queries with depth tokens coreDistilling current and future perceptual queries from pretrained perception models usedDepth distillation optional—
MiMo-V2.5 usedReverse KL divergence used—
Mistral-Large-4 coreFull-vocabulary logit distillation core—
MolmoAct 2 coreDistillation of specialist depth estimates into autoregressively predicted depth tokens core—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreReverse KL divergence coreBackbone-to-MTP-head knowledge distillation usedCombined KL-divergence and cross-entropy distillation loss usedForward-KL distillation with a sink token usedFull next-token distribution distillation usedKL-divergence distillation objective usedQuantization-aware distillation usedSkew reverse-KL distillation used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreReverse KL divergence coreBackbone-to-MTP-head knowledge distillation usedCombined KL-divergence and cross-entropy distillation loss usedForward-KL distillation with a sink token usedFull next-token distribution distillation usedKL-divergence distillation objective usedQuantization-aware distillation usedSkew reverse-KL distillation used—
Qwen3.8-Flash-Next usedDense-attention distillation to initialize the QSA indexer used—
↑