Model techniques map
Taxonomytraining objective

taxonomy area · pipeline stage 02

training objective

29 methods filed at this node or below it, from the sources of 17 models.

training objective

Matching aids for the classifier: the loss the model is trained against.

In this branch 29

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 1

SigLIP sigmoid contrastive loss used · 1 source · 1 quote

language modelling objective 6

Cross-entropy objective used · 1 source · 1 quote
Next-token prediction used · 1 source · 1 quote
Sampled-token objective used · 1 source · 1 quote
Stricter token constraints during training used · 1 source · 1 quote
Text pre-training used · 1 source · 1 quote
Fill-in-the-middle (FIM) completion optional · 1 source · 1 quote

multi-token prediction objective 6

Multi-Token Prediction used · 3 sources · 5 quotes
Multi-Token Prediction Boosting used · 3 sources · 5 quotes
Multi-step MTP training used · 2 sources · 2 quotes
Continued pretraining for MTP layers used · 1 source · 1 quote
Frozen-backbone MTP-head fine-tuning used · 1 source · 1 quote
Shared-weight Multi-Token Prediction used · 1 source · 1 quote

distillation objective 7

Full-vocabulary logit distillation used · 2 sources · 2 quotes
Temperature-scaled forward KL distillation used · 2 sources · 2 quotes
Dense-attention distillation used · 1 source · 1 quote
Hidden-state caching for KL distillation used · 1 source · 1 quote
Quantization-aware distillation used · 1 source · 1 quote
Reverse KL divergence used · 1 source · 1 quote
Logit matching evaluated · 2 sources · 3 quotes

auxiliary loss 9

KL alignment loss used · 3 sources · 4 quotes
Sample-level attention masking used · 3 sources · 3 quotes
Domain-specific KL regularization strength used · 1 source · 2 quotes
Gradient detachment used · 1 source · 2 quotes
Language modeling loss plus KL loss used · 1 source · 1 quote
Loss masking used · 1 source · 1 quote
MoE sequence auxiliary loss used · 1 source · 1 quote
Sequence-level balance loss used · 1 source · 1 quote
Unfinished-trajectory loss masking used · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled herelanguage modelling objectivemulti-token prediction objectivedistillation objectiveauxiliary loss
DeepSeek-V4.1-Flash usedSigLIP sigmoid contrastive loss used—Next-token prediction used———Sample-level attention masking usedSequence-level balance loss used—
DeepSeek-V4-Flash-0731 optional—Fill-in-the-middle (FIM) completion optional————
NVIDIA-Nemotron-3-Ultra-550B-A55B used—Sampled-token objective used—Frozen-backbone MTP-head fine-tuning usedMulti-Token Prediction Boosting usedShared-weight Multi-Token Prediction used—Temperature-scaled forward KL distillation usedLogit matching evaluated—Unfinished-trajectory loss masking used—
MiMo-V2.6-Flash used—Cross-entropy objective used———Loss masking used—
DeepSeek-V4-Flash used———Full-vocabulary logit distillation used—Sample-level attention masking used—
MiMo-V2.5 used—Text pre-training used—Multi-Token Prediction used—Reverse KL divergence used—MoE sequence auxiliary loss used—
MiniMax-M3 used————Gradient detachment usedKL alignment loss usedLanguage modeling loss plus KL loss used—
DeepSeek-V3.2 used—Stricter token constraints during training used———Domain-specific KL regularization strength usedKL alignment loss used—
DeepSeek-V4-Flash-Vision-Exp optional—Fill-in-the-middle (FIM) completion optional————
DeepSeek-V4-Pro used———Full-vocabulary logit distillation used—Sample-level attention masking used—
DeepSeek-V4-Pro-0813 optional—Fill-in-the-middle (FIM) completion optional————
Laguna-S-2.1 used———Hidden-state caching for KL distillation usedQuantization-aware distillation used——
MiMo-V2.5-Pro used—Text pre-training used—Multi-Token Prediction used———
MiMo-V2.6-Pro used—Cross-entropy objective used———Loss masking used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B used——Continued pretraining for MTP layers usedMulti-Token Prediction usedMulti-Token Prediction Boosting used———
Qwen3.5-397B-A17B used——Multi-step MTP training used———
Qwen3.8-Flash-Next used——Multi-Token Prediction used—Dense-attention distillation used——