Open Models Almanac
Methodstraining objectiveauxiliary loss

single mention · training objective

MoE sequence auxiliary loss

An auxiliary balancing loss with token counts taken within one sequence, usually at a very small coefficient beside a selection bias, as a guard against extreme imbalance in a single sequence. DeepSeek-V3 uses 0.0001; the MiMo-V2-Flash report prints 1.0×10⁻⁵ in pretraining.

Written for the explainer Expert load balancing.

Also called MoE sequence auxiliary loss coefficient.

Filed under training objective › auxiliary loss