single mention · training objective
MoE sequence auxiliary loss
An auxiliary balancing loss with token counts taken within one sequence, usually at a very small coefficient beside a selection bias, as a guard against extreme imbalance in a single sequence. DeepSeek-V3 uses 0.0001; the MiMo-V2-Flash report prints 1.0×10⁻⁵ in pretraining.
Written for the explainer Expert load balancing.
Also called MoE sequence auxiliary loss coefficient.
Filed under training objective › auxiliary loss