single source · model architecture
Modality-specific load balancing
Keeps a separate selection bias per modality, updated independently, so that balance is held among image tokens and among text tokens rather than only over their mix. DeepSeek-V4.1 reports it with no measurement of its effect.
Written for the explainer Expert load balancing.
Also called Modality-specific expert load balancing.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing
Explainer · C3 · 30 minExpert load balancingBalance over the whole batch lets experts specialize and leaves GPUs waiting; balance inside each micro-batch is faster and costs quality; a selection bias drops the loss and can quietly override the routerExplainer · D4 · 25 minMultimodal training data: when pixels and sound enter the recipeImages from the first step train both modalities together, at a cost in text tokens; images added later keep a finished text model, which then has to be protectedExplainer · X1 · 30 minGetting images and video into a decoder: encoders, projectors, and native fusionHow the mapped models turn pixels into decoder positions, when the vision part trains, and how many positions one image takes