single mention · model architecture
Exponential moving average of estimated quantiles
Averaging quantile estimates over optimizer steps, so that a quantile bias passes less of one batch's sampling noise. The Kimi K3 report says it can improve balance further, and does not say K3 uses it.
Written for the explainer Expert load balancing.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing