Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Exponential moving average of estimated quantiles

Averaging quantile estimates over optimizer steps, so that a quantile bias passes less of one batch's sampling noise. The Kimi K3 report says it can improve balance further, and does not say K3 uses it.

Written for the explainer Expert load balancing.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing