Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Expert bias update factor

The step size of a fixed-step selection bias: after each batch, each expert's bias moves by this amount, down if it was over its share and up if under. It acts as a learning rate on the dual of the balanced assignment problem.

Written for the explainer Expert load balancing.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing