single mention · model architecture
Expert bias update factor
The step size of a fixed-step selection bias: after each batch, each expert's bias moves by this amount, down if it was over its share and up if under. It acts as a learning rate on the dual of the balanced assignment problem.
Written for the explainer Expert load balancing.
Filed under model architecture › channel mixer › mixture of experts › expert load balancing