specific method · filed under model architecture
Load-error gradient injection
Load-Error Injection (LEI) is an expert load balancing method for MoE routing with sigmoid scores that balances each microbatch individually.1 It adds a rescaled version of each expert's relative load error to the sigmoid-score gradient of every token in the microbatch, instead of using a shared global bias or auxiliary loss.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We introduce LEI to balance each microbatch, which directly injects each expert’s deviation from its target into the score gradients
— tech reportWe inject a rescaled version of the load-error vector into the sigmoid-score gradient of every token in the microbatch
— tech report
Also called Load-Error Injection, Load-Error Injection (LEI).