Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

specific method · filed under model architecture

Load-error gradient injection

Load-Error Injection (LEI) is an expert load balancing method for MoE routing with sigmoid scores that balances each microbatch individually.1 It adds a rescaled version of each expert's relative load error to the sigmoid-score gradient of every token in the microbatch, instead of using a shared global bias or auxiliary loss.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We introduce LEI to balance each microbatch, which directly injects each expert’s deviation from its target into the score gradients — tech report
  2. We inject a rescaled version of the load-error vector into the sigmoid-score gradient of every token in the microbatch — tech report

Also called Load-Error Injection, Load-Error Injection (LEI).

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Aleph Alpha

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 1

Kolibri-1

Aleph Alphaadopted2 sources

Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)

coremodel architecturein KolibriAleph Alpha
2 more from this model

We introduce LEI to balance each microbatch, which directly injects each expert’s deviation from its target into the score gradients.

usedunclearin KolibriAleph Alpha

We additionally injected load-error gradients into the router scores with a weight of 1e-5 and a soft-clamp value of 1.0.

usedoptimizationin KolibriAleph Alpha
↑