single mention · inference & serving
MoonEP token migration for load balancing
Moves tokens from overloaded ranks to underloaded ranks until the target balanced load is reached.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See serving parallelism for the methods filed beside it.
Filed under inference & serving › serving parallelism