single mention · optimization
Fully balanced expert-parallel training
A fully balanced expert-parallel training method keeps expert load balanced during training at large expert-parallel scales.1 It achieves this by using static shapes and avoiding host synchronization on the critical path.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
To prevent expert imbalance from degrading throughput at large expert-parallel scales, we introduce a fully balanced expert-parallel training method
— Kimi K3 Tech Blog: Open Frontier Intelligencewith static shapes and no host synchronization on the critical path
— Kimi K3 Tech Blog: Open Frontier Intelligence
Filed under optimization › training parallelism