Open Models Almanac
Methodsoptimizationtraining parallelism

single mention · optimization

Fully balanced expert-parallel training

A fully balanced expert-parallel training method keeps expert load balanced during training at large expert-parallel scales.1 It achieves this by using static shapes and avoiding host synchronization on the critical path.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. To prevent expert imbalance from degrading throughput at large expert-parallel scales, we introduce a fully balanced expert-parallel training method — Kimi K3 Tech Blog: Open Frontier Intelligence
  2. with static shapes and no host synchronization on the critical path — Kimi K3 Tech Blog: Open Frontier Intelligence

Filed under optimization › training parallelism