Open Models Almanac
Methodsoptimizationtraining parallelism

single mention · optimization

Hybrid ZeRO bucket assignment for Muon

Hybrid ZeRO bucket assignment for Muon is a strategy for combining the Muon optimizer with ZeRO data parallelism.1 Because Muon needs the full gradient matrix while ZeRO partitions element-wise updates, it limits ZeRO parallelism, assigns parameter matrices to ranks with a knapsack algorithm, and pads buckets for efficient reduce-scatter.2,3,4

Trade-off When data parallelism exceeds the ZeRO limit, Muon updates are computed redundantly across extra groups, trading computation for reduced total bucket memory.5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. To address this conflict, we design a hybrid strategy of ZeRO bucket assignment for Muon — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. The Muon optimizer requires the full gradient matrix to compute parameter updates, which presents a challenge when combined with the Zero Redundancy Optimizer (ZeRO — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. we limit the maximum size of ZeRO parallelism and employ a knapsack algorithm to assign parameter matrices to these ranks — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  4. The bucket on each rank is padded to match the size of the largest bucket across ranks, facilitating efficient reduce-scatter operations — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  5. we compute the Muon update redundantly across the extra data-parallel groups, trading computation for reduced total bucket memory — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Filed under optimization › training parallelism