Open Models Almanac
Methodsmodel architecturechannel mixermixture of expertsexpert load balancing

single mention · model architecture

Uniform-bin histogram quantile estimation

Uniform-bin histogram quantile estimation approximates a quantile of router scores by partitioning their range into B uniform bins and counting values per bin.1 Each rank accumulates local counts into a histogram, an all-reduce pools them, and the quantile is read from the first bin whose cumulative count reaches the target, interpolating within it.2

Trade-off It is accurate to within the bin width and cheap, needing only one integer all-reduce per layer per step instead of exchanging raw margins.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We partition this interval into B uniform bins — k3 tech report
  2. a single all-reduce sums the local counts into the global histogram, and every rank recovers the quantile from the same pooled counts — k3 tech report
  3. the only communication is one integer all-reduce of nB values per layer per step — k3 tech report

Also called B uniform bins.

Filed under model architecture › channel mixer › mixture of experts › expert load balancing