implementation detail · filed under model architecture
Histogram-based quantile estimation
Estimates expert quantiles from pooled histogram bin counts, combined across ranks with an all-reduce.
Also called Histogram estimation, B uniform bins.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
We instead read each expert’s quantile from a histogram of its margins: a single all-reduce sums the per-rank bin counts, and the quantile is recovered from the pooled counts.
At the end of the step, a single all-reduce sums the local counts into the global histogram, and every rank recovers the quantile from the same pooled counts.
Filed alongside
Other methods under model architecture :: channel mixer :: mixture of experts :: expert load balancing.