Open Models Almanac
Methodsinference & servinginference quantization

specific method · filed under inference & serving

MSE-based weight-scale selection

MSE-based weight-scale selection is a quantization calibration method that picks each block's weight scale to minimize quantization error.1,2 Unlike max-based scaling, which sets the scale from the block's absolute maximum, it searches over candidate scale values instead of taking the observed maximum.3,4

Trade-off It reduces weight reconstruction error more than max calibration, but downstream evaluations did not show a consistent accuracy increase.5

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. MSE: Each block scale is selected to minimize quantization error by sweeping over FP8 scale values — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. MSE-based scaling minimizes reconstruction error — NVIDIA Nemotron 3 Ultra Technical Report
  3. Max: The scale is determined from the observed absolute maximum — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  4. max-based scaling uses the block absolute maximum — NVIDIA Nemotron 3 Ultra Technical Report
  5. MSE calibration gave an additional weight-MSE reduction of 27.1%, downstream evaluations did not show a consistent accuracy increase — NVIDIA Nemotron 3 Ultra Technical Report

Also called MSE-based scaling, Mean-squared-error weight-scale selection, MSE, MSE-based weight scaling, MSE calibration.

sources
3
models
3
lab adopt it
1
strongest
used

Documented in

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 3used 1

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAdiscussed2 sources

MSE calibration gave an additional weight-MSE reduction of 27.1%, downstream evaluations did not show a consistent accuracy increase

evaluatedpost trainingin Nemotron 3 UltraNVIDIA
2 more from this model

we experimented with max-based, MSE-based, and Four-Over-Six scaling

evaluatedpost trainingin Nemotron 3 UltraNVIDIA

MSE-based scaling minimizes reconstruction error

evaluatedpost trainingin Nemotron 3 UltraNVIDIA
↑