Hence we use the MSE calibration for the PTQ and AutoQuant recipes.
specific method · filed under inference & serving
MSE-based weight-scale selection
MSE-based weight-scale selection is a quantization calibration method that picks each block's weight scale to minimize quantization error.1,2 Unlike max-based scaling, which sets the scale from the block's absolute maximum, it searches over candidate scale values instead of taking the observed maximum.3,4
Trade-off It reduces weight reconstruction error more than max calibration, but downstream evaluations did not show a consistent accuracy increase.5
How this definition was written
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
MSE: Each block scale is selected to minimize quantization error by sweeping over FP8 scale values
— NVIDIA Nemotron 3.5 Super and Lightning Technical ReportMSE-based scaling minimizes reconstruction error
— NVIDIA Nemotron 3 Ultra Technical ReportMax: The scale is determined from the observed absolute maximum
— NVIDIA Nemotron 3.5 Super and Lightning Technical Reportmax-based scaling uses the block absolute maximum
— NVIDIA Nemotron 3 Ultra Technical ReportMSE calibration gave an additional weight-MSE reduction of 27.1%, downstream evaluations did not show a consistent accuracy increase
— NVIDIA Nemotron 3 Ultra Technical Report
Also called MSE-based scaling, Mean-squared-error weight-scale selection, MSE, MSE-based weight scaling, MSE calibration.
Documented in
Evidence
4 quotes from 3 sources, one card per model, each opening with its strongest.
MSE calibration gave an additional weight-MSE reduction of 27.1%, downstream evaluations did not show a consistent accuracy increase
2 more from this model
we experimented with max-based, MSE-based, and Four-Over-Six scaling
MSE-based scaling minimizes reconstruction error