Open Models Almanac
Methodsinference & servinginference quantization

single mention · inference & serving

Restricted-search MSE weight-scale selection

MSE 4-over-6 is a weight-scale selection scheme for quantization that picks each block scale to minimize quantization error by sweeping over FP8 scale values.1 It differs from plain MSE search by restricting the scale search to a narrower space.2

Trade-off Evaluation found it recovers accuracy worse than unrestricted MSE scale selection.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. MSE: Each block scale is selected to minimize quantization error by sweeping over FP8 scale values — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. MSE 4-over-6: MSE-based scale selection restricted to a narrower search space — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  3. the MSE algorithm is superior in terms of accuracy recovery compared to the Max, and MSE 4-over-6 — NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called MSE 4-over-6.

Filed under inference & serving › inference quantization