Open Models Almanac
Methodsinference & servinginference quantization

implementation detail · filed under inference & serving

Block-scaled INT8 quantization with stochastic rounding

Block-scaled INT8 quantization with stochastic rounding quantizes the Mamba SSM cache from FP32 precision to 8-bit INT8.1,2 Stochastic rounding and mantissa precision keep the quantized cache accurate, outperforming FP8 E4M3 which degrades accuracy.3,4

Trade-off It largely preserves FP32-cache accuracy and verbosity while compressing the cache, but no explicit cost is stated beyond that.5

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. we quantize the Mamba SSM cache from the original FP32 precision to lower precisions — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. Block-scaled INT8 quantization with stochastic rounding — NVIDIA Nemotron 3 Ultra Technical Report
  3. the mantissa precision and stochastic rounding are key ingredients for preserving accuracy after Mamba cache quantization — NVIDIA Nemotron 3 Ultra Technical Report
  4. FP8 E4M3 quantization degrades accuracy, likely because it provides lower effective precision for these cache values than INT8 — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  5. Block-scaled INT8 quantization with stochastic rounding largely preserves FP32-cache accuracy and verbosity — NVIDIA Nemotron 3 Ultra Technical Report

Also called Block-scaled INT8 quantization.

sources
2
model
1
labs adopt it
0
strongest
evaluated

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 2

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAdiscussed2 sources

Block-scaled INT8 quantization with stochastic rounding largely preserves FP32-cache accuracy

evaluatedpost trainingin Nemotron 3 SuperNVIDIA
1 more from this model

Block-scaled INT8 quantization with stochastic rounding largely preserves FP32-cache accuracy

evaluatedinference servingin Nemotron 3 SuperNVIDIA
↑