general family · filed under inference & serving
Post-training quantization
Post-training quantization (PTQ) quantizes an already-trained model's weights to a lower-precision format for efficient inference.1 It works by calibrating the quantizers on a sample of data to produce a quantized checkpoint.2
Trade-off It enables efficient low-precision inference but can shift accuracy relative to the BF16 baseline.3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We apply post-training quantization (PTQ) using NVIDIA Model Optimizer to quantize Nemotron 3.5 Super to NVFP4 for efficient inference on NVIDIA GPUs
— NVIDIA Nemotron 3.5 Super and Lightning Technical ReportCalibration uses 512 packed multimodal samples
— NVIDIA Nemotron 3.5 Super and Lightning Technical Reportfor efficient inference on NVIDIA GPUs
— NVIDIA Nemotron 3.5 Super and Lightning Technical ReportDeltas are relative to the BF16 baseline
— NVIDIA Nemotron 3.5 Super and Lightning Technical Report
Also called Post-Training Quantization (PTQ).