Open Models Almanac
Methodsinference & servinginference quantization

general family · filed under inference & serving

Post-training quantization

Post-training quantization (PTQ) quantizes an already-trained model's weights to a lower-precision format for efficient inference.1 It works by calibrating the quantizers on a sample of data to produce a quantized checkpoint.2

Trade-off It enables efficient low-precision inference but can shift accuracy relative to the BF16 baseline.3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We apply post-training quantization (PTQ) using NVIDIA Model Optimizer to quantize Nemotron 3.5 Super to NVFP4 for efficient inference on NVIDIA GPUs — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. Calibration uses 512 packed multimodal samples — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  3. for efficient inference on NVIDIA GPUs — NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  4. Deltas are relative to the BF16 baseline — NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called Post-Training Quantization (PTQ).

sources
5
models
4
lab adopt it
1
strongest
used

Documented in

Evidence

5 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
used 4not used 1

We apply post-training quantization (PTQ) and quantization-aware distillation (QAD) with ModelOpt on Lightning 3.5 and Super 3.5 VL

usedinference servingin Nemotron 3.5 SuperNVIDIA
1 more from this model

The model uses NVIDIA Model Optimizer for post-training quantization and quantization-aware distillation (QAD).

usedinference servingin Nemotron 3.5 Super VLNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑