Open Models Almanac
Methodsinference & servinginference quantization

single mention · inference & serving

8-bit quantization

8-bit quantization loads a model's weights at 8-bit precision instead of the usual higher precision, reducing memory requirements for inference.1 It preserves accuracy comparable to full-precision inference, but the added quantization operations can slow down inference on some GPUs.2

Trade-off It cuts memory requirements but can slow inference due to quantization overhead.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. These approaches load the weights of the network at lower precision, thereby trading off reduced memory requirements for potentially reduced inference speed and accuracy — OpenVLA: An Open-Source Vision-Language-Action Model
  2. We observe that 8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations — OpenVLA: An Open-Source Vision-Language-Action Model
  3. 8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations — OpenVLA: An Open-Source Vision-Language-Action Model

Also called 8-bit model quantization, 8-bit.

Filed under inference & serving › inference quantization