general family · filed under inference & serving
Quantization
Quantization loads a model's weights at lower numerical precision for inference.1 It reduces the memory needed to serve the model, at the cost of potentially slower inference and reduced accuracy.2
Trade-off It cuts GPU memory requirements, but added quantization operations can slow inference, with 8-bit slower and 4-bit faster than higher precision.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
These approaches load the weights of the network at lower precision
— OpenVLA: An Open-Source Vision-Language-Action Modelthereby trading off reduced memory requirements for potentially reduced inference speed and accuracy
— OpenVLA: An Open-Source Vision-Language-Action Model8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations. 4-bit inference achieves higher throughput
— OpenVLA: An Open-Source Vision-Language-Action Model
Also called quantization algorithms, quantized versions, Reduced-precision quantized inference.