Open Models Almanac
Methodsinference & servinginference quantization

general family · filed under inference & serving

Quantization

Quantization loads a model's weights at lower numerical precision for inference.1 It reduces the memory needed to serve the model, at the cost of potentially slower inference and reduced accuracy.2

Trade-off It cuts GPU memory requirements, but added quantization operations can slow inference, with 8-bit slower and 4-bit faster than higher precision.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. These approaches load the weights of the network at lower precision — OpenVLA: An Open-Source Vision-Language-Action Model
  2. thereby trading off reduced memory requirements for potentially reduced inference speed and accuracy — OpenVLA: An Open-Source Vision-Language-Action Model
  3. 8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations. 4-bit inference achieves higher throughput — OpenVLA: An Open-Source Vision-Language-Action Model

Also called quantization algorithms, quantized versions, Reduced-precision quantized inference.

sources
4
models
3
labs adopt it
3
strongest
core

Documented in

Google DeepMind
OpenVLA project (Stanford et al.)
Tencent (Hunyuan)

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
optional 3core 1

Gemma 4 31B

Google DeepMindadopted2 sources

Quantization is a key technology to run models on consumer hardware by reducing their memory footprint while also accelerating decode speed.

coreoptimizationin Gemma 4Google
1 more from this model

quantized versions run natively on consumer GPUs to power your IDEs, coding assistants and agentic workflows

optionalinference servingin Gemma 4Google

Hy3

Tencent (Hunyuan)adopted1 source

AngelSlim supports a comprehensive suite of compression tools ... including common quantization algorithms, low-bit quantization

optionalsoftware implementationin AngelSlimTencent Hy Team
↑