general family · filed under optimization
Quantization-Aware Training
Training a model with quantization in the loop so it can adapt to quantized computation; the evidence does not specify a single bit format.
Also called Quantization-Aware Training (QAT).
- sources
- 8
- models
- 4
- labs adopt it
- 3
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
8 spans quoted from the sources, strongest treatment first.
By simulating quantization during training, QAT minimizes quality loss when the model is compressed.
Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility.
In addition, we apply quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations (§ 4.1.4).
Quantization-aware training starts at the supervised fine-tuning stage, using MXFP4 weights and MXFP8 activations
We provide quantized versions of our models trained with quantization-aware training [Jacob et al., 2018, QAT] to reduce their parameter memory footprint and latency with minimal impact on quality.
we introduce Quantization-Aware Training (QAT) [QAT] during the post-training stage
K3 employs quantization-aware training (QAT) starting from the supervised fine-tuning stage, not post-training quantization.
Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility.
Filed alongside
Other methods under optimization :: quantization-aware training.