Open Models Almanac
Methodsoptimizationquantization-aware training

specific method · filed under optimization

MXFP4 Quantization-Aware Training

MXFP4 quantization-aware training applies FP4 quantization during post-training so the model adapts to the precision degradation introduced by quantization.1 FP32 master weights are quantized to FP4 and dequantized back to FP8 for computation, with gradients propagated to the FP32 weights as through a straight-through estimator.2,3

Trade-off It targets inference acceleration and reduced memory traffic at deployment, at the cost of the precision degradation the model must adapt to.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we introduce Quantization-Aware Training (QAT) [QAT] during the post-training stage, enabling the model, including those of teacher and reference models, to adapt to the precision degradation introduced by quantization — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. the FP32 master weights maintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for computation — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. gradients are computed with respect to the same FP8 weights in the forward pass and directly propagated back to the FP32 master weights, equivalent to applying the Straight-Through Estimator (STE — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  4. To achieve inference acceleration and reducing memory traffic at deployment — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called FP4 Quantization-Aware Training, MXFP4 quantization-aware training (QAT), MXFP4 quantization-aware post-training.

sources
4
models
5
labs adopt it
3
strongest
used

Documented in

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

DeepSeek-V4-Pro

DeepSeekadopted2 sources

we incorporate FP4 quantization-aware training for MoE expert weights and the indexer QK path

usedpost trainingin DeepSeek-V4 seriesDeepSeek
1 more from this model

FP4 Quantization-Aware Training applies MXFP4 quantization to MoE expert weights and the indexer QK path during post-training.

usedpost trainingin DeepSeek-V4DeepSeek

Kimi K3

Moonshot AIadopted1 source

To reduce memory footprint and serving cost at deployment, we quantize the MoE expert weights — which dominate the model’s parameter memory — to MXFP4, with activations computed in MXFP8, while all non-expert components remain in higher precision. We perform quantization-aware training (QAT) throughout the entire post-training stage

usedunclearin Kimi K3Moonshot AI
↑