Techniquesoptimizationquantization-aware training

specific method · filed under optimization

MXFP4 Quantization-Aware Post-Training

QAT applied during post-training to quantize MoE expert weights to MXFP4, with activations computed in MXFP8 in the supplied evidence.

source: 1
model: 1
lab adopt it: 1
strongest: used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1

Documented in

Evidence

1 span quoted from the sources, strongest treatment first.

To reduce memory footprint and serving cost at deployment, we quantize the MoE expert weights — which dominate the model’s parameter memory — to MXFP4, with activations computed in MXFP8, while all non-expert components remain in higher precision. We perform quantization-aware training (QAT) throughout the entire post-training stage

usedunclearin Kimi K3Moonshot AI

k3 tech report·Kimi Team

Filed alongside

Other methods under optimization :: quantization-aware training.

Quantization-Aware Training FP4 Quantization FP4 Quantization-Aware Training MXFP4 Weights with MXFP8 Activations Stochastic Rounding INT4 Quantization-Aware Training NVFP4 Training Per-Block Scalar Scaling Q4_0 Quantization Format Quantize-Dequantize Training Random Hadamard Transforms Stochastic Rounding for Mamba Cache Stochastic Rounding of Gradients Straight-Through Estimator