Techniquesoptimizationquantization-aware training

specific method · filed under optimization

NVFP4 Training

Pretraining using the NVFP4 number format, reported on a hybrid Mamba-MoE architecture.

Also called quantized pretraining with NVFP4.

source: 1
model: 1
lab adopt it: 1
strongest: used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 1

Documented in

NVIDIA-Nemotron-3.5-Lightning-30B-A3B

Evidence

1 span quoted from the sources, strongest treatment first.

We demonstrate stable and accurate pretraining on a hybrid Mamba-MoE architecture for up to 25T tokens using the NVFP4 number format.

usedoptimizationin Nemotron 3 familyNVIDIA

NVIDIA Nemotron 3: Efficient and Open Intelligence·NVIDIA·2025-12-24

Filed alongside

Other methods under optimization :: quantization-aware training.

Quantization-Aware Training FP4 Quantization FP4 Quantization-Aware Training MXFP4 Weights with MXFP8 Activations Stochastic Rounding INT4 Quantization-Aware Training MXFP4 Quantization-Aware Post-Training Per-Block Scalar Scaling Q4_0 Quantization Format Quantize-Dequantize Training Random Hadamard Transforms Stochastic Rounding for Mamba Cache Stochastic Rounding of Gradients Straight-Through Estimator