specific method · filed under optimization
NVFP4 pretraining
NVFP4 pretraining is a training recipe that leverages Transformer Engine’s open-source cuBLAS NVFP4 GEMM kernels for fprop, dgrad, and wgrad.1 The method utilizes the E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on inputs to wgrad, and stochastic rounding on gradients.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We trained Nemotron 3 Ultra using the same NVFP4 pretraining recipe as Nemotron 3 Super nvidia2026nemotron3superopen, leveraging Transformer Engine’s open-source cuBLAS NVFP4 GEMM kernels for fprop, dgrad, and wgrad
— Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningNVFP4 layers use the E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on inputs to wgrad, and stochastic rounding on gradients nvidia2025pretraininglargelanguagemodels
— Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Also called NVFP4 Pre-training, NVFP4 pre-training recipe, Pre-training with NVFP4 quantization, NVFP4 recipe.