Open Models Almanac
Methodsoptimizationtraining precision

specific method · filed under optimization

NVFP4 pretraining

NVFP4 pretraining is a training recipe that leverages Transformer Engine’s open-source cuBLAS NVFP4 GEMM kernels for fprop, dgrad, and wgrad.1 The method utilizes the E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on inputs to wgrad, and stochastic rounding on gradients.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We trained Nemotron 3 Ultra using the same NVFP4 pretraining recipe as Nemotron 3 Super nvidia2026nemotron3superopen, leveraging Transformer Engine’s open-source cuBLAS NVFP4 GEMM kernels for fprop, dgrad, and wgrad — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. NVFP4 layers use the E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on inputs to wgrad, and stochastic rounding on gradients nvidia2025pretraininglargelanguagemodels — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

Also called NVFP4 Pre-training, NVFP4 pre-training recipe, Pre-training with NVFP4 quantization, NVFP4 recipe.

sources
5
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

5 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 3

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted4 sources

it is trained using an NVFP4 pre-training recipe to maximize compute efficiency

coretraining objectivein Nemotron 3 UltraNVIDIA
3 more from this model

trained using an NVFP4 pre-training recipe to maximize compute efficiency

coreoptimizationin Nemotron 3 UltraNVIDIA

trained Nemotron 3 Ultra using the same NVFP4 pretraining recipe

coreoptimizationin Nemotron 3 UltraNVIDIA

NVFP4 pre-training

usedtraining objectivein Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑