implementation detail · filed under optimization
NVFP4
NVFP4 is a 4-bit training or quantization format used for weights, activations, or gradients.
Also called NVFP4 4-bit Training Format, ultraefficient 4-bit NVFP4 training format, NVFP4 precision format, NVFP4 quantization, NVFP4 checkpoint, NVFP4 quantized checkpoints.
- sources
- 10
- models
- 4
- labs adopt it
- 3
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Pretraining Large Language Models with NVFP4 paper arxiv.orgNVIDIA's report on NVFP4 pretraining
- NVFP4 - Transformer Engine documentation docs docs.nvidia.comofficial format documentation
Evidence
13 spans quoted from the sources, strongest treatment first.
The pre-training phase used an NVFP4 recipe.
The majority of linear layers use NVFP4 for weights, activations, and gradients
These figures use the final NVFP4 weights that NVIDIA recommends for inference
use NVIDIA’s ultraefficient 4-bit NVFP4 training format on the NVIDIA Blackwell architecture
NVFP4 quantized checkpoints
We release a single NVFP4 checkpoint for Nemotron 3 Ultra
Pretrained in NVFP4.
open-sourcing the training recipes, data, and RL environments. Checkpoints ... NVFP4 quantized
We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4
We release a single NVFP4 checkpoint for Nemotron 3 Ultra
The NVFP4 checkpoint offers a quantized alternative that reduces the aggregated VRAM requirement to at least 600 GB.
NVIDIA publishes both BF16 and NVFP4 results across knowledge, reasoning, coding, agents, instruction following and long context.
Step 3.7 Flash is also available in an NVFP4-quantized variant for efficient deployment on NVIDIA GPUs
Filed alongside
Other methods under optimization :: training precision.