Open Models Almanac

Inference & serving · I1

Quantization for serving: which bits to remove, and what each removal saves

Fewer bits in the weights save memory and bytes read; fewer in the activations raise peak arithmetic where the matrix units multiply that format and shrink what crosses between GPUs; fewer in the KV cache let a GPU hold more sessions; and when values share a scale, one large value decides the grid for the rest

Illustration of a wooden sorting tray with a few rows of shallow compartments holding teal pebbles in a small number of standard sizes. One large coral stone rests beside the tray on a folded cloth, with a small gold gauge next to it.

NVIDIA’s Nemotron 3.5 Super VL has 124.453 billion parameters. In 16-bit floating point that is about 249 GB (computed). The released checkpoint is 74.987 GB, an average of 4.82 bits per parameter. Per GB200 GPU it served 2.43 times the output of the 16-bit weights at a promised 50 tokens per second per user, and, taken as a median of per-benchmark ratios, its score came to 99.62% of the 16-bit model’s (the report’s §4 and Figure 10). Few deployments would choose 16 bits, so 8 bits is the fairer comparison. There vLLM’s recipe for Mistral Large 4 (another model, and per-token speed rather than throughput at a promised rate) reports 1.09 to 1.18 times FP8 at TP8 on eight GB300 GPUs, without speculative decoding.

From June 29 to July 2, 2026, the team behind umans.ai let volunteers use a third-party 4-bit conversion of GLM-5.2, served at 200+ tokens per second. Five benchmarks on its model card were within about a point of the 8-bit baseline. One volunteer reported that the model’s chain of thought collapsed, “not even at deep context”, others went back to the 8-bit version, and the team closed the experiment.

The names on model cards (“FP8”, “NVFP4”, “MXFP4”, “INT4 (W4A16)”, “Q4_K_M”) differ in four ways: the format of each element, how scales are shared between elements, which tensors are quantized (weights, activations or the KV cache), and whether the hardware can multiply in that format or only store it.

The background you need

00 gives the cost model and 02 the serving levers. The short version: decode, which produces one token per step, is limited by bytes moved from memory, and prefill, which reads the prompt, is limited by arithmetic. A Tensor Core is the part of an NVIDIA GPU that multiplies matrices. It works at its highest rate only in the number formats it was built for: the Hopper generation (H100, H200) goes down to FP8, and the Blackwell generation (B200, B300 and their GB variants) also multiplies in 4-bit formats. A kernel is the piece of software that performs one such multiplication, and I2 covers the kernels.

The unit here is bits per element (BPE): the stored bits, scales included, divided by the parameter count. BF16 and FP16 are 16-bit floating-point formats, and BF16 is the one models are usually trained in.

Quantizing a number means dividing by a scale and picking the nearest representable value:

x^=s⋅Q(x/s)\hat{x} = s \cdot Q(x / s)

Here QQ rounds to the format’s grid and ss is the scale. Everything below is a choice of QQ, a choice of how ss is set and how many elements share it, or a way of changing xx before rounding.

What a format is

An integer format has evenly spaced values. In the 4-bit integer grid most often meant by INT4, the values run from −7-7 to +7+7 in steps of one. Floating-point formats use some bits for an exponent, so the values crowd towards zero.

FP8 comes in two flavours, E4M3 (four exponent bits, three mantissa bits) and E5M2. E4M3 reserves no exponent for infinity, which is why it reaches ±448\pm 448 where an IEEE layout would stop at 240. Its smallest non-zero magnitude is 2−92^{-9}, a ratio of 229,376 between the extremes (computed). NVIDIA recommends E4M3 for weights and activations.

FP4 is E2M1: one sign, two exponent, one mantissa bit, 16 codes. Plus zero and minus zero are the same number, so it holds 15 distinct values: 0, ±0.5\pm 0.5, ±1\pm 1, ±1.5\pm 1.5, ±2\pm 2, ±3\pm 3, ±4\pm 4 and ±6\pm 6. Its largest magnitude is twelve times its smallest non-zero one. A value rounds to zero when it is closer to 0 than to the smallest non-zero value: in E2M1 that is everything below 0.25 on the grid, in INT4 everything below 0.5.

Figure 1.The positive half of two 4-bit grids. Both have 15 values counting zero and the negatives. E2M1's steps double as the values grow, so it is finer than INT4 near zero and coarser near the top. Mapped so a tensor's largest value $a$ reaches each grid's top, the band that rounds to zero ends at $a/24$ for E2M1 and $a/14$ for INT4.

Which grid wins depends on the distribution. In a toy of a single best scale on independent values, E2M1 and INT4 give almost the same error on Gaussian values, and E2M1 is better by 1.7 dB on Laplace values and 2.1 dB on a heavier-tailed Student-t (computed).

With only 15 values, a format needs a scale that maps them onto the range of the data. How many elements share one ss is the scale’s granularity.

One outlier, with and without block scales

Take a row of 64 weights with mostly small values and one large one. (This is a toy: a seeded Laplace vector with scale 0.02 and one value set to 0.9.) With one scale per tensor, the scale comes from the largest value. E2M1 maps 0.9 to its top value of 6, so s=0.15s = 0.15, and anything below 0.25×0.15=0.03750.25 \times 0.15 = 0.0375 rounds to zero. In the toy 54 of the 64 values do. INT4, with a step of 0.9/7≈0.130.9/7 \approx 0.13, zeroes 59. FP8, with its wide range, zeroes none.

With one scale per block of 16, the outlier affects only the values in its own block. Over the 48 values outside it, the root-mean-square error is 0.0151 for FP4 with one scale per tensor and 0.0022 for NVFP4 with block scales. FP8 gets 0.0005 on the same values, so NVFP4’s error is still about 4.5 times as large, at 4.5 bits against 8 (computed). On real output-head weights NVIDIA measures a ratio of 3.6 (0.0952 against 0.0265, Table 39).

Toy row of 64 values (seeded, not a model's).Filled: the value. Ring: what it comes back as.one scale for all 64−0.100.1016324864value 0: 0.0750 comes back as 0.1286value 1: -0.0020 comes back as 0.0000value 2: 0.0003 comes back as 0.0000value 3: 0.0111 comes back as 0.0000value 4: 0.0145 comes back as 0.0000value 6: -0.0091 comes back as 0.0000value 7: 0.0391 comes back as 0.0000value 8: -0.0141 comes back as 0.0000value 9: 0.0360 comes back as 0.0000value 10: 0.0122 comes back as 0.0000value 11: -0.0112 comes back as 0.0000value 12: 0.0590 comes back as 0.0000value 13: 0.0193 comes back as 0.0000value 14: 0.0095 comes back as 0.0000value 15: -0.0117 comes back as 0.0000value 16: -0.0128 comes back as 0.0000value 17: -0.0148 comes back as 0.0000value 18: -0.0047 comes back as 0.0000value 19: 0.0129 comes back as 0.0000value 20: 0.0224 comes back as 0.0000value 21: -0.0415 comes back as 0.0000value 22: 0.0031 comes back as 0.0000value 23: 0.0561 comes back as 0.0000value 24: -0.0287 comes back as 0.0000value 25: 0.0195 comes back as 0.0000value 26: -0.0268 comes back as 0.0000value 27: -0.0165 comes back as 0.0000value 28: 0.0146 comes back as 0.0000value 29: -0.0581 comes back as 0.0000value 30: 0.0058 comes back as 0.0000value 31: -0.0327 comes back as 0.0000value 32: 0.0017 comes back as 0.0000value 33: 0.0652 comes back as 0.1286value 34: -0.0242 comes back as 0.0000value 35: 0.0111 comes back as 0.0000value 36: -0.0021 comes back as 0.0000value 37: -0.0046 comes back as 0.0000value 38: 0.0023 comes back as 0.0000value 39: 0.0104 comes back as 0.0000value 40: -0.0025 comes back as 0.0000value 41: -0.0060 comes back as 0.0000value 42: -0.0034 comes back as 0.0000value 43: 0.0011 comes back as 0.0000value 44: 0.0135 comes back as 0.0000value 45: 0.0033 comes back as 0.0000value 46: -0.0139 comes back as 0.0000value 47: 0.0883 comes back as 0.1286value 48: -0.0169 comes back as 0.0000value 49: -0.0239 comes back as 0.0000value 50: 0.0050 comes back as 0.0000value 51: -0.0653 comes back as -0.1286value 52: -0.0022 comes back as 0.0000value 53: 0.0243 comes back as 0.0000value 54: 0.0064 comes back as 0.0000value 55: -0.0064 comes back as 0.0000value 56: -0.0190 comes back as 0.0000value 57: -0.0068 comes back as 0.0000value 58: -0.0128 comes back as 0.0000value 59: -0.0099 comes back as 0.0000value 60: -0.0136 comes back as 0.0000value 61: 0.0111 comes back as 0.0000value 62: 0.0159 comes back as 0.0000value 63: 0.0111 comes back as 0.0000outlier 0.90 → 0.90 (off the axis)Toy row of 64 values (seeded, not a model's). Filled: the value. Ring: what it comes back as.one scale for all 64−0.100.1016324864value 0: 0.0750 comes back as 0.1286value 1: -0.0020 comes back as 0.0000value 2: 0.0003 comes back as 0.0000value 3: 0.0111 comes back as 0.0000value 4: 0.0145 comes back as 0.0000value 6: -0.0091 comes back as 0.0000value 7: 0.0391 comes back as 0.0000value 8: -0.0141 comes back as 0.0000value 9: 0.0360 comes back as 0.0000value 10: 0.0122 comes back as 0.0000value 11: -0.0112 comes back as 0.0000value 12: 0.0590 comes back as 0.0000value 13: 0.0193 comes back as 0.0000value 14: 0.0095 comes back as 0.0000value 15: -0.0117 comes back as 0.0000value 16: -0.0128 comes back as 0.0000value 17: -0.0148 comes back as 0.0000value 18: -0.0047 comes back as 0.0000value 19: 0.0129 comes back as 0.0000value 20: 0.0224 comes back as 0.0000value 21: -0.0415 comes back as 0.0000value 22: 0.0031 comes back as 0.0000value 23: 0.0561 comes back as 0.0000value 24: -0.0287 comes back as 0.0000value 25: 0.0195 comes back as 0.0000value 26: -0.0268 comes back as 0.0000value 27: -0.0165 comes back as 0.0000value 28: 0.0146 comes back as 0.0000value 29: -0.0581 comes back as 0.0000value 30: 0.0058 comes back as 0.0000value 31: -0.0327 comes back as 0.0000value 32: 0.0017 comes back as 0.0000value 33: 0.0652 comes back as 0.1286value 34: -0.0242 comes back as 0.0000value 35: 0.0111 comes back as 0.0000value 36: -0.0021 comes back as 0.0000value 37: -0.0046 comes back as 0.0000value 38: 0.0023 comes back as 0.0000value 39: 0.0104 comes back as 0.0000value 40: -0.0025 comes back as 0.0000value 41: -0.0060 comes back as 0.0000value 42: -0.0034 comes back as 0.0000value 43: 0.0011 comes back as 0.0000value 44: 0.0135 comes back as 0.0000value 45: 0.0033 comes back as 0.0000value 46: -0.0139 comes back as 0.0000value 47: 0.0883 comes back as 0.1286value 48: -0.0169 comes back as 0.0000value 49: -0.0239 comes back as 0.0000value 50: 0.0050 comes back as 0.0000value 51: -0.0653 comes back as -0.1286value 52: -0.0022 comes back as 0.0000value 53: 0.0243 comes back as 0.0000value 54: 0.0064 comes back as 0.0000value 55: -0.0064 comes back as 0.0000value 56: -0.0190 comes back as 0.0000value 57: -0.0068 comes back as 0.0000value 58: -0.0128 comes back as 0.0000value 59: -0.0099 comes back as 0.0000value 60: -0.0136 comes back as 0.0000value 61: 0.0111 comes back as 0.0000value 62: 0.0159 comes back as 0.0000value 63: 0.0111 comes back as 0.0000outlier 0.90 → 0.90 (off the axis)

INT4 per tensor: 59 of 64 values round to zero. RMSE over values 16 to 63 (the 48 outside the outlier's block of 16): 0.02358. The outlier 0.9 comes back as 0.90.

Figure 2.A toy row of 64 values with one outlier. With a per-tensor scale at four bits nearly everything else rounds to zero. With block scales, mostly the outlier's own block suffers, and the other blocks get scales that suit them. FP8's wide range needs no help.

How far a block scale helps depends on the tails. In a toy with independent unit-variance values, the mean of the largest of 16 is 2.1 to 2.4 under a Gaussian, Laplace or Student-t distribution, while the largest of 1,024 is 3.5, 5.3 and 10.5 (computed). A small block removes most of the difference. Where a tensor has no outliers a block scale can change almost nothing: NVIDIA found its output head’s pooled error “identical to four decimal places” from group size 1 to 64 (App. B.5).

A set of values sharing one scale is called a block in the MX formats, a group in INT4, and a tile when it is a two-dimensional patch of a matrix. Smaller ones need more stored scales and more work in the kernel:

FormatElementScaleShared byBPE
FP8E4M3one 32-bit number, in softwarea tensor (or a 128 × 128 tile)8
INT4 with group 128−7…7-7 \ldots 716-bit float128 elements4.125
FP4E2M1one 32-bit number, in softwarea tensor4
MXFP8E4M3 or E5M2power of two (E8M0)32 elements8.25
MXFP4E2M1power of two (E8M0)32 elements4.25
NVFP4E2M1E4M3, plus a 32-bit number per tensor16 elements4.5
GGUF Q4_K4-bit integer6-bit scale and minimum, two 16-bit super-block scales32 elements in a super-block of 2564.5

(Bits computed from the formats’ definitions.)

MXFP4 is the Open Compute Project’s format (the OCP Microscaling spec v1.0, 2023). Its scale is E8M0, an unsigned exponent with no mantissa, so a block scale can only be a power of two, which NVIDIA says has “slightly reduced computational complexity”. Write aa for a block’s largest absolute value (often written amax). The spec’s default scale is 2⌊log⁡2a⌋−22^{\lfloor \log_2 a \rfloor - 2}, which puts aa somewhere in [4,8)[4, 8) on the grid, and E2M1 stops at 6, so values above 6 are clamped, by as much as 25% (computed). NVIDIA’s Model Optimizer rounds the scale up instead. Nothing clips, but the blocks that would have clipped get twice the scale, and in a heavy-tailed toy that gives 13% more error than the spec’s rule (computed). In the row above the outlier of 0.9 comes back as 0.75 under the spec’s rule and 1.0 under NVIDIA’s, so two files labelled MXFP4 can hold different numbers.

NVFP4 (NVIDIA, 2025) has one scale per 16 values, stored in E4M3, which isn’t limited to powers of two, so a block’s maximum can be mapped close to 6. E4M3’s range is smaller than E8M0’s, and a block of very small weights would need a scale below its minimum of 2−92^{-9}, so the whole tensor is first divided by one 32-bit number that brings it within reach. NVIDIA’s blog reports 1% or less degradation against FP8 on DeepSeek-R1-0528, and against MXFP4 only a qualitative “lower risk”.

NVIDIA also compared weight-scale rules on Nemotron 3.5 Super VL, over 12 benchmarks at one recipe. Setting a block’s scale from amax recovered 98.58% of the 16-bit score, where recovery is the quantized model’s score divided by the 16-bit one’s, and a sweep minimizing squared error recovered 100.06% (§4.2, Table 13).

What gets quantized

Routed and shared experts are 94% of Nemotron 3.5 Super VL’s parameters and about 88% of the released checkpoint’s bytes (computed from the config, since the report publishes no shares). Most of the byte savings therefore come from the experts.

Those shares describe storage. A decode step reads only the experts its tokens pick, 22 of 512 per layer in this model, so for one token the active experts are about 6.6B of the 12.2B parameters read (computed, ignoring the drafter). As the batch grows different tokens pick different experts, and once all are read, experts are 95% of the parameters read (91% of the released recipe’s weight bytes). The released recipe cuts weight bytes per step by a factor of 2.5 for one token and 3.4 when every expert is read, against BF16 (computed). Table 12 of the report lists each module’s precision under three recipes:

Cons. = Conservative, Aggr. = Aggressive,V. aggr. = Very aggressive. A strongershade = fewer bits per value. Bars: shareof parameters.BF16referenceFP8referenceCons.explored,not releasedAggr.releasedV. aggr.explored,not releasedVision: C-RADIO encoder0.51%BF16BF16BF16BF16BF16Vision: VL connector0.15%BF16BF16BF16BF16BF16MoE + shared GEMM94.35%BF16FP8NVFP4a: FP8 considered for the shared expert during exploration; the Conservative BPE estimate assumes NVFP4 for all shared experts.NVFP4NVFP4Latent down/up proj0.28%BF16FP8BF16BF16NVFP4Router / gate (compute)0.07%FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.Attention QKV / out proj0.26%BF16FP8BF16FP8NVFP4KV cacheruntime, not storedBF16FP8FP8FP8FP4BMM1 (QK^T)runtime, not storedBF16FP8FP8FP8FP4BMM2 (PV)runtime, not storedBF16FP8FP8FP8FP8Mamba linear3.52%BF16FP8BF16FP8NVFP4Mamba SSM cacheruntime, not storedFP32FP32FP32FP16+SRFP16+SRMamba convolution0.00%BF16BF16BF16BF16BF16Embedding0.43%BF16BF16BF16BF16BF16lm_head0.43%BF16FP8BF16FP8W4A16Avg. bits per element16.00estimate8.10estimate5.15estimate4.82measured4.64estimateBF16referenceFP8referenceConservativeexplored,not releasedAggressivereleasedVeryaggressiveexplored,not releasedShare of parameters(computed; bars, 0 to 100%)Vision: C-RADIO encoder0.51%BF16BF16BF16BF16BF16Vision: VL connector0.15%BF16BF16BF16BF16BF16MoE + shared GEMM94.35%BF16FP8NVFP4a: FP8 considered for the shared expert during exploration; the Conservative BPE estimate assumes NVFP4 for all shared experts.NVFP4NVFP4Latent down/up proj0.28%BF16FP8BF16BF16NVFP4Router / gate (compute)0.07%FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.FP32b: FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column.Attention QKV / out proj0.26%BF16FP8BF16FP8NVFP4KV cacheruntime, not storedBF16FP8FP8FP8FP4BMM1 (QK^T)runtime, not storedBF16FP8FP8FP8FP4BMM2 (PV)runtime, not storedBF16FP8FP8FP8FP8Mamba linear3.52%BF16FP8BF16FP8NVFP4Mamba SSM cacheruntime, not storedFP32FP32FP32FP16+SRFP16+SRMamba convolution0.00%BF16BF16BF16BF16BF16Embedding0.43%BF16BF16BF16BF16BF16lm_head0.43%BF16FP8BF16FP8W4A16Avg. BPE (whole checkpoint)16.00estimate8.10estimate5.15estimate4.82measured4.64estimateOnly 4.82 is measured; theothers are the report'sestimates.
Figure 3.Precision per module in NVIDIA's three recipes for Nemotron 3.5 Super VL, with the BF16 and FP8 references; only the Aggressive recipe was released. The BPE row is the whole checkpoint; only 4.82 is measured, the others are the report's estimates. KV-cache and activation rows do not enter it. Bars show each module's share of parameters (our calculation, since the report has no such column).

Experts are 4-bit in every recipe, and narrowing the other, small modules saves few bytes. NVIDIA’s sensitivity scan found “the lm_head accounts for nearly half of the NVFP4 sensitivity, attention and Mamba projections are the next most sensitive families, and MoE experts are comparatively robust” (§4.3). Elsewhere (§4.6) the report places the visible costs in generation length and long-context retrieval, with average accuracy barely moving.

At the end of a generation the model chooses between stopping and continuing, a small margin between two logits. On identical hidden states that choice changed 2.45% of the time with an NVFP4 output head and 0.78% with an FP8 one (Table 39), and NVIDIA concludes “the lm_head should stay at FP8 weights or above”. Lightning’s head is NVFP4 W4A16, for a stated aim to “minimize weight fetch”.

Of the 16 LLM labs in this corpus, eight state or configure a precision for both expert weights and attention projections in a release with 4-bit experts (NVIDIA, DeepSeek, Moonshot, Z.ai on Ascend, Mistral, OpenAI, MiniMax, Thinking Machines). All eight keep attention at higher precision than the experts, and none of the 16 states the reverse. FP8-only recipes, such as GLM-5.3’s default or Kolibri-1, keep the two equal. Routers are also left unconverted in gpt-oss, GLM-5.3, MiniMax M3.1-Flash and Kolibri-1.

Labs differ on drafters. NVIDIA quantizes Nemotron 3.5 Super’s MTP experts and keeps its attention in BF16. With the single FP8 output head it shipped, acceptance length at draft length 5 is 4.063 against 4.109 with a BF16 drafter and head (Table 20), a 1.1% drop for a projected 13% speed-up (simulated, one GB200). vLLM’s Kimi K3 recipes keep the drafter (DSpark, see 02) in BF16. Drafting also changes the sums above: a verify step covers several positions and reads more experts than a plain step, and NVIDIA’s MTP drafter must share the backbone’s KV format. I5 takes up the trade.

Weights, activations and the matrix units

Write WmmAnn for weights stored in mm bits and activations in nn bits. A matrix multiplication reads weights WW from memory and takes activations xx from the previous layer. A product of two rounded things has three errors:

(W+E)(x+e)−Wx=Ex+We+Ee(W + E)(x + e) - Wx = Ex + We + Ee

W4A16 keeps only the first. W4A8 and W4A4 add WeWe, and its size depends on the activations.

memory: weightsstored in 4 bitsactivations fromthe previous layer4 bitsexpand to 16 bitsmultipliermultiplies in BF1616 bits16 bitsother GPUsover NVLinkaccumulatesto FP32memory: weightsstored in 4 bitsactivations fromthe previous layer4 bitsexpand to 16 bitsmultipliermultiplies in BF1616 bits16 bitsother GPUsover NVLinkaccumulatesto FP32

What the removed bits change: weight bytes read.

Figure 4.Where the bits are in one multiplication. W4A16, W4A8 and W4A4 read the same weight bytes. They differ at the multiplier and in the activation bytes that cross between GPUs.

At small batches decode is memory-bound. W4A16 stores the experts in about a quarter of their bytes and leaves the activations at 16 bits, and the kernel (Marlin in vLLM and SGLang) expands the weights just before the multiplication, so the arithmetic is still BF16 and any GPU can do it. LMSYS: “H-series GPUs do not have native INT4 Tensor Cores. W4A16 essentially still uses BF16 Tensor Cores”. On Hopper that makes W4A16 the opposite trade from W8A8. FP8 runs at twice BF16’s peak there, so prefill and any other compute-bound phase are faster in W8A8, and decode and capacity are better in W4A16. On a Qwen3-235B rollout LMSYS found INT4 at “roughly the same performance tier as FP8”. Which wins depends on the batch.

In W4A4 the matrix units themselves multiply 4-bit numbers. NVIDIA states FP4 math at twice the rate of FP8 on GB200 and three times on GB300. Blackwell’s Tensor Cores take “partial dot-products over the block, multiply each partial product by the corresponding scale factors”, one for the weight block and one for the activation block, and accumulate to FP32, so the block scale is part of the hardware operation. AMD’s MI350X and MI355X list MXFP4 too, and vLLM’s recipes offer W4A4 (“a4w4”) there, for some models by default and for others behind a flag.

W4A4 also rounds the activations to four bits, and how much that matters depends on what they are. In Nemotron’s experts the second linear layer’s input is 90 to 99% exactly zero, so a block holds one to three non-zero values and rounding has little to destroy. Its Mamba output projection receives values that are small and never exactly zero: about 16 survive per block and 27.6% fall below a/24a/24, which is where E2M1 rounds to zero with the block’s largest value mapped to 6 (App. B.4). NVIDIA kept that layer out of W4A4.

The three layouts read the same weight bytes. They differ in the arithmetic of the expert multiplications and in the activation bytes moved between GPUs, so W4A4’s faster arithmetic pays only at batches large enough for those multiplications to become compute-bound. Nemotron 3.5 Lightning, designed for local agentic use as well as datacenters, deploys W4A16 on every GPU including Blackwell, with the stated aim “to minimize weight fetch” and keep activation precision. On MI350X, at TP4 with an FP8 Mistral Large 4 re-quantized to MXFP4 experts at load, an FP8 KV cache and DSpark, vLLM measured W4A8 and W4A4 within 3% in tokens per second.

W4A8 keeps FP8 activations. SGLang serves DeepSeek-V4’s FP4 experts this way by default on Blackwell; on Hopper its default is W4A16, and FlashInfer’s Humming kernels offer W4A8 behind a flag. Against W8A8 it saves the weight bytes and keeps the same peak: its report says “the peak FLOPs for FP4 × FP8 operations are currently the same as FP8 × FP8 on existing hardware”, adding they could be 1/3 more efficient on future hardware. Against W4A16 it has the same bytes and FP8’s arithmetic. The multiplication runs on FP8 numbers, so the weights are converted to FP8 first, which DeepSeek reports as lossless as long as the FP4 scales inside a 128 × 128 FP8 tile don’t differ by too much. NVFP4 is harder, and NVIDIA chose not to ship W4A8 for Nemotron 3 Ultra on Hopper: NVFP4’s elements times its E4M3 scales cover more range than FP8 and saturate on a direct cast, the detour through BF16 adds an operation before each multiplication, and W8A8 was already slower than W4A16 there.

An activation has no stored scale, because its values appear at run time. Its scale is either computed on each forward pass (dynamic) or calibrated beforehand on sample text and shipped with the model (static). Nemotron 3 Ultra does both: dynamic block scales for its NVFP4 experts, static per-tensor scales for its FP8 layers. A dynamic scale takes some work in the kernel, which Blackwell does in hardware for NVFP4. A static one can saturate on traffic unlike its calibration set.

NVIDIA’s Figure 10 compares the NVFP4 and BF16 checkpoints of Nemotron 3.5 Super VL on an agentic workload (20 turns of 2K in, 1K out, after a 50K prefix, with prefix reuse on):

Figure 5.NVFP4 against BF16 weights, Nemotron 3.5 Super VL, vLLM. Hopper runs W4A16 and GB200 runs W4A4, so each panel pairs a recipe with its GPU. The drafter setting and GPU count are not stated.

The ratio of the two throughput gains is 1.3 (2.43/1.862.43/1.86), and it can’t be read as the value of 4-bit arithmetic: the two bars differ in GPU generation, memory and kernel, and H100 80 GB, with less memory, is likely where freeing weight memory helps more (our inference). The report doesn’t say why the minimum-latency gains are smaller. The weight bytes of one token fall less than a full read’s (computed above), and a long context reads its KV cache too. A claim such as “4-bit is 2.4× faster on Blackwell and barely faster on Hopper” takes its first number from the throughput pair and its second from the minimum-latency pair.

Fitting the model matters as much. For Nemotron 3 Ultra on an 8-GPU H100 node, NVIDIA compared a candidate FP8 checkpoint, which would have left about 10 GiB per GPU for caches, with the NVFP4 checkpoint it released, which leaves about 40 GiB. The FP8 budget capped the batch, and the report found W4A16 on or above W8A8 across the range it cared about.

Methods that place the loss

The methods in this section are post-training quantization (PTQ): they convert a finished model and don’t train it further. (This is unrelated to the post-training stage of a recipe, where quantization-aware training, QAT, runs.) Calibration means running a small sample of text through the model to measure its activations. GPTQ and AWQ quantize weights only, so they produce W4A16 models.

The baseline rounds every weight to its nearest grid point, which minimizes the error of the weights. The output error of a layer is a different quantity. With E=W−W^E = W - \hat W and Σ\Sigma the second moment of the input, the expected squared output error is tr⁡(EΣE⊤)\operatorname{tr}(E\Sigma E^\top) (derived). For fixed scales, if Σ\Sigma were proportional to the identity, rounding to nearest would be the best one could do. The methods below use parts of Σ\Sigma: AWQ and SmoothQuant a per-channel size (mean and maximum activation) that stands in for its diagonal, GPTQ the whole matrix.

Results below are often perplexity, the exponential of the average negative log-likelihood per token: lower is better.

Round-to-nearest with group scales

AWQ’s authors call round-to-nearest “quite strong when using a small group size like 128”. It needs no calibration data, so a comparison of methods should include it.

GPTQ

GPTQ (Frantar et al., 2022) minimizes the error in a layer’s output on a calibration set: min⁡W^∥WX−W^X∥2\min_{\hat W} \|WX - \hat WX\|^2. It quantizes one column of weights at a time, pushing each rounding error into the columns not yet fixed through XX⊤XX^\top, layer by layer. On OPT-175B, 4-bit GPTQ gives a perplexity of 8.37 against 8.34 at 16 bits and 10.54 for plain rounding (no block scale in either; the paper reports grouped runs separately, and group 128 improves GPTQ by about 0.3). It took 4.2 hours on one A100.

AWQ

A channel is a column of the weight matrix, one input dimension. Since WX=(WD)(D−1X)WX = (WD)(D^{-1}X) for any invertible DD, weights and activations can be rewritten before rounding without changing the output. AWQ, SmoothQuant and rotations each choose a different DD.

The authors of AWQ (Lin et al., 2023) observe that a few channels carry the output. They scale a salient channel up by cc before rounding and its activations down by cc. If the channel doesn’t become its group’s largest, the step stays about the same and its error falls by about cc on a uniform grid (on E2M1 the steps grow with the value, so this holds only while the scaled weights stay below 2 on the grid). On OPT-6.7B at INT3, keeping just 1% of weights in FP16, chosen by activation size, gives a perplexity of 11.39 against 23.54 for rounding (10.86 unquantized). Other channels lose accuracy if the step grows. AWQ sets its scales from each channel’s mean activation and one fitted exponent, a far smaller statistic than the input covariance GPTQ uses, and in its authors’ own test of a mismatched calibration set (PubMed against Enron, OPT-6.7B) its perplexity worsened by 0.5 to 0.6 against 2.3 to 4.9 for GPTQ.

One corpus model calibrates AWQ on realistic traffic: Laguna XS.2 uses 128 long-context agentic trajectories. Poolside reports a “non-negligible quality drop” until it moved the last 10 of the 40 layers to INT8.

SmoothQuant

SmoothQuant (Xiao et al., 2022) moves outliers from activations to weights, with a per-channel scale cj=max⁡∣Xj∣α/max⁡∣Wj∣1−αc_j = \max|X_j|^{\alpha} / \max|W_j|^{1-\alpha} and α\alpha near 0.5. It targets W8A8 INT8: on OPT-175B, average accuracy is 71.6% in FP16, 32.3% with per-tensor INT8, 71.2% with SmoothQuant and 71.4% for per-channel INT8, which INT8 kernels can’t run. At four bits, with rounded weights and no groups, QuaRot’s table (quoting Shao et al.) gives a perplexity of 83.12 on Llama-2-7B against a 5.47 baseline.

Rotations

An orthogonal change of basis spreads a large coordinate over all the others and leaves the vector’s length unchanged. A Hadamard matrix has entries of ±1\pm 1 up to a constant and orthogonal columns, so multiplying by it is a rotation, in O(dlog⁡d)O(d \log d) like an FFT.

  • value
  • what it comes back as
  • thin lines: the 4-bit grid
Toy row, 64 values; axis ends near 0.900.−0.9000.90coordinate 0: 0.0750 comes back as 0.0750coordinate 1: -0.0020 comes back as 0.0000coordinate 2: 0.0003 comes back as 0.0000coordinate 3: 0.0111 comes back as 0.0000coordinate 4: 0.0145 comes back as 0.0000coordinate 5: 0.9000 comes back as 0.9000coordinate 6: -0.0091 comes back as 0.0000coordinate 7: 0.0391 comes back as 0.0750coordinate 8: -0.0141 comes back as 0.0000coordinate 9: 0.0360 comes back as 0.0000coordinate 10: 0.0122 comes back as 0.0000coordinate 11: -0.0112 comes back as 0.0000coordinate 12: 0.0590 comes back as 0.0750coordinate 13: 0.0193 comes back as 0.0000coordinate 14: 0.0095 comes back as 0.0000coordinate 15: -0.0117 comes back as 0.0000coordinate 16: -0.0128 comes back as 0.0000coordinate 17: -0.0148 comes back as 0.0000coordinate 18: -0.0047 comes back as 0.0000coordinate 19: 0.0129 comes back as 0.0000coordinate 20: 0.0224 comes back as 0.0000coordinate 21: -0.0415 comes back as -0.0750coordinate 22: 0.0031 comes back as 0.0000coordinate 23: 0.0561 comes back as 0.0750coordinate 24: -0.0287 comes back as 0.0000coordinate 25: 0.0195 comes back as 0.0000coordinate 26: -0.0268 comes back as 0.0000coordinate 27: -0.0165 comes back as 0.0000coordinate 28: 0.0146 comes back as 0.0000coordinate 29: -0.0581 comes back as -0.0750coordinate 30: 0.0058 comes back as 0.0000coordinate 31: -0.0327 comes back as 0.0000coordinate 32: 0.0017 comes back as 0.0000coordinate 33: 0.0652 comes back as 0.0750coordinate 34: -0.0242 comes back as 0.0000coordinate 35: 0.0111 comes back as 0.0000coordinate 36: -0.0021 comes back as 0.0000coordinate 37: -0.0046 comes back as 0.0000coordinate 38: 0.0023 comes back as 0.0000coordinate 39: 0.0104 comes back as 0.0000coordinate 40: -0.0025 comes back as 0.0000coordinate 41: -0.0060 comes back as 0.0000coordinate 42: -0.0034 comes back as 0.0000coordinate 43: 0.0011 comes back as 0.0000coordinate 44: 0.0135 comes back as 0.0000coordinate 45: 0.0033 comes back as 0.0000coordinate 46: -0.0139 comes back as 0.0000coordinate 47: 0.0883 comes back as 0.0750coordinate 48: -0.0169 comes back as 0.0000coordinate 49: -0.0239 comes back as 0.0000coordinate 50: 0.0050 comes back as 0.0000coordinate 51: -0.0653 comes back as -0.0750coordinate 52: -0.0022 comes back as 0.0000coordinate 53: 0.0243 comes back as 0.0000coordinate 54: 0.0064 comes back as 0.0000coordinate 55: -0.0064 comes back as 0.0000coordinate 56: -0.0190 comes back as 0.0000coordinate 57: -0.0068 comes back as 0.0000coordinate 58: -0.0128 comes back as 0.0000coordinate 59: -0.0099 comes back as 0.0000coordinate 60: -0.0136 comes back as 0.0000coordinate 61: 0.0111 comes back as 0.0000coordinate 62: 0.0159 comes back as 0.0000coordinate 63: 0.0111 comes back as 0.0000Toy row, 64 values; axis ends near 0.900.−0.9000.90coordinate 0: 0.0750 comes back as 0.0750coordinate 1: -0.0020 comes back as 0.0000coordinate 2: 0.0003 comes back as 0.0000coordinate 3: 0.0111 comes back as 0.0000coordinate 4: 0.0145 comes back as 0.0000coordinate 5: 0.9000 comes back as 0.9000coordinate 6: -0.0091 comes back as 0.0000coordinate 7: 0.0391 comes back as 0.0750coordinate 8: -0.0141 comes back as 0.0000coordinate 9: 0.0360 comes back as 0.0000coordinate 10: 0.0122 comes back as 0.0000coordinate 11: -0.0112 comes back as 0.0000coordinate 12: 0.0590 comes back as 0.0750coordinate 13: 0.0193 comes back as 0.0000coordinate 14: 0.0095 comes back as 0.0000coordinate 15: -0.0117 comes back as 0.0000coordinate 16: -0.0128 comes back as 0.0000coordinate 17: -0.0148 comes back as 0.0000coordinate 18: -0.0047 comes back as 0.0000coordinate 19: 0.0129 comes back as 0.0000coordinate 20: 0.0224 comes back as 0.0000coordinate 21: -0.0415 comes back as -0.0750coordinate 22: 0.0031 comes back as 0.0000coordinate 23: 0.0561 comes back as 0.0750coordinate 24: -0.0287 comes back as 0.0000coordinate 25: 0.0195 comes back as 0.0000coordinate 26: -0.0268 comes back as 0.0000coordinate 27: -0.0165 comes back as 0.0000coordinate 28: 0.0146 comes back as 0.0000coordinate 29: -0.0581 comes back as -0.0750coordinate 30: 0.0058 comes back as 0.0000coordinate 31: -0.0327 comes back as 0.0000coordinate 32: 0.0017 comes back as 0.0000coordinate 33: 0.0652 comes back as 0.0750coordinate 34: -0.0242 comes back as 0.0000coordinate 35: 0.0111 comes back as 0.0000coordinate 36: -0.0021 comes back as 0.0000coordinate 37: -0.0046 comes back as 0.0000coordinate 38: 0.0023 comes back as 0.0000coordinate 39: 0.0104 comes back as 0.0000coordinate 40: -0.0025 comes back as 0.0000coordinate 41: -0.0060 comes back as 0.0000coordinate 42: -0.0034 comes back as 0.0000coordinate 43: 0.0011 comes back as 0.0000coordinate 44: 0.0135 comes back as 0.0000coordinate 45: 0.0033 comes back as 0.0000coordinate 46: -0.0139 comes back as 0.0000coordinate 47: 0.0883 comes back as 0.0750coordinate 48: -0.0169 comes back as 0.0000coordinate 49: -0.0239 comes back as 0.0000coordinate 50: 0.0050 comes back as 0.0000coordinate 51: -0.0653 comes back as -0.0750coordinate 52: -0.0022 comes back as 0.0000coordinate 53: 0.0243 comes back as 0.0000coordinate 54: 0.0064 comes back as 0.0000coordinate 55: -0.0064 comes back as 0.0000coordinate 56: -0.0190 comes back as 0.0000coordinate 57: -0.0068 comes back as 0.0000coordinate 58: -0.0128 comes back as 0.0000coordinate 59: -0.0099 comes back as 0.0000coordinate 60: -0.0136 comes back as 0.0000coordinate 61: 0.0111 comes back as 0.0000coordinate 62: 0.0159 comes back as 0.0000coordinate 63: 0.0111 comes back as 0.0000

Original, 64 values: RMSE of the 4-bit reconstruction 0.0154; 54 of 64 round to zero; length 0.9256.

64 values: RMSE 0.0154 before rotation, 0.0115 after. 8 values: RMSE 0.0146 before rotation, 0.0339 after.

Wx = (WQ)(QTx)

Figure 6.A toy before and after a random Hadamard rotation, with one 4-bit E2M1 scale. At 64 values the rotation lowers the error (RMSE 0.0154 to 0.0115). At 8 values it raises it (0.0146 to 0.0339), because every rotated value lands near the top of the grid where E2M1 is coarsest. Rotation keeps the length, and in a model it has to be undone or absorbed in the next layer.

QuaRot (Ashkboos et al., 2024) applies this to Llama-2. RMSNorm divides by a length that rotation doesn’t change, so the layer commutes with the rotation once its per-coordinate gain is folded into the adjacent weights. With weights rounded by GPTQ and activations and KV cache all in 4 bits, Llama-2-70B goes from perplexity 3.32 to 3.79, or 3.61 with groups of 128. At inference QuaRot adds about one and a half Hadamard transforms per layer. GLM-5 on Ascend names QuaRot, and Laguna uses a rotation before INT4. NVIDIA’s QAD report says “common PTQ algorithms often fail to improve over baseline NVFP4 performance because the small block size neutralizes traditional outlier mitigation techniques” (we did not check the paper it cites). This suggests the block scale already does part of what these methods do.

Mistral’s report says that for Large 4’s 606 GB NVFP4 checkpoint “naive abs-max quantization already suffices and matches the FP8 reference”, with FP8 as the baseline and 200M tokens of calibration data.

When PTQ alone loses too much, some labs train the quantized model. Quantization-aware distillation (QAD) fine-tunes a quantized student against the 16-bit teacher: on Llama Nemotron Super V1, AIME25 goes 46.0 (BF16), 32.3 (PTQ), 45.6 (QAD). Nemotron 3.5 Super’s released checkpoint is PTQ plus QAD, and Poolside recovered Laguna’s NVFP4 quality with it. O2 covers QAD and training in the format.

The KV cache as its own target

The KV cache holds activations for every token of every session, and at long context it, not the weights, limits how many sessions a GPU can hold. A 4-bit cache needs no 4-bit arithmetic: cached values are dequantized before attention uses them. DeepSeek’s V4.1 report says of its FP4 cache that it “reduces storage rather than accelerates matrix multiplication”. That reasoning doesn’t reach 8 bits, where attention often multiplies FP8 directly (Nemotron’s Table 12).

FP8 is the usual format, in several layouts. GLM-5.3 in vLLM stores (512+64)×2=1,152(512 + 64) \times 2 = 1{,}152 bytes per token per layer in BF16 and 512+16+64×2=656512 + 16 + 64 \times 2 = 656 in FP8 (02). That is 0.57 of the BF16 bytes rather than 0.50, because the 64 position (RoPE) values stay in BF16 and the format carries 16 bytes of scales.

Figure 7.Bytes per token per layer in five cache layouts, as implemented in vLLM. Row labels give each layout's value counts, which differ, so compare the segments. DeepSeek's V4.1 report specifies FP4 for this cache; vLLM's commands use the FP8 layout shown.

Scales vary as well. In vLLM --kv-cache-dtype fp8 without a calibrated checkpoint uses a scale of 1.0, which works because FP8’s range is wide; dataset calibration is the option its docs recommend. Kolibri-1 uses unit scales too. Laguna XS.2 is the one corpus model with calibrated per-tensor maximum scales, chosen “to maximize compatibility”.

With a smaller cache, one GPU holds more sessions and more cached prefixes. MiniMax-M3’s vLLM recipe gets “~1.5× the KV pool” with FP8, and Laguna’s FP8 cache “roughly doubles the number of concurrent trajectories a single replica can carry”. Idle prefixes occupy the same pool, so when memory pressure would otherwise evict them a smaller format can keep more of them resident and, in LMSYS’s words, “reduce repeated prefill work”. On an agentic benchmark (Qwen3.5-397B, eight GPUs, TP8), the FP8 cache’s hit rate fell much lower than NVFP4’s past 12 concurrent sessions. A KV saving can reappear as prefill arithmetic avoided, on traffic that repeats.

DeepSeek-V4.1-Flash goes further, storing its main KV cache in E2M1 with one E4M3 scale per 16 channels and training the model to it, reporting “only marginal performance degradation”. It drops NVFP4’s tensor scale, since elements and scales can reach 448×6=2688448 \times 6 = 2688 and the cached vector, RMS-normalized over 512 channels, is bounded near 512≈22.6\sqrt{512} \approx 22.6. It also quantizes the position values, after applying RoPE: the other order would give “only a marginal accuracy improvement” and add work to each decode step. vLLM’s published command lines for V4.1 still pass --kv-cache-dtype fp8. NVIDIA measured NVFP4 KV on Nemotron 3.5 and got a median recovery of 99.36% against 99.52% for FP8 KV on 35 benchmarks, with two different checkpoints, which the report warns reflects “the full weight-and-cache configuration, not KV quantization alone”. NVIDIA shipped the model with FP8 KV.

SGLang’s experiment keeps FP8 weights in both arms and changes only the KV format. On Qwen3.8-27B an NVFP4 cache lowered GSM8K by 4 of 1,311 answers (0.31 points), GPQA-Diamond by 4 of 396 (1.01), and SWE-bench Verified by 8 of 500 tasks (1.60), with AIME 2025 equal. On Qwen3.5-397B, GSM8K lost one answer and the rest matched. These are a handful of items per benchmark, enough to bound the loss without measuring it. “Two models and a small set of tasks do not establish a general relationship between model size and quantization tolerance,” the post says. Its per-tensor scale was left at 1.0, and the post says calibration may close the gap. SGLang calls the feature experimental, on Blackwell-class GPUs and two kinds of model.

Capacity is 1.78× before overheads. On one RTX PRO 6000 Blackwell with Qwen3.8-27B and the prefix cache off, peak-batch decode was 26 to 30% faster at equal session counts, as less cache has to be read, and raising the session count to fill the extra room lifted decode throughput by 37 to 78%, more at longer contexts. Mean time to first token rose 0.2 to 0.4% at equal concurrency, which the post attributes mainly to quantization and dequantization overhead.

Who quantizes, and when

Some labs train 4-bit experts in: Kimi K3 in MXFP4, DeepSeek-V4 with FP4 experts, gpt-oss in MXFP4. Some run PTQ themselves, as NVIDIA and Mistral do, and many publish FP8 checkpoints without saying how they were made. Third parties re-quantize: vLLM’s recipe says Inferact’s GLM-5.3-NVFP4 converts only the expert linears. A conversion between 4-bit formats can leave the weights exactly as they were: NVIDIA’s card for DeepSeek-V4.1-Flash reports that all weight blocks converted from MXFP4 to NVFP4 losslessly, and it still measured small differences on six benchmarks, moving in both directions (GPQA Diamond 91.035 to 90.783, SciCode 54.401 to 53.994), while the activations moved to W4A4 and the Engram tables were converted lossily from FP8 to MXFP4 (about 12% relative error), and the card doesn’t separate those effects. “Lossless weight conversion does not imply identical inference outputs,” it says.

Online quantization makes FP8 from a BF16 checkpoint at load time, with no calibration data, as --quantization fp8_per_channel does for MiniMax-M3 “with gsm8k unchanged”, one benchmark. Load-time re-quantization converts an existing file: VLLM_FP8_MOE_REQUANT=mxfp4 turns Mistral Large 4’s FP8 experts into MXFP4 at load, layer by layer to keep peak memory low, and with those experts one 8-GPU AMD node serves two four-GPU replicas. vLLM’s recipe reports 22 to 71% more output tokens per second for MXFP4 experts over FP8 at TP8 on MI350, at 1 to 64 concurrent requests. The conversion rounds already-rounded FP8 values a second time, and we found no accuracy measurement for that path.

Local serving uses GGUF, on CPUs and laptops, where the Tensor Core argument above doesn’t apply. Its k-quants are integer formats with super-blocks of 256 values and 6-bit block scales, and Q4_K_M is a mix: under llama.cpp’s rules half the layers’ value projections and FFN down-projections and the output head go to Q6_K, 6.56 bits, so on a Llama-3.1-8B shape 19% of parameters are at 6 bits and the average is 4.89 (computed; the README measures 4.8944). A model trained in INT4 is close to lossless at Q4 only if the grids match. Kimi’s weights sit on −7-7 to 77, and llama.cpp’s Q4_0 assumes −8-8. Unsloth reports that using max/−7-7 instead of max/−8-8 as Q4_0’s scale on the MoE layers takes absolute error from 1.8% to about zero.

Measuring the cost

The most-reported statistic is median recovery. Nemotron 3.5 Super VL’s 99.62% is a median over a table on which individual rows run from 93.6% to 111.5%:

37 rows in Table 23; the report says 36. The report's median is 99.62%; over all 37 rows it is 99.70%.

9498102106110100%reportedmedian 99.62%SciCode (problem): BF16 15.10, NVFP4 14.13, recovery 93.58%SciCode (problem) 93.6%Tau3 Banking: BF16 18.88, NVFP4 17.78, recovery 94.17%Tau3 Banking 94.2%MMLU-ProX: BF16 76.91, NVFP4 73.76, recovery 95.90%ScreenSpot Pro: BF16 43.53, NVFP4 42.25, recovery 97.06%NV Chat Benchmark: BF16 52.67, NVFP4 51.93, recovery 98.60%AA-LCR: BF16 68.62, NVFP4 67.69, recovery 98.64%AA-Omniscience Accuracy: BF16 26.20, NVFP4 25.86, recovery 98.70%CharXiv (RQ): BF16 80.82, NVFP4 79.88, recovery 98.84%SciCode (subtask): BF16 47.13, NVFP4 46.61, recovery 98.90%HLE (no tools, text-only): BF16 31.79, NVFP4 31.44, recovery 98.90%SWE-Bench Multilingual: BF16 67.72, NVFP4 67.00, recovery 98.94%Tau2-AA (Airline): BF16 76.50, NVFP4 75.75, recovery 99.02%Video-MME-v2 (w/o sub.): BF16 32.85, NVFP4 32.54, recovery 99.06%SWE-Bench Pro: BF16 51.78, NVFP4 51.30, recovery 99.07%LiveCodeBench v6: BF16 88.22, NVFP4 87.41, recovery 99.08%WMT24++ (en→xx): BF16 52.32, NVFP4 51.94, recovery 99.27%HLE (full): BF16 29.20, NVFP4 29.00, recovery 99.32%MathVision: BF16 78.16, NVFP4 77.79, recovery 99.53%Baby Vision: BF16 29.60, NVFP4 29.51, recovery 99.70%OCRBenchV2 ZH: BF16 59.15, NVFP4 59.03, recovery 99.80%WMT24++ COMET: BF16 84.80, NVFP4 84.76, recovery 99.95%SWE-Bench Verified: BF16 71.10, NVFP4 71.07, recovery 99.96%Tau2 Telecom: BF16 73.19, NVFP4 73.19, recovery 100.00%MMLongBench-Doc: BF16 62.82, NVFP4 62.82, recovery 100.00%OCR Reasoning: BF16 65.08, NVFP4 65.09, recovery 100.02%Tau2 Retail: BF16 69.24, NVFP4 69.30, recovery 100.09%OCRBenchV2 EN: BF16 68.12, NVFP4 68.34, recovery 100.32%V*: BF16 85.60, NVFP4 85.95, recovery 100.41%GPQA Diamond (no tools): BF16 89.52, NVFP4 89.90, recovery 100.42%GraphWalks (≤1M): BF16 89.71, NVFP4 90.34, recovery 100.70%IFBench: BF16 70.86, NVFP4 71.50, recovery 100.90%APEX Shortlist (no tools): BF16 88.36, NVFP4 89.36, recovery 101.13%BIRD (text2sql, no tools): BF16 59.58, NVFP4 60.60, recovery 101.71%AA-Omni Non-Hallucination: BF16 69.96, NVFP4 72.33, recovery 103.39%Terminal-Bench v2.1: BF16 60.67, NVFP4 62.78, recovery 103.48%CritPt: BF16 10.86, NVFP4 11.71, recovery 107.83%CritPt 107.8%GDPVal: BF16 49.98, NVFP4 55.74, recovery 111.52%GDPVal 111.5%recovery, NVFP4 / BF16, percent (linear axis, not from 0)949698100102104106108110112100%median 99.62% (reported)SciCode (problem): BF16 15.10, NVFP4 14.13, recovery 93.58%SciCode (problem) 93.6%Tau3 Banking: BF16 18.88, NVFP4 17.78, recovery 94.17%Tau3 Banking 94.2%MMLU-ProX: BF16 76.91, NVFP4 73.76, recovery 95.90%ScreenSpot Pro: BF16 43.53, NVFP4 42.25, recovery 97.06%NV Chat Benchmark: BF16 52.67, NVFP4 51.93, recovery 98.60%AA-LCR: BF16 68.62, NVFP4 67.69, recovery 98.64%AA-Omniscience Accuracy: BF16 26.20, NVFP4 25.86, recovery 98.70%CharXiv (RQ): BF16 80.82, NVFP4 79.88, recovery 98.84%SciCode (subtask): BF16 47.13, NVFP4 46.61, recovery 98.90%HLE (no tools, text-only): BF16 31.79, NVFP4 31.44, recovery 98.90%SWE-Bench Multilingual: BF16 67.72, NVFP4 67.00, recovery 98.94%Tau2-AA (Airline): BF16 76.50, NVFP4 75.75, recovery 99.02%Video-MME-v2 (w/o sub.): BF16 32.85, NVFP4 32.54, recovery 99.06%SWE-Bench Pro: BF16 51.78, NVFP4 51.30, recovery 99.07%LiveCodeBench v6: BF16 88.22, NVFP4 87.41, recovery 99.08%WMT24++ (en→xx): BF16 52.32, NVFP4 51.94, recovery 99.27%HLE (full): BF16 29.20, NVFP4 29.00, recovery 99.32%MathVision: BF16 78.16, NVFP4 77.79, recovery 99.53%Baby Vision: BF16 29.60, NVFP4 29.51, recovery 99.70%OCRBenchV2 ZH: BF16 59.15, NVFP4 59.03, recovery 99.80%WMT24++ COMET: BF16 84.80, NVFP4 84.76, recovery 99.95%SWE-Bench Verified: BF16 71.10, NVFP4 71.07, recovery 99.96%Tau2 Telecom: BF16 73.19, NVFP4 73.19, recovery 100.00%MMLongBench-Doc: BF16 62.82, NVFP4 62.82, recovery 100.00%OCR Reasoning: BF16 65.08, NVFP4 65.09, recovery 100.02%Tau2 Retail: BF16 69.24, NVFP4 69.30, recovery 100.09%OCRBenchV2 EN: BF16 68.12, NVFP4 68.34, recovery 100.32%V*: BF16 85.60, NVFP4 85.95, recovery 100.41%GPQA Diamond (no tools): BF16 89.52, NVFP4 89.90, recovery 100.42%GraphWalks (≤1M): BF16 89.71, NVFP4 90.34, recovery 100.70%IFBench: BF16 70.86, NVFP4 71.50, recovery 100.90%APEX Shortlist (no tools): BF16 88.36, NVFP4 89.36, recovery 101.13%BIRD (text2sql, no tools): BF16 59.58, NVFP4 60.60, recovery 101.71%AA-Omni Non-Hallucination: BF16 69.96, NVFP4 72.33, recovery 103.39%Terminal-Bench v2.1: BF16 60.67, NVFP4 62.78, recovery 103.48%CritPt: BF16 10.86, NVFP4 11.71, recovery 107.83%CritPt 107.8%GDPVal: BF16 49.98, NVFP4 55.74, recovery 111.52%GDPVal 111.5%recovery, NVFP4 / BF16, percent (linear axis, not from 0)

Hover or focus a dot for the benchmark and its two scores.

Figure 8.Accuracy recovery of the released checkpoint (PTQ plus QAD) against BF16 weights with an FP8 KV cache, one dot per benchmark. The report's median is 99.62%. Recoveries above 100% show the spread that noise and side effects alone can produce.

NVIDIA’s acceptance rule was 99% median recovery and at most 5% more output tokens. A token count shows runaway generation, which a score can miss. The checkpoint came in at 0.44% more tokens, excluding non-completing Terminal-Bench rollouts; with them included it reaches 2.99%. Median recovery also has a scope: 99.62% is over 36 benchmarks, and the 99.52% quoted above is the FP8-KV figure over 35.

Perplexity can differ from usefulness on a task. On Qwen3.5-397B, vLLM’s AMD notes give 3.43 for MXFP4 W4A4 and 3.39 for GPTQ-Int4 W4A16 on the same wikitext-2 tokens, with no task accuracy reported.

The same model name is served at different precisions. Artificial Analysis’s GLM-5.2 providers page, in November 2026, lists Novita, SiliconFlow and GMI at FP8, DeepInfra and Nebius at FP4, Parasail at NVFP4, and 12 of 18 endpoints with no precision tag. Its Endpoint Accuracy Index highlights output-token limits and tool-call handling, so an endpoint can lose for reasons other than quantization. Moonshot’s Kimi Vendor Verifier checks third-party deployments and targets “KV cache bugs and quantization degradation that short benchmarks hide”. The umans team put the difference this way: “A one-shot benchmark samples the noise once and averages it away. An agent is an autoregressive chain.” They attribute their failure to a third-party PTQ, which is not a controlled result.

No LLM lab in this corpus other than NVIDIA reports a numeric recovery of PTQ against its own unquantized model, so most quantized releases can be checked only by running them.

The choices side by side

“Expert weights” ratios are against BF16 experts. Whole-checkpoint ratios are smaller (Nemotron’s 4.82 BPE is 0.30).

ChoiceBytes, against BF16Peak arithmeticRuns onLimit or loss
W8A8 (FP8)expert weights × 0.52× BF16 at peak (NVIDIA: FP8 on Hopper, MXFP8 on Blackwell)Hopper, Blackwell, AMDlittle reported
W4A16expert weights × 0.26–0.28as BF16any GPU with a kernelweights only, no activation grid; slower than W8A8 at prefill
W4A8expert weights × 0.26–0.28as W8A8 at peak (DeepSeek)Blackwell; Hopper with FlashInfer; AMD; AscendNVFP4 weights saturate when cast to FP8
W4A4expert weights × 0.26–0.284× on GB200, 6× on GB300 (FP8’s 2× times FP4’s 2× and 3×; NVIDIA)Blackwell; AMD CDNA4activations on a 4-bit grid; some layers excluded; needs compute-bound batches
FP8 KVcache × 0.57 on GLM-5.3’s layout2× where attention multiplies FP8most kernels (GLM-5.3-Flash on Hopper needs BF16)small; sliding-window layers “more sensitive” (vLLM)
NVFP4 or FP4 KVcache × 0.31 (352 of 1,152 B)nonefew kernels, Blackwell-class0 to 1.6 points in SGLang’s one study

Where the models in the map sit

From reports and configs. A blank means searched and not found. “Not converted” means listed among the modules skipped by the checkpoint’s quantization config, which doesn’t state their precision.

ModelExpertsExpert activationsAttentionOutput headKV cacheMade by
Nemotron 3.5 Super VLNVFP4FP4 on Blackwell, BF16 elsewhereFP8FP8FP8 (NVFP4 evaluated)lab, PTQ + QAD
Nemotron 3.5 LightningNVFP4BF16BF16NVFP4FP8lab, PTQ + QAD
Nemotron 3 Ultrarouted NVFP4, shared FP8FP4 on Blackwell, BF16 on HopperBF16BF16FP8lab, PTQ
DeepSeek-V4FP4FP8 (Blackwell default)FP8BF16 RoPE, FP8 restlab, QAT
DeepSeek-V4.1-FlashMXFP4FP8MXFP8BF16FP4 main (report); FP8 in vLLM’s commandslab, QAT
Kimi K3MXFP4MXFP8higher precisionFP8 in recipeslab, QAT
GLM-5.3 (default)FP8FP8not convertedFP8lab, FP8 file
GLM-5.3-NVFP4NVFP4 (routed only)BF16FP8Inferact
Mistral Large 4 NVFP4NVFP4as trainedFP8lab, PTQ
MiniMax M3.1-FlashNVFP4MXFP8not convertedlab, QAT
gpt-oss-120bMXFP4BF16 recommended; MXFP8 on the Blackwell pathBF16BF16FP8 optionlab, post-trained
Kolibri-1FP8FP8FP8BF16FP8, unit scaleslab, QAT in RL
Laguna XS.2 (INT4)INT4 first 30 layers, INT8 last 10BF16FP8, per-tensor maximumlab, PTQ with AWQ

Easy to get wrong

  • “Four-bit” doesn’t describe a whole checkpoint. Nemotron’s 4.82 BPE is an average, every recipe holds some layers at more than four bits, and Q4_K_M has about 19% of parameters at 6 bits.
  • MXFP4 and NVFP4 are two formats. Both use E2M1 elements; they differ in block size, scale format and the existence of a tensor scale, and MXFP4’s scale rule varies by implementation.
  • A speed-up figure needs its GPU, operating point and baseline. 1.86× and 2.43× are at a throughput target against BF16, 1.13× and 1.30× at minimum latency, and FP8 is the baseline that matters to most deployers.
  • A decode saving isn’t the storage ratio. W4 cuts expert bytes to a quarter, but a step reads other weights too, and at batch 1 the released recipe cuts weight bytes by 2.5×.
  • Median recovery doesn’t isolate PTQ. Nemotron’s 99.62% is PTQ followed by distillation. On the 13 benchmarks of Table 16, with an NVFP4 KV cache, PTQ alone on the same recipe reached 98.98%, and PTQ plus 200 QAD iterations 100.12%.
  • A native 4-bit release isn’t automatically lossless in GGUF. The block scale has to land on the training grid, and only the MoE tensors are INT4.

Check yourself

Under FP4 with one scale per tensor, a tensor’s largest value is 3.0. Which values round to zero?

ss maps 3.0 to 6, so s=0.5s = 0.5. Elements of magnitude below 0.25 on the grid round to zero, and in real units that is 0.25×0.5=0.1250.25 \times 0.5 = 0.125. (A value of exactly 0.125 ties, and ties go to zero.) With a block scale, ss differs per block, so blocks of small values keep them.

Why does a 4-bit KV cache need no 4-bit arithmetic units, when W4A4 does?

A KV cache is read and dequantized before use; the attention multiplication can run on 8- or 16-bit numbers. W4A4 gains its speed only if the matrix unit consumes the 4-bit operands directly.

A checkpoint runs as NVFP4 W4A16 on an H100. What do the removed bits change, and what does it lose?

The weight bits save memory and weight bytes read. Arithmetic stays BF16, so compute-bound phases run at BF16’s speed, slower than W8A8, and the activations are unchanged and lose nothing. The weights lose precision.

A card reports 99.6% median recovery. What can’t you read from that number?

The spread across benchmarks (Nemotron’s rows run from 93.6% to 111.5%), and how many more tokens the model spends; a runaway loop raises tokens and can leave a score unchanged. The baseline also matters: is it the 16-bit model, FP8, or an earlier checkpoint?

In the map

Methods

Taxonomy nodes

Further reading

  • OCP Microscaling Formats specification v1.0 (2023) and Rouhani et al., arXiv:2310.10537: the MX formats and the default scale rule.
  • NVIDIA, “Introducing NVFP4” (Alvarez et al., 2025): the format and Blackwell’s support.
  • Frantar et al., GPTQ (arXiv:2210.17323); Lin et al., AWQ (arXiv:2306.00978); Xiao et al., SmoothQuant (arXiv:2211.10438); Ashkboos et al., QuaRot (arXiv:2404.00456): the PTQ methods.
  • Nemotron 3.5 Super and Lightning technical report, §4 and Appendix B: the most detailed public account of a per-module quantization study.
  • LMSYS, “Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache” (2026-09-16): the KV experiment with FP8 weights on both sides.
  • umans.ai, “GLM-5.2 NVFP4: fast, cheap, and not worth serving” (July 2026): a deployer’s account of a benchmark-passing failure.
  • Nagel et al., “A White Paper on Neural Network Quantization” (arXiv:2106.08295): definitions and the classical recipe.
↑