Open Models Almanac

Optimization · O2

Training precision: when the model learns about its deployment format

Low-precision arithmetic in training buys speed, and training against the serving format buys accuracy at serving time; the earlier quantization-aware training starts, the longer the model adapts and the more of the recipe runs on a quantized model

Illustration of two trees. On the left a small, well-shaped tree grows in a shallow teal tray, its roots spread to fit it. Beside it a much taller tree with its root ball wrapped in coral cloth stands on a low wheeled board, and an empty tray of the same kind sits next to it on the board, with a pair of gold pruning shears resting on its rim.

Kimi K3 has 2.78 trillion parameters, and 2.72 trillion of them sit in its routed experts, the many feed-forward blocks of which a router picks a few for each token (computed from config.json: 92 MoE layers × 896 experts × three 3,584 × 3,072 matrices). Stored in BF16, two bytes per parameter, the model would take 5.56 TB. Moonshot ships the experts in MXFP4, a 4-bit format, and with everything else left in BF16 the checkpoint comes to about 1.56 TB (computed). Its report says Moonshot applies “quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations”, so for all of fine-tuning and reinforcement learning the network ran with its experts on the 4-bit grid it would be served on.

NVIDIA pre-trained Nemotron 3 Ultra on 20T tokens with the linear layers of most blocks computing in NVFP4, also a 4-bit format, but made the NVFP4 checkpoint it released after training, by rounding a BF16 checkpoint: “We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4”. Computing in a format during training and adapting the model to the format it will be served in are two different jobs, and labs do each at different stages.

This article separates the two: what low precision does to a training step, the three pre-training regimes the labs report, quantization-aware training and when each lab starts it, and the methods that leave quantization to the end.

Where the evidence comes from

Training precision is not visible in a model’s files. A config.json records the format the released weights are stored in, which says nothing about how they were trained, so every training fact here comes from a lab’s technical report, then its model card or blog. We count labs, not models. This site tracks 16 labs with language models: Aleph Alpha, Alibaba (Qwen), DeepSeek, Google DeepMind, Liquid AI, MiniMax, Mistral, Moonshot, NVIDIA, OpenAI, Poolside, StepFun, Tencent, Thinking Machines, Xiaomi and Z.ai. Of these, 5 state the number format of their pre-training run, 7 call something they did quantization-aware training, and 2 describe quantization-aware distillation. NVIDIA’s model cards also call its NVFP4 pre-training “quantization-aware”; we count that as low-precision training, not QAT. Where a lab says nothing, the cell is blank, which does not mean the lab chose against the method.

The background you need

I1 defines the formats; here is the one-line version of each. A float’s name gives its exponent bits (E) and mantissa bits (M, the stored significant digits). BF16 has FP32’s 8-bit exponent and 7 stored mantissa bits: the same range as FP32, far fewer digits. FP8 comes as E4M3 (largest value 448) and E5M2 (wider range, fewer digits). FP4 means E2M1, which has 15 distinct values: 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4 and ±6. A 4-bit grid is useless without a scale, so MXFP4 gives each block of 32 values one power-of-two scale (4.25 bits per value) and NVFP4 gives each block of 16 an FP8 scale plus one FP32 scale per tensor (4.5 bits per value). INT4 here is the symmetric integers −7 to 7 times a group scale.

A training step multiplies matrices three times per linear layer. Store the weight WW as (outputs × inputs), as PyTorch does. With input activations XX (tokens × inputs), the forward pass computes Y=XW⊤Y = XW^\top (NVIDIA calls this product Fprop). The backward pass receives ∂L/∂Y\partial L/\partial Y and computes the gradient for earlier layers, (∂L/∂Y) W(\partial L/\partial Y)\,W (Dgrad), and the gradient for the weights, (∂L/∂Y)⊤X(\partial L/\partial Y)^\top X (Wgrad). Each is a GEMM, a general matrix multiply. Training “in FP8” can mean any subset of the three, and the inputs of each can be rounded differently.

Then there is what training stores per parameter between steps. With the Adam optimizer, the usual mixed-precision setup keeps a 16-bit copy of the weights for the matrix multiplications (2 bytes), a 16-bit gradient (2), a 32-bit “master” copy of the weights (4) and Adam’s two running averages in 32 bits (4 + 4). That is 16 bytes per parameter, the figure the ZeRO paper uses. Activations come on top and grow with batch size and sequence length rather than with parameters.

Why a 16-bit weight can’t learn on its own

Suppose a weight sits at exactly 1.0 and the optimizer asks to move it by Δw=10−4\Delta w = 10^{-4}. In BF16, the representable numbers just above 1.0 are 2−7≈0.00782^{-7} \approx 0.0078 apart. Round-to-nearest sends 1.00011.0001 back to 1.01.0, because a value returns to its starting point whenever

∣Δw∣<12 spacing(w).|\Delta w| < \tfrac{1}{2}\,\mathrm{spacing}(w).

For a BF16 weight in [1,2)[1, 2) the spacing is 2−72^{-7} and the threshold 2−82^{-8}: at w=1w = 1 any upward update under 1/256 of the weight is lost, and near w=2w = 2 any update under 1/512 (computed). Micikevicius et al. (2017), who introduced mixed-precision training for FP16, give the same condition with FP16’s ten mantissa bits: an update is lost when the weight is “at least 2048 times larger” than it. For BF16 the ratio is 256, eight times smaller. Adam’s step is roughly the learning rate, so a weight near 1, such as a normalization gain, loses every step at a learning rate of 3×10−43 \times 10^{-4}. Matrix weights start near 0.02, where such a step survives, but they lose every step below a learning rate of about 6×10−56 \times 10^{-5} (computed): the end of a decaying schedule, and the rates used for fine-tuning a finished model. Nothing reports the lost updates. On a Mandarin speech model, Micikevicius et al. found that “updating FP16 weights results in 80% relative accuracy loss” compared with updating an FP32 master copy.

  • exact sum w0 + n·Δw
  • FP32 master copy
  • its BF16 copy (forward pass)
  • BF16, stochastic rounding
  • ±1 sd of stochastic rounding
  • BF16, round to nearest
1.01.52.005,00010,000optimizer step (weight on a linear axis, not from 0)1.01.52.002,0004,0006,0008,00010,000optimizer step (weight on a linear axis, not from 0)

BF16 spacing at w = 1: 2⁻⁷ = 0.0078; half: 0.0039; your update Δw = 0.0001 is below it: every round-to-nearest step is lost.

After 10,000 steps: exact 2; FP32 master 2.000166 (BF16 copy 2); stochastic rounding 2.01563 (±1 sd band: 0.088); round to nearest 1.

Figure 1.One weight, 10,000 steps of the same small update. Kept in BF16 and rounded to nearest, the weight never moves, because each update is under half the spacing between BF16 numbers at 1.0. An FP32 master copy adds every update exactly and gives the forward pass a rounded copy. Stochastic rounding moves up with probability proportional to the update and ends near the right value, within the shaded band.

Labs use three remedies. The first is a master copy in FP32, whose spacing near 1.0 is 2−232^{-23}: the optimizer adds the update there, and each step rounds a fresh low-precision copy from it for the matrix multiplications. Micikevicius et al. made this the first of their three rules, with loss scaling (multiplying the loss so small FP16 gradients don’t underflow, which BF16’s wide exponent makes unnecessary) and accumulating products in FP32. The second is stochastic rounding (Gupta et al., 2015): round up with probability equal to the fraction of the gap covered, so that on average the rounded value equals the exact one (E[SR(x)]=x\mathbb{E}[\mathrm{SR}(x)] = x). Its cost is noise. For an update much smaller than the spacing, one stochastic step is far noisier than one round-to-nearest step, but its error grows like N\sqrt{N} over NN steps while round-to-nearest’s bias grows like NN. The third is to keep sums in FP32 wherever many small numbers are added, such as gradient accumulation and reductions across GPUs. A fourth, compensated (Kahan) summation, keeps a 16-bit correction term beside each 16-bit weight; Zamirai et al. (2020) found it matched 32-bit training on their models, and no lab in the map reports it.

Training “in FP8” or “in NVFP4” means the matrix multiplications run on a low-precision copy rounded from FP32 weights at every step, and quantization-aware training, as we will see, is the same structure with the serving format in place of the compute format.

What training in low precision saves, and what it doesn’t

The low-precision copy is a small part of what training stores. Under ZeRO’s mixed-precision Adam the BF16 copy is 2 of 16 bytes per parameter. NVIDIA’s NVFP4 recipe keeps gradients in FP32 as well as the master weights and optimizer states (“The main weights (stored by the optimizer), weight gradients … and optimizer states are also kept in FP32”). With FP32 gradients a BF16 copy makes 18 bytes per parameter and an NVFP4 copy 16.56, a cut of at most 8% (computed). Two things shrink even that. Trainers commonly shard the optimizer state across data-parallel GPUs (ZeRO; Mistral and Google name it), while below ZeRO’s third stage (Mistral uses the second) each GPU keeps a full compute copy, so per GPU the copy is a larger share. And frameworks often keep the BF16 copy beside the low-precision one, in which case nothing is saved. DeepSeek-V3 cut the state itself, tracking Adam’s moments in BF16 and keeping master weights and gradients in FP32: 13 bytes if an FP8 copy is kept (computed).

Low-precision compute saves arithmetic, activation memory and network traffic. NVIDIA’s NVFP4 paper lists FP4 tensor-core throughput at 2× FP8 on GB200 and 3× on GB300 (two NVIDIA Blackwell systems), and FP8 at twice BF16. The inputs that linear layers save for the backward pass halve in FP8 and shrink up to 3.6× in NVFP4 (computed, scales included); attention, normalization and the layers kept in high precision save theirs in BF16, so recomputation and offloading, as in Kimi K3, remain the main tools for activation memory. DeepSeek-V3 also sends tokens to their experts in FP8 and gathers the results in BF16. None of the sources reports an end-to-end training speedup for NVFP4. At serving time the low-precision copy is the whole checkpoint.

Figure 2.Bytes stored per parameter in training (top) and serving (bottom), summed over all GPUs. Every training recipe keeps an FP32 master copy and FP32 or BF16 optimizer state, so moving the compute copy from BF16 to NVFP4 cuts 18 bytes to 16.56 at best. With the optimizer state sharded across data-parallel GPUs, each GPU's share of the compute copy is larger. At serving the compute copy is the checkpoint. Computed from the byte counts the sources state; activations, which depend on batch and sequence length, are left out.

Training in BF16, and the sums every format keeps

Two labs in the map describe BF16 runs. Poolside trained Laguna M.1 and XS.2 with “BF16 mixed precision training throughout all stages with the master weights in FP32 and most computations in BF16”, with RMSNorm and RoPE partly in FP32; its later Laguna S 2.1, the model this site tracks, did its RL “in FP8 precision”. Aleph Alpha’s model card for Kolibri-1 says training used “bfloat16 parameters, float32 reductions”; it doesn’t mention a master copy, and the report itself is silent on pre-training precision.

Whatever the compute format, the sums across GPUs are where the reported failures sit. In data parallelism each GPU computes gradients on its own share of the batch and the gradients are summed across GPUs; in tensor parallelism each GPU holds a slice of a layer and the partial results are summed. NVIDIA’s Ultra diverged around 8T tokens after the output layer’s weight-gradient accumulation moved from FP32 to BF16, which lost the small contribution of its multi-token prediction heads; Poolside’s Laguna M.1 became unstable while its output layer’s input gradient was summed across tensor-parallel GPUs in BF16. Both moved the sum back to FP32, as S1 describes.

Training in FP8

FP8 training puts all three multiplications of a linear layer on FP8 tensor cores. DeepSeek-V3’s report is the most detailed public account, and later labs cite it. It runs “all three GEMMs associated with the Linear operator … in FP8”, which “theoretically doubles the computational speed compared with the original BF16 method” and lets activations be stored in FP8 for the backward pass. DeepSeek-V3 makes three choices.

It scales at a fine grain. One scale per tensor lets a single outlier set the grid for millions of values (I1 shows this), so DeepSeek-V3 scales activations in tiles of 1 × 128 (one token, 128 channels) and weights in blocks of 128 × 128. It computes each scale from the current tile, where the first FP8 recipes used “delayed” scaling from the largest values of earlier steps, which a sudden spike overflows. Fine scales also let it use E4M3 for gradients, where earlier recipes used E5M2.

It accumulates in FP32. Tensor cores add FP8 products into an accumulator, and on the H800 that accumulator keeps “around 14 bits”. On a product with 4,096 terms DeepSeek measured “a maximum relative error of nearly 2%”. The fix copies partial sums into FP32 registers every 128 terms.

It keeps sensitive parts out of FP8. “The embedding module, the output head, MoE gating modules, normalization operators, and attention operators” stay in BF16 or FP32, and master weights and gradients stay in FP32. DeepSeek validated the recipe at two smaller scales, about 1T tokens each, where the loss stayed within 0.25% of a BF16 run. The report shows no BF16 run of V3 itself.

Among the labs in the map, Xiaomi’s MiMo-V2-Flash report says it uses “an FP8 mixed-precision framework similar to DeepSeek-V3”, keeping attention output projections, the embedding and the output head in BF16 and the MoE router in FP32; the MiMo-V2.5 cards say “FP8 mixed precision”. The MiMo-V2.6 report states no pre-training precision. Mistral Large 4’s report says one sentence: “we use FP8 tensor cores for fine-grained low-precision training”, with no layer list or block size. DeepSeek-V4 inherits V3’s framework and calls it “the existing FP8 training framework” without saying in so many words that pre-training ran in FP8. It adds one detail on the gradient side: MoE gradients travel between GPUs in BF16 “in a stochastic rounding manner”, and each GPU sums what it receives in FP32, “to avoid accumulation errors introduced by low-precision adders”.

Training in NVFP4

Four bits is a much coarser grid than eight: E2M1’s top step, from 4 to 6, is half the value below it. NVIDIA’s NVFP4 pre-training paper reports that “training diverges when every linear layer is quantized to FP4”, and gives a recipe of four parts. The same recipe trained Nemotron 3 Super and Ultra. In every recipe here, attention’s own products (queries times keys, scores times values) stay in BF16.

  1. Keep “a few sensitive linear layers in higher precision (15% of the network, with the majority of high precision layers at the end of the network)”. Ultra kept “the final 15% of the network (16 layers)” of 108, plus Mamba output projections, latent projections, attention projections, the multi-token prediction layers and the embeddings. The family white paper gives BF16 for most of these and MXFP8 for the Mamba output projections, which in NVFP4 flushed up to 40% of values to zero on the smaller Nemotron 3 Nano.
  2. Apply a random Hadamard transform of size 16 to the inputs of Wgrad only. A Hadamard rotation spreads an outlier’s energy over its block (I1’s rotations section). Rotating both inputs of Wgrad along the token dimension they share leaves their exact product unchanged, so only the rounding sees the rotated values. The paper found “no measurable benefit for Fprop and Dgrad at smaller scales”.
  3. Scale weights in two-dimensional 16 × 16 blocks, activations and gradients in 1 × 16.
  4. Use stochastic rounding for gradients and round-to-nearest-even for weights and activations.

The third rule has no counterpart in inference quantization. A block’s scale has to run along the dimension a product sums over, so the hardware can take it outside each dot product. Fprop’s dot products run along WW‘s rows (over its input dimension), Dgrad’s down its columns (over its output dimension), so each needs its own quantized copy. With 1 × 16 strips the two copies disagree: in a 64 × 64 Gaussian matrix, 91% of entries round to a different value in the two layouts (computed). The backward pass then differentiates a different function from the one the forward pass computed, which the paper calls “effectively breaking the chain rule”. A square tile has the same scale whichever way it is read. It costs some precision, since one tile’s largest value sets the scale for 256 entries instead of 16: the toy’s rounding error grows by about 1.2 times (computed over five seeds). Activations and gradients meet the same mismatch, between Fprop and Wgrad and between Dgrad and Wgrad, and keep their finer strips because “training is less sensitive to inconsistencies in activation tensors than weight tensors”. DeepSeek-V3’s 128 × 128 weight blocks solve the same problem in FP8.

  • quantized value, negative to positive (one scale for both panels)
  • cell rounds differently in the two passes
  • faint lines: scale blocks

W quantized for Fprop (scales along rows)

sum

W quantized for Dgrad (scales along columns)

sum

cells that differ

cells that differ: 91.1% | RMS error vs W: 1.87e−3 (tiles / strips: 1.23×), over the 64 × 64 matrix; its top-left 32 × 32 corner is drawn

Figure 3.Fprop's dot products run along the weight matrix's rows, Dgrad's down its columns, and a block scale has to run along the summed dimension. With a scale per 1 × 16 strip, most entries land on a different grid point in the two passes (coral). A 16 × 16 tile has one scale whichever way it is read, so both passes use the same rounded matrix, at somewhat higher rounding error. Activations and gradients meet the same mismatch, which the paper finds training tolerates better. Toy: Gaussian weights, E2M1 values with an exact per-block scale.

The fourth rule follows from what backpropagation is. Given the forward pass, Dgrad and Wgrad are linear in the incoming gradient, so zero-mean noise from stochastic rounding of gradients averages out: the weight gradient stays unbiased. Round-to-nearest on gradients is biased instead. With a block scale of (block maximum)/6, every entry under 1/24 of the block’s largest rounds to zero every time. On weights or activations the noise passes through nonlinearities and the loss, where it no longer averages out, and applying stochastic rounding there “causes divergence” in a 1.2B test model.

NVIDIA’s 12B model trained on 10T tokens in NVFP4 reached 62.58% on MMLU-Pro against 62.62% for the same model trained in FP8, a difference within noise, while other benchmarks moved by up to about 4 points either way (code 56.7 against 59.5, which the paper suspects is “noisy evaluations”; GSM8k 92.3 against 89.1). Its validation loss stayed within 1% of FP8’s while the learning rate was constant, widening to “slightly above 1.5%” as it decayed. In an ablation, switching the forward pass to higher precision for the last 18% of tokens recovered most of the gap (1.5% to 0.5%), with about 6% of the run’s compute in higher precision; Ultra’s report doesn’t say it did this. On Ultra, NVIDIA branched BF16 runs from the NVFP4 run at 5T, 10T and 16T tokens and continued each for 74B tokens, “a proxy for high-precision training”. The relative train-loss gap averaged below 0.4%. A short BF16 continuation measures only the part of the gap that switching precision recovers; on the 12B model that was two thirds of it, so 0.4% is probably a lower bound (our inference). Ultra’s report has no downstream comparison of the two; the only downstream claim is for an 8B-active test model in the white paper, whose evaluations were “comparable”.

Ultra’s report describes two divergences, and neither was traced to NVFP4: one was the BF16 sum above, the other “undetermined”, and switching all tensors to BF16 “did not resolve the training divergence discussed in section 2.7”. NVIDIA then cut the run to 20T tokens. A new format still makes a stage that can’t be rerun harder to debug. NVIDIA is the only lab in the map that reports pre-training in a 4-bit format.

Why did Ultra still need PTQ? Its training grid was not its serving grid. Training used 16 × 16 weight tiles and kept attention and the last 15% of layers high; the release uses 1 × 16 blocks with calibrated scales, NVFP4 for the routed experts only and FP8 for shared experts and Mamba layers; and post-training ended in a BF16 checkpoint. A model trained in FP8 and shipped in FP8 with the same block scheme is the converse case: its weights were on the serving grid from the first step. DeepSeek’s FP8 configs (128 × 128, as V3 trained) fit that pattern, and Mistral’s FP8 release may (its report gives no block size); neither lab says the release reuses the trainer’s quantizer.

Quantization-aware training

The second job is making the model good in its serving format. Post-training quantization (PTQ) rounds a finished model, as I1 describes, often after calibration: running sample text through the model to choose the scales. Quantization-aware training (QAT) keeps training while the forward pass sees the rounded weights. Jacob et al. (2017) gave the recipe that most labs cite. The weights stay in floating point “so that they can be easily nudged by small amounts”, and each forward pass replaces them with Q(w)Q(w), their values on the serving grid, a step known as fake quantization or quantize-dequantize.

QQ is a staircase, whose derivative is zero almost everywhere. Backpropagation through it would give no gradient at all. The straight-through estimator (Bengio et al., 2013) treats the rounding as the identity in the backward pass:

∂Q(w)∂w:=1.\frac{\partial Q(w)}{\partial w} := 1 .

(With block scales set from the block’s maximum nothing is clipped, and the scale is treated as a constant.) What this computes is the gradient of the full-precision loss at the rounded weights, applied to the float weights: no variance, and a bias that shrinks with the grid spacing. Bengio et al. call it “clearly a biased estimator”. This is the master-copy structure from the start of the article. The difference from native low-precision training is in the backward pass. QAT rounds weights (and often activations) onto the serving grid for the forward pass and adds no rounding of gradients; Dgrad uses the rounded weight, and its arithmetic is whatever the trainer runs in, FP8 at DeepSeek. Native FP8 or NVFP4 training rounds the inputs of all three multiplications, to run faster. The QAD paper (Xin et al., 2026) says QAT “only quantizes the weights and activations, thus only helping forward pass”.

Figure 4.Which inputs of a linear layer's three products are rounded, by regime. Training in FP8 or NVFP4 rounds all of them to run faster, with stochastic rounding on NVFP4's gradients. QAT and QAD round the forward pass's weights (and often activations) onto the format the model will be served in; Dgrad and Wgrad reuse those rounded values, and the gradients stay unrounded. Master weights are FP32 where a source states it; Poolside's QAD trains BF16 weights.

To see where the straight-through gradient leads, take a quadratic loss 12(w−w∗)⊤S(w−w∗)\tfrac12 (w - w^\ast)^\top S (w - w^\ast) with SS positive definite. The straight-through gradient is S (Q(w)−w∗)S\,(Q(w) - w^\ast), which is zero only if the optimum w∗w^\ast happens to be on the grid. Otherwise the float weight never settles: it drifts to a rounding boundary and Q(w)Q(w) flips between neighbouring grid points, which Nagel et al. (2022) call oscillation. The time-average of Q(w)Q(w) approaches w∗w^\ast, but the network is served with one grid point, the one in use when training stops, and that need not be the best one.

  • float weight w (from w*)
  • rounded weight Q(w)
  • visits, last 1,000 steps
  • faint dashes: rounding boundaries; ellipses: equal loss
01234012w1 (grid units)w2PTQ: round w*best grid pointw*01234012w1 (grid units)w2PTQ: round w*best grid pointw*

w* = (2.6, 1.3); rounding w* (PTQ) gives (3.0, 1.5), excess 0.344; the best grid point is (3.0, 1.0), excess 0.034.

step 4,000: Q(w) = (3.0, 1.5), excess loss 0.344; float w = (2.511, 1.256); time-average of Q(w) over the last 1,000 steps (2.600, 1.300). Visits in the last 1,000 steps: (3.0, 1.5) ×600, (2.0, 1.0) ×400.

Figure 5.A toy QAT run on a two-weight least-squares problem with 4-bit (E2M1) weights. The gradient is computed at the rounded weight and applied to the float weight. It never reaches zero off the grid, so the float weight ends up circling a rounding boundary and the rounded weight alternates between two grid points, which average to the full-precision optimum (star). Neither is the best grid point. Toy numbers, not a lab's model.

A decaying learning rate shrinks the oscillation but not its rate (Nagel et al.: it “reduces the amplitude of the oscillations but do[es] not affect their frequency”), so the run ends on whichever neighbouring grid point is in use at the last step. On a larger toy, a least-squares problem with a 32-weight block in E2M1, with the block scale recomputed each step, that gave between 0.78 and 1.05 times PTQ’s excess loss with the learning rate decayed to zero, and 1.5 to 4.4 times with it constant (computed). A single block of a convex problem has nothing else to adapt; in a network, the other weights and layers shift to compensate for each one’s rounding, which is where QAT gains over PTQ. No lab says whether its learning rate decays during QAT.

The labs’ implementations differ in detail:

  • DeepSeek-V4’s experts: “the FP32 master weights maintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for computation”. The FP4-to-FP8 step is lossless for DeepSeek’s weights, which it checked, so QAT runs on the existing FP8 training framework “without any modification”, and gradients go “directly … back to the FP32 master weights, equivalent to applying the Straight-Through Estimator”.
  • Aleph Alpha’s trainer “applies the same FP8 rounding as the inference engine but computes in BF16, without FP8 kernels”, with the straight-through estimator in the backward pass.
  • Xiaomi applies quantize-dequantize to MiMo-V2.6’s experts “after each parameter update”, following “the numeric constraints of the MXFP4 Humming GEMM kernels” used by its rollout engine (the inference server that generates RL’s training samples), “so the two engines see identical expert weights”.
  • Z.ai built “a quantization kernel applicable to both training and offline weight quantization, which ensures bitwise-identical behavior between training and inference”.

The trainer’s rounding should be the engine’s, down to how the scales are chosen, or the model adapts to a slightly different grid. Exporting Q(w)Q(w) itself makes the weights match; activations, rounded on the fly by each side, are where they can drift. What QAT costs depends on the trainer. At DeepSeek it is almost nothing, since FP4 values are computed in its FP8 framework anyway; Aleph Alpha’s emulation gives up FP8 kernels in training; a BF16 trainer pays an extra rounding pass per weight and activation, which the SGLang RL team, reproducing Kimi K2-Thinking’s INT4 recipe, found “noticeably slower than BF16”.

When QAT starts

Five of the seven labs that describe QAT say when it begins: mid-training, SFT, RL, or, at DeepSeek, “post-training” without a stage.

LabModelStage (lab’s words)Format and what is quantized
XiaomiMiMo-V2.6“during mid-training”, FP32 master weights carried into RLMXFP4, experts
MoonshotKimi K3“from the SFT stage onward”, “covering both SFT and RL”MXFP4 expert weights, MXFP8 activations; attention, shared experts, routers higher
Z.aiGLM-5“in the SFT stage”INT4; tensors not stated
DeepSeekV4“during the post-training stage”, teacher and reference models includedMXFP4 expert weights and the sparse-attention indexer’s query–key path
DeepSeekV4.1-Flash“QAT during post-training”adds the main KV cache: E2M1 with one E4M3 scale per 16 channels
Aleph AlphaKolibri-1RL onlyFP8 (E4M3) weights in 128 × 128 blocks, activations, KV cache; embeddings, norms, router in BF16
GoogleGemma 4not statedQ4_0 blocks; a mobile mix of int2/int4 weights and int8 activations
MiniMaxM3.1-Flashnot stated“the routed MoE experts and the attention paths”

OpenAI says it “post-trained the models with quantization of the MoE weights to MXFP4” for gpt-oss, without the word quantization-aware or a mechanism, so it stays out of the count. Xiaomi’s “during mid-training” is the earliest start among the seven; the report’s reason is “allowing the model to adapt to low-precision computation while preserving model quality”. When RL begins, Xiaomi carries over “the FP32 master weights and Muown’s row state from the SFT checkpoint” (Muown is the Muon variant Xiaomi uses), “to stabilize MXFP4 training”, so QAT runs from mid-training through every later stage. DeepSeek quantizes its teacher and reference models too, the models whose outputs post-training compares the student with, so every model in the loop computes in the served format.

Labs apply QAT to the routed experts first: they hold nearly all the parameters (97.9% of Kimi K3’s) and tolerate rounding better. NVIDIA’s sensitivity analysis on Nemotron 3.5 found “MoE experts are comparatively robust” while the output layer “accounts for nearly half of the NVFP4 sensitivity”, though the same report blames 4-bit experts for “runaway loop generation in reasoning failures”. The KV cache, the keys and values a model keeps for every earlier token while it generates, is a target too, for storage: FP4 in V4.1 “nearly halves the storage footprint” against V4’s FP8 cache.

An earlier start gives the model more steps on the grid and lets every later stage, including RL, optimize the model that will actually be served. It also means every SFT and RL experiment runs on a quantized model, with whatever instability that brings, and a bug in the quantizer contaminates everything after it. We found no source that compares start points on one model. The nearest evidence comes from NVIDIA’s QAD paper, which ran QAT as a single extra stage after a finished multi-stage pipeline (SFT, RL, merging) and found that “NVFP4 QAT breaks the RL model’s capabilities”: on Nemotron 3 Nano and AceReason Nemotron 1.1 7B, QAT on SFT or RL-generated data scored below plain PTQ on every benchmark listed. That is QAT after RL, not during it, and the paper has no BF16 run on the same data, so part of the drop may come from more task-loss training after RL rather than from the 4-bit grid. Either reading suggests (ours) why the labs that use QAT put it inside their own post-training rather than after it.

  • quantization-aware training
  • trained in a format the lab also ships
  • low-precision compute, not shipped as trained
  • quantization after training (PTQ)
  • PTQ, then distillation (QAD)
  • QAT, stage not stated
premidSFTRLafterNVIDIA Nemotron 3 UltraNVIDIA Nemotron 3 Ultra, low-precision compute: pre-trained with the NVFP4 recipe: "NVFP4 pretraining" (Ultra report §2.2); precision of later stages not statedNVFP4 computeNVIDIA Nemotron 3 Ultra, PTQ: "We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4" (Ultra report §4)PTQ to NVFP4 mixXiaomi MiMo-V2.6Xiaomi MiMo-V2.6, QAT: "MXFP4 quantization-aware training (QAT) during mid-training" (§3.2); FP32 master weights carried into RL (§5.1); QDQ "after each parameter update" (§6.4)MXFP4 expertsMoonshot Kimi K3Moonshot Kimi K3, QAT: "QAT from the SFT stage onward" (§4.1.1); "covering both SFT and RL" (§4.1.4)MXFP4 experts, MXFP8 act.Z.ai GLM-5Z.ai GLM-5, QAT: "we apply INT4 QAT in the SFT stage" (§2.4.3); whether it continues into RL is not statedINT4Z.ai GLM-5, PTQ: PTQ with msModelSlim to W4A8 "to fit the 750B parameter GLM-5 model onto a single Atlas 800T A3 machine" (§5)W4A8 for AscendDeepSeek V4, V4.1DeepSeek V4, V4.1, QAT: "during the post-training stage", teacher and reference models included (V4 §5.2.1); V4.1 "QAT during post-training" for the main KV cache (V4.1 §2.4.4)post-training: MXFP4 experts; V4.1 FP4 KVAleph Alpha Kolibri-1Aleph Alpha Kolibri-1, QAT: "The trainer therefore uses FP8 quantisation-aware training" in RL (§3.2.1.3, App. I.4.1)FP8Poolside Laguna XS.2 (report)Poolside Laguna XS.2 (report), PTQ, then QAD: "pre-trained and fine-tuned using the BF16 datatype"; PTQ to FP8 and INT4; NVFP4 after "quantization-aware distillation (QAD)" (Laguna report §5)FP8, INT4; NVFP4 + QADMistral Large 4Mistral Large 4, trained in a shipped format: "we use FP8 tensor cores for fine-grained low-precision training" (§2.3); FP8 trainer and rollouts in RL (§4.4.2); mid-training and SFT precision not stated; ships an FP8 checkpointFP8Mistral Large 4, PTQ: "naive abs-max quantization" of the routed experts, "200M tokens from our SFT mixture for calibration" (§6)NVFP4 expertsNVIDIA Nemotron 3.5 SuperNVIDIA Nemotron 3.5 Super, PTQ, then QAD: "the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations" (3.5 report §4.1)NVFP4, 200 QAD iterationsGoogle Gemma 4Google Gemma 4, QAT, stage not stated: "quantized versions of our models trained with quantization-aware training" (Gemma 4 report §1, §2.5); no stage givenQAT, stage not statedMiniMax M3.1-FlashMiniMax M3.1-Flash, QAT, stage not stated: "the routed MoE experts and the attention paths are quantized during training" (M3.1-Flash model card); no stage givenQAT, stage not statedpre-trainingmid-trainingSFTRLafter trainingNVIDIANemotron 3 UltraNVIDIA Nemotron 3 Ultra, low-precision compute: pre-trained with the NVFP4 recipe: "NVFP4 pretraining" (Ultra report §2.2); precision of later stages not statedNVFP4 computeNVIDIA Nemotron 3 Ultra, PTQ: "We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4" (Ultra report §4)PTQ to NVFP4 mixXiaomiMiMo-V2.6Xiaomi MiMo-V2.6, QAT: "MXFP4 quantization-aware training (QAT) during mid-training" (§3.2); FP32 master weights carried into RL (§5.1); QDQ "after each parameter update" (§6.4)MXFP4 expertsMoonshotKimi K3Moonshot Kimi K3, QAT: "QAT from the SFT stage onward" (§4.1.1); "covering both SFT and RL" (§4.1.4)MXFP4 experts, MXFP8 act.Z.aiGLM-5Z.ai GLM-5, QAT: "we apply INT4 QAT in the SFT stage" (§2.4.3); whether it continues into RL is not statedINT4Z.ai GLM-5, PTQ: PTQ with msModelSlim to W4A8 "to fit the 750B parameter GLM-5 model onto a single Atlas 800T A3 machine" (§5)W4A8 for AscendDeepSeekV4, V4.1DeepSeek V4, V4.1, QAT: "during the post-training stage", teacher and reference models included (V4 §5.2.1); V4.1 "QAT during post-training" for the main KV cache (V4.1 §2.4.4)post-training: MXFP4 experts; V4.1 FP4 KVAleph AlphaKolibri-1Aleph Alpha Kolibri-1, QAT: "The trainer therefore uses FP8 quantisation-aware training" in RL (§3.2.1.3, App. I.4.1)FP8PoolsideLaguna XS.2 (report)Poolside Laguna XS.2 (report), PTQ, then QAD: "pre-trained and fine-tuned using the BF16 datatype"; PTQ to FP8 and INT4; NVFP4 after "quantization-aware distillation (QAD)" (Laguna report §5)FP8, INT4; NVFP4 + QADMistralLarge 4Mistral Large 4, trained in a shipped format: "we use FP8 tensor cores for fine-grained low-precision training" (§2.3); FP8 trainer and rollouts in RL (§4.4.2); mid-training and SFT precision not stated; ships an FP8 checkpointFP8Mistral Large 4, PTQ: "naive abs-max quantization" of the routed experts, "200M tokens from our SFT mixture for calibration" (§6)NVFP4 expertsNVIDIANemotron 3.5 SuperNVIDIA Nemotron 3.5 Super, PTQ, then QAD: "the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations" (3.5 report §4.1)NVFP4, 200 QAD iterationsGoogleGemma 4Google Gemma 4, QAT, stage not stated: "quantized versions of our models trained with quantization-aware training" (Gemma 4 report §1, §2.5); no stage givenQAT, stage not statedMiniMaxM3.1-FlashMiniMax M3.1-Flash, QAT, stage not stated: "the routed MoE experts and the attention paths are quantized during training" (M3.1-Flash model card); no stage givenQAT, stage not stated

Hover, focus or tap a mark for the lab's own words and the section they come from.

Figure 6.Where each lab brings its serving format into training, from its own reports. Solid bars are quantization-aware training; the outlined bar is training in a format the lab also ships (Mistral, FP8); the hatched bar is low-precision compute in a format the lab did not ship as trained (Ultra); dots are quantization after training, with a short bar where distillation follows. Two labs say they use QAT without saying when.

Two motives that look alike

Labs give two reasons for QAT, with different requirements.

The first is serving memory and speed: a model trained against its 4-bit grid loses less when served on it. DeepSeek wants “inference acceleration and reducing memory traffic at deployment”; Moonshot “to reduce memory footprint and serving cost”; Google “to reduce their parameter memory footprint and latency”; MiniMax “reducing memory footprint and improving serving efficiency”. For this motive, QAT has to end in exactly the released format, and it can start late.

The second is consistency in reinforcement learning. RL samples from an inference engine, and if the engine serves in FP8 while the trainer doesn’t round, the rollouts come from a slightly different policy; without QAT the engine in effect re-quantizes the trainer’s weights at each sync, a fresh PTQ every time. Aleph Alpha “observed RL training diverge in this configuration”, and QAT in the trainer is its fix, the only reason it gives. For this motive QAT has to run through RL, whatever is shipped: Kolibri-1 is released in both BF16 and FP8. Moonshot and DeepSeek state both motives; Xiaomi gives quality at low precision and identical rollout weights. Mistral matches an FP8 engine with an FP8 trainer without calling it QAT, and Poolside kept BF16 inference weights for Laguna M.1’s and XS.2’s RL, then ran Laguna S 2.1’s RL in FP8 without saying how. S5 covers the evidence on what a mismatch does and the corrections that bound it.

Quantizing after training, and distilling the gap away

Four labs quantize a finished model for at least one release: Mistral, NVIDIA (both model families, whatever the training format), Poolside, and Z.ai for Huawei’s Ascend chips.

Mistral’s is the simplest: it rounded Mistral Large 4’s routed experts to NVFP4 with “naive abs-max quantization”, using “200M tokens from our SFT mixture for calibration”, and reports that the 606 GB checkpoint matches “the FP8 reference in all evaluations”, enough to serve on four GB300 GPUs. The baseline is the FP8 release, not a BF16 model. Ultra’s PTQ recipe mixes formats by sensitivity (NVFP4 routed experts, FP8 shared experts and Mamba layers, BF16 attention) at 5.03 bits per value on average, and kept a median 96.8 to 98.5% of BF16 scores on an intermediate checkpoint, depending on how the scales were chosen; raising the budget to 7.19 bits “leaves every benchmark unchanged within noise”.

Where PTQ falls short, two labs use quantization-aware distillation (QAD), an idea from 2017 and 2018 (Mishra and Marr; Polino et al.). Start from the PTQ model and train it as in QAT, with float weights, fake quantization and the straight-through gradient, but with a different loss: a KL divergence to the full-precision model’s next-token distribution, as in distillation (S6), on a fixed dataset. The student never samples, so this is not on-policy. NVIDIA’s recipe for Nemotron 3.5 Super and Lightning trains “only the weights of language-model GEMMs with enabled quantizers”, freezes everything else, keeps the calibrated scales from PTQ, and switches off the ordinary next-token loss, the MTP loss and the load-balancing loss. Poolside used the same idea for Laguna XS.2’s NVFP4 weights after “direct post-training NVFP4 quantization caused a non-negligible quality drop”.

The QAD paper’s Table 1 shows why to distil instead of continuing with the task loss, as QAT does. On Llama Nemotron Super V1, QAT and QAD reach almost the same cross-entropy on held-out SFT data (0.408 and 0.416, BF16 0.408), but QAT’s KL divergence from the BF16 model is 0.311 against QAD’s 0.004. QAT “effectively act[s] as an additional post-training stage”, moving the model to another solution that fits the data equally well, while QAD keeps the original model’s behaviour, which is what post-training produced.

Figure 7.Scores in NVIDIA's QAD paper for four models quantized to NVFP4 after their full post-training. QAT and QAD here are one extra stage on SFT or cold-start data, not QAT inside a lab's own SFT or RL, and the paper has no BF16 run on the same data. AIME has 30 problems, so one problem is 3.3 points. The paper picked each trained model's checkpoint by benchmark score, a choice PTQ does not have.

On NVIDIA’s own Nemotron 3.5 Super, in an ablation of a 4.8-bit recipe against an earlier BF16 version, PTQ kept a median 97.68% of the BF16 scores and QAD raised that to between 98.36% and 99.18%, best after 200 iterations on the SFT data alone, with 1.11% more output tokens than BF16. Mixing in RL rollouts from the teacher “provided only modest benefits in our ablations”, and more iterations “were not uniformly beneficial”. Against the final BF16 model, the released checkpoint (a 4.82-bit “aggressive” recipe, PTQ plus 200 QAD iterations) keeps a median 99.62% across 36 benchmarks.

The costs are small next to training: “about 27 GB200 GPU-hours per PTQ candidate”, and “about 2 hours (i.e., 1024 GB200 GPU-hours)” per 100 QAD iterations. QAD can’t adapt the model to the format beyond the full model’s behaviour, and matches that behaviour only on the fixed data it sees.

The choices side by side

ApproachWhat runs in low precisionBackward passWhat it buysWhat it costsLabs (stated)
BF16 mixed precisionthe compute copy (BF16)BF16, FP32 sums where sensitivethe baselinecare with sums across GPUsPoolside, Aleph Alpha
FP8 trainingall three multiplications, fine-grained scalesFP8up to 2× peak arithmetic, half the linear layers’ saved activationsFP32 accumulation, sensitive parts kept highMistral, Xiaomi (V2-Flash); DeepSeek implied
NVFP4 trainingmost linear layers, all three multiplicationsNVFP4 with stochastic rounding, Hadamard on Wgrad2–3× FP8’s peak arithmetic15% of layers kept high; loss gap ~1% vs FP8; a new format in the longest runNVIDIA
QATforward weights (often activations) on the serving gridgradients unrounded; trainer’s own arithmeticthe model adapts to the format it is served in; RL rollouts matchextra rounding per step; oscillation; every later stage runs on a quantized modelXiaomi, Moonshot, Z.ai, DeepSeek, Aleph Alpha, Google, MiniMax
PTQnothing in trainingnonecheapest; no training riskan accuracy gap, larger at 4 bitsMistral, NVIDIA, Poolside, Z.ai (for Ascend)
PTQ + QADquantized weights trained against the full modelgradients unroundedcloses most of PTQ’s gap without moving the model~1,000 GPU-hours per 100 iterations at NVIDIA; matches only on fixed dataNVIDIA, Poolside

Where the models in the map sit

From each lab’s own documents; a blank means we searched and found nothing.

LabPre-training computeMaster weightsServing format entersRelease format (config or card)Motive stated
NVIDIA (Nemotron 3 Ultra)NVFP4, 15% of layers higherPTQ after trainingNVFP4 / FP8 / BF16 mix, 5.03 bitsserving
NVIDIA (Nemotron 3.5 Super, Lightning)Lightning card: “an NVFP4 recipe”PTQ, then 200 QAD iterationsNVFP4 / FP8, 4.82 bits (Super)serving
Xiaomi (MiMo-V2.6)not stated (V2-Flash: FP8)FP32QAT from mid-trainingMXFP4 experts, FP8 rest (store_dtype: mxfp4)quality; identical rollout weights
Moonshot (Kimi K3)QAT from SFTMXFP4 experts (mxfp4-pack-quantized)serving; RL consistency
Z.ai (GLM-5)QAT in SFT; PTQ for AscendBF16, FP8 (no INT4 file found)accuracy at low precision
DeepSeek (V4, V4.1)FP8 framework (implied)FP32QAT in post-trainingFP4 experts, FP8 rest (expert_dtype: fp4; V4.1’s FP4 experts shown by config; its report doesn’t restate expert QAT)serving; rollout = deployment
Aleph Alpha (Kolibri-1)BF16 parameters, FP32 reductions (card)QAT in RL onlyBF16 and FP8RL consistency
Mistral (Large 4)FP8PTQ after trainingFP8; NVFP4 experts, 606 GBserving
Poolside (Laguna XS.2, from its report; S 2.1 is in the map)BF16FP32 (one RL section says BF16)PTQ; QAD for NVFP4FP8, INT4, NVFP4low-VRAM devices
Google (Gemma 4)QAT, stage not statedQ4_0, mobile mixmemory and latency
MiniMax (M3.1-Flash)QAT, stage not statedNVFP4 experts, MXFP8 restmemory and serving
OpenAI (gpt-oss)“post-trained … with quantization”MXFP4 MoE weightsmemory

Qwen, StepFun, Tencent and Thinking Machines state no training precision and release FP8 or NVFP4 checkpoints without saying how they made them. Liquid AI states BF16 with FP32 master weights only for an earlier model’s fine-tuning.

Easy to get wrong

  • Reading a checkpoint’s format as its training precision. DeepSeek’s cards say “FP4 + FP8 Mixed”; that describes the files. Nemotron 3 Ultra was pre-trained in NVFP4 and its NVFP4 release was still made by PTQ from BF16.
  • Taking “trained in FP8” or “in NVFP4” to mean the whole state. Every recipe here keeps FP32 master weights or FP32 sums, and the 4-bit copy saves at most 8% of the per-parameter training state.
  • Treating the NVFP4 loss gaps as one number. 0.4% is Ultra against short BF16 continuations, probably a lower bound; 1 to 1.5% is the 12B model against FP8; neither is a BF16 model trained from scratch.
  • Counting a lab’s QAT as evidence about when QAT should start. Each lab reports one start point, and the only quantified comparison of QAT against alternatives (the QAD paper) ran it after RL, not during.
  • Reading “matches” across baselines. Mistral’s NVFP4 matches its FP8 checkpoint; NVIDIA’s recovery percentages are against BF16; the QAD paper’s comparisons pick checkpoints by benchmark score.

Check yourself

A weight in BF16 equals 1.5. The optimizer adds 0.003. What does the weight become, and what would it become with an FP32 master copy?

In [1,2)[1, 2) BF16 numbers are 2−7=0.00782^{-7} = 0.0078 apart, so the update must be at least half that, 0.0039, to change the weight. 0.003 is smaller: the weight stays 1.5. With an FP32 master copy, the master becomes 1.503, and its BF16 copy still rounds to 1.5 (the nearest BF16 numbers are 1.5 and 1.5078). After two such steps the master is 1.506 and the copy rounds to 1.5078: the updates were kept, and reach the forward pass once they add up to more than half a spacing.

Kimi K3’s routed experts hold 2.723T parameters. How many bytes do they take in BF16, in MXFP4, and in a Muon trainer’s state at 14 bytes per parameter?

BF16: 2.723T × 2 = 5.45 TB. MXFP4 at 4.25 bits: 2.723T × 4.25 / 8 = 1.45 TB. Training state: 2.723T × 14 = 38.1 TB. The served 4-bit experts are about 4% of what the trainer holds for the same parameters.

Why does NVIDIA’s NVFP4 recipe use 16 × 16 scale tiles for weights but 1 × 16 strips for activations?

A block’s scale runs along the dimension a product sums over. The weight’s products run along its rows in Fprop and down its columns in Dgrad, so strips would round it differently in the two, and the backward pass would differentiate a different matrix. A square tile gives one rounded matrix for both, at a somewhat higher rounding error. Activations have the same mismatch between Fprop and Wgrad, and keep the finer strips because training tolerates inconsistent activations better than inconsistent weights.

A lab says its QAT keeps the model’s validation loss equal to the BF16 model’s. Why might its quantized model still behave differently?

Equal loss on the data does not mean equal output distributions. In the QAD paper, QAT matched BF16’s cross-entropy (0.408) but drifted to a KL divergence of 0.311 from it, while QAD stayed at 0.004. Training on the task loss can move the model to another solution that fits the data as well, and lose behaviour that came from earlier stages such as RL.

In the map

Methods

Taxonomy nodes

Further reading

  • Micikevicius et al., Mixed Precision Training, arXiv:1710.03740: FP32 master weights, loss scaling and FP32 accumulation, with the experiments behind them.
  • Gupta et al., Deep Learning with Limited Numerical Precision, arXiv:1502.02551, and Zamirai et al., Revisiting BFloat16 Training, arXiv:2010.06192: stochastic rounding and Kahan summation as fixes for lost updates.
  • Jacob et al., arXiv:1712.05877: the simulated-quantization recipe QAT descends from, for 8-bit integer inference.
  • Bengio, Léonard and Courville, arXiv:1308.3432: the straight-through estimator and its bias.
  • Nagel et al., Overcoming Oscillations in Quantization-Aware Training, arXiv:2203.11086: why QAT weights flip between grid points, and two remedies.
  • DeepSeek-V3 technical report §3.3, arXiv:2412.19437: the FP8 training framework, fine-grained scales and accumulation.
  • NVIDIA, Pretraining Large Language Models with NVFP4, arXiv:2509.25149: the four-part recipe, its ablations, and NVFP4 against MXFP4.
  • Xin et al., Quantization-Aware Distillation for NVFP4, arXiv:2601.20088: QAD against PTQ and QAT on SFT- and RL-trained models.
  • Nemotron 3 Ultra report §2.2, §2.7, §4 and Nemotron 3.5 report §4: NVFP4 pre-training at scale, its divergences, and the PTQ and QAD recipes.
  • DeepSeek-V4 report §5.2.1 and V4.1 report §2.4.4; Kimi K3 report §4.1.4; MiMo-V2.6 report §3.2, §6.4; Kolibri-1 report §3.2.1.3 and Appendix I.4.1: each lab’s QAT in its own words.
↑