Kimi K3 has 2.78 trillion parameters, and 2.72 trillion of them sit in its routed experts, the many feed-forward blocks of which a router picks a few for each token (computed from config.json: 92 MoE layers × 896 experts × three 3,584 × 3,072 matrices). Stored in BF16, two bytes per parameter, the model would take 5.56 TB. Moonshot ships the experts in MXFP4, a 4-bit format, and with everything else left in BF16 the checkpoint comes to about 1.56 TB (computed). Its report says Moonshot applies “quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations”, so for all of fine-tuning and reinforcement learning the network ran with its experts on the 4-bit grid it would be served on.
NVIDIA pre-trained Nemotron 3 Ultra on 20T tokens with the linear layers of most blocks computing in NVFP4, also a 4-bit format, but made the NVFP4 checkpoint it released after training, by rounding a BF16 checkpoint: “We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4”. Computing in a format during training and adapting the model to the format it will be served in are two different jobs, and labs do each at different stages.
This article separates the two: what low precision does to a training step, the three pre-training regimes the labs report, quantization-aware training and when each lab starts it, and the methods that leave quantization to the end.
Where the evidence comes from
Training precision is not visible in a model’s files. A config.json records the format the released weights are stored in, which says nothing about how they were trained, so every training fact here comes from a lab’s technical report, then its model card or blog. We count labs, not models. This site tracks 16 labs with language models: Aleph Alpha, Alibaba (Qwen), DeepSeek, Google DeepMind, Liquid AI, MiniMax, Mistral, Moonshot, NVIDIA, OpenAI, Poolside, StepFun, Tencent, Thinking Machines, Xiaomi and Z.ai. Of these, 5 state the number format of their pre-training run, 7 call something they did quantization-aware training, and 2 describe quantization-aware distillation. NVIDIA’s model cards also call its NVFP4 pre-training “quantization-aware”; we count that as low-precision training, not QAT. Where a lab says nothing, the cell is blank, which does not mean the lab chose against the method.
The background you need
I1 defines the formats; here is the one-line version of each. A float’s name gives its exponent bits (E) and mantissa bits (M, the stored significant digits). BF16 has FP32’s 8-bit exponent and 7 stored mantissa bits: the same range as FP32, far fewer digits. FP8 comes as E4M3 (largest value 448) and E5M2 (wider range, fewer digits). FP4 means E2M1, which has 15 distinct values: 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4 and ±6. A 4-bit grid is useless without a scale, so MXFP4 gives each block of 32 values one power-of-two scale (4.25 bits per value) and NVFP4 gives each block of 16 an FP8 scale plus one FP32 scale per tensor (4.5 bits per value). INT4 here is the symmetric integers −7 to 7 times a group scale.
A training step multiplies matrices three times per linear layer. Store the weight as (outputs × inputs), as PyTorch does. With input activations (tokens × inputs), the forward pass computes (NVIDIA calls this product Fprop). The backward pass receives and computes the gradient for earlier layers, (Dgrad), and the gradient for the weights, (Wgrad). Each is a GEMM, a general matrix multiply. Training “in FP8” can mean any subset of the three, and the inputs of each can be rounded differently.
Then there is what training stores per parameter between steps. With the Adam optimizer, the usual mixed-precision setup keeps a 16-bit copy of the weights for the matrix multiplications (2 bytes), a 16-bit gradient (2), a 32-bit “master” copy of the weights (4) and Adam’s two running averages in 32 bits (4 + 4). That is 16 bytes per parameter, the figure the ZeRO paper uses. Activations come on top and grow with batch size and sequence length rather than with parameters.
Why a 16-bit weight can’t learn on its own
Suppose a weight sits at exactly 1.0 and the optimizer asks to move it by . In BF16, the representable numbers just above 1.0 are apart. Round-to-nearest sends back to , because a value returns to its starting point whenever
For a BF16 weight in the spacing is and the threshold : at any upward update under 1/256 of the weight is lost, and near any update under 1/512 (computed). Micikevicius et al. (2017), who introduced mixed-precision training for FP16, give the same condition with FP16’s ten mantissa bits: an update is lost when the weight is “at least 2048 times larger” than it. For BF16 the ratio is 256, eight times smaller. Adam’s step is roughly the learning rate, so a weight near 1, such as a normalization gain, loses every step at a learning rate of . Matrix weights start near 0.02, where such a step survives, but they lose every step below a learning rate of about (computed): the end of a decaying schedule, and the rates used for fine-tuning a finished model. Nothing reports the lost updates. On a Mandarin speech model, Micikevicius et al. found that “updating FP16 weights results in 80% relative accuracy loss” compared with updating an FP32 master copy.
- exact sum w0 + n·Δw
- FP32 master copy
- its BF16 copy (forward pass)
- BF16, stochastic rounding
- ±1 sd of stochastic rounding
- BF16, round to nearest
BF16 spacing at w = 1: 2⁻⁷ = 0.0078; half: 0.0039; your update Δw = 0.0001 is below it: every round-to-nearest step is lost.
After 10,000 steps: exact 2; FP32 master 2.000166 (BF16 copy 2); stochastic rounding 2.01563 (±1 sd band: 0.088); round to nearest 1.
Labs use three remedies. The first is a master copy in FP32, whose spacing near 1.0 is : the optimizer adds the update there, and each step rounds a fresh low-precision copy from it for the matrix multiplications. Micikevicius et al. made this the first of their three rules, with loss scaling (multiplying the loss so small FP16 gradients don’t underflow, which BF16’s wide exponent makes unnecessary) and accumulating products in FP32. The second is stochastic rounding (Gupta et al., 2015): round up with probability equal to the fraction of the gap covered, so that on average the rounded value equals the exact one (). Its cost is noise. For an update much smaller than the spacing, one stochastic step is far noisier than one round-to-nearest step, but its error grows like over steps while round-to-nearest’s bias grows like . The third is to keep sums in FP32 wherever many small numbers are added, such as gradient accumulation and reductions across GPUs. A fourth, compensated (Kahan) summation, keeps a 16-bit correction term beside each 16-bit weight; Zamirai et al. (2020) found it matched 32-bit training on their models, and no lab in the map reports it.
Training “in FP8” or “in NVFP4” means the matrix multiplications run on a low-precision copy rounded from FP32 weights at every step, and quantization-aware training, as we will see, is the same structure with the serving format in place of the compute format.
What training in low precision saves, and what it doesn’t
The low-precision copy is a small part of what training stores. Under ZeRO’s mixed-precision Adam the BF16 copy is 2 of 16 bytes per parameter. NVIDIA’s NVFP4 recipe keeps gradients in FP32 as well as the master weights and optimizer states (“The main weights (stored by the optimizer), weight gradients … and optimizer states are also kept in FP32”). With FP32 gradients a BF16 copy makes 18 bytes per parameter and an NVFP4 copy 16.56, a cut of at most 8% (computed). Two things shrink even that. Trainers commonly shard the optimizer state across data-parallel GPUs (ZeRO; Mistral and Google name it), while below ZeRO’s third stage (Mistral uses the second) each GPU keeps a full compute copy, so per GPU the copy is a larger share. And frameworks often keep the BF16 copy beside the low-precision one, in which case nothing is saved. DeepSeek-V3 cut the state itself, tracking Adam’s moments in BF16 and keeping master weights and gradients in FP32: 13 bytes if an FP8 copy is kept (computed).
Low-precision compute saves arithmetic, activation memory and network traffic. NVIDIA’s NVFP4 paper lists FP4 tensor-core throughput at 2× FP8 on GB200 and 3× on GB300 (two NVIDIA Blackwell systems), and FP8 at twice BF16. The inputs that linear layers save for the backward pass halve in FP8 and shrink up to 3.6× in NVFP4 (computed, scales included); attention, normalization and the layers kept in high precision save theirs in BF16, so recomputation and offloading, as in Kimi K3, remain the main tools for activation memory. DeepSeek-V3 also sends tokens to their experts in FP8 and gathers the results in BF16. None of the sources reports an end-to-end training speedup for NVFP4. At serving time the low-precision copy is the whole checkpoint.
- compute copy of the weights
- gradient
- FP32 master weights
- optimizer moment m
- optimizer moment v
Kimi K3, 2.78T parameters: ~39 TB of training state at 14 B/parameter (Muon, FP32 momentum assumed); ~1.56 TB checkpoint with MXFP4 experts
Training in BF16, and the sums every format keeps
Two labs in the map describe BF16 runs. Poolside trained Laguna M.1 and XS.2 with “BF16 mixed precision training throughout all stages with the master weights in FP32 and most computations in BF16”, with RMSNorm and RoPE partly in FP32; its later Laguna S 2.1, the model this site tracks, did its RL “in FP8 precision”. Aleph Alpha’s model card for Kolibri-1 says training used “bfloat16 parameters, float32 reductions”; it doesn’t mention a master copy, and the report itself is silent on pre-training precision.
Whatever the compute format, the sums across GPUs are where the reported failures sit. In data parallelism each GPU computes gradients on its own share of the batch and the gradients are summed across GPUs; in tensor parallelism each GPU holds a slice of a layer and the partial results are summed. NVIDIA’s Ultra diverged around 8T tokens after the output layer’s weight-gradient accumulation moved from FP32 to BF16, which lost the small contribution of its multi-token prediction heads; Poolside’s Laguna M.1 became unstable while its output layer’s input gradient was summed across tensor-parallel GPUs in BF16. Both moved the sum back to FP32, as S1 describes.
Training in FP8
FP8 training puts all three multiplications of a linear layer on FP8 tensor cores. DeepSeek-V3’s report is the most detailed public account, and later labs cite it. It runs “all three GEMMs associated with the Linear operator … in FP8”, which “theoretically doubles the computational speed compared with the original BF16 method” and lets activations be stored in FP8 for the backward pass. DeepSeek-V3 makes three choices.
It scales at a fine grain. One scale per tensor lets a single outlier set the grid for millions of values (I1 shows this), so DeepSeek-V3 scales activations in tiles of 1 × 128 (one token, 128 channels) and weights in blocks of 128 × 128. It computes each scale from the current tile, where the first FP8 recipes used “delayed” scaling from the largest values of earlier steps, which a sudden spike overflows. Fine scales also let it use E4M3 for gradients, where earlier recipes used E5M2.
It accumulates in FP32. Tensor cores add FP8 products into an accumulator, and on the H800 that accumulator keeps “around 14 bits”. On a product with 4,096 terms DeepSeek measured “a maximum relative error of nearly 2%”. The fix copies partial sums into FP32 registers every 128 terms.
It keeps sensitive parts out of FP8. “The embedding module, the output head, MoE gating modules, normalization operators, and attention operators” stay in BF16 or FP32, and master weights and gradients stay in FP32. DeepSeek validated the recipe at two smaller scales, about 1T tokens each, where the loss stayed within 0.25% of a BF16 run. The report shows no BF16 run of V3 itself.
Among the labs in the map, Xiaomi’s MiMo-V2-Flash report says it uses “an FP8 mixed-precision framework similar to DeepSeek-V3”, keeping attention output projections, the embedding and the output head in BF16 and the MoE router in FP32; the MiMo-V2.5 cards say “FP8 mixed precision”. The MiMo-V2.6 report states no pre-training precision. Mistral Large 4’s report says one sentence: “we use FP8 tensor cores for fine-grained low-precision training”, with no layer list or block size. DeepSeek-V4 inherits V3’s framework and calls it “the existing FP8 training framework” without saying in so many words that pre-training ran in FP8. It adds one detail on the gradient side: MoE gradients travel between GPUs in BF16 “in a stochastic rounding manner”, and each GPU sums what it receives in FP32, “to avoid accumulation errors introduced by low-precision adders”.
Training in NVFP4
Four bits is a much coarser grid than eight: E2M1’s top step, from 4 to 6, is half the value below it. NVIDIA’s NVFP4 pre-training paper reports that “training diverges when every linear layer is quantized to FP4”, and gives a recipe of four parts. The same recipe trained Nemotron 3 Super and Ultra. In every recipe here, attention’s own products (queries times keys, scores times values) stay in BF16.
- Keep “a few sensitive linear layers in higher precision (15% of the network, with the majority of high precision layers at the end of the network)”. Ultra kept “the final 15% of the network (16 layers)” of 108, plus Mamba output projections, latent projections, attention projections, the multi-token prediction layers and the embeddings. The family white paper gives BF16 for most of these and MXFP8 for the Mamba output projections, which in NVFP4 flushed up to 40% of values to zero on the smaller Nemotron 3 Nano.
- Apply a random Hadamard transform of size 16 to the inputs of Wgrad only. A Hadamard rotation spreads an outlier’s energy over its block (I1’s rotations section). Rotating both inputs of Wgrad along the token dimension they share leaves their exact product unchanged, so only the rounding sees the rotated values. The paper found “no measurable benefit for Fprop and Dgrad at smaller scales”.
- Scale weights in two-dimensional 16 × 16 blocks, activations and gradients in 1 × 16.
- Use stochastic rounding for gradients and round-to-nearest-even for weights and activations.
The third rule has no counterpart in inference quantization. A block’s scale has to run along the dimension a product sums over, so the hardware can take it outside each dot product. Fprop’s dot products run along ‘s rows (over its input dimension), Dgrad’s down its columns (over its output dimension), so each needs its own quantized copy. With 1 × 16 strips the two copies disagree: in a 64 × 64 Gaussian matrix, 91% of entries round to a different value in the two layouts (computed). The backward pass then differentiates a different function from the one the forward pass computed, which the paper calls “effectively breaking the chain rule”. A square tile has the same scale whichever way it is read. It costs some precision, since one tile’s largest value sets the scale for 256 entries instead of 16: the toy’s rounding error grows by about 1.2 times (computed over five seeds). Activations and gradients meet the same mismatch, between Fprop and Wgrad and between Dgrad and Wgrad, and keep their finer strips because “training is less sensitive to inconsistencies in activation tensors than weight tensors”. DeepSeek-V3’s 128 × 128 weight blocks solve the same problem in FP8.
- quantized value, negative to positive (one scale for both panels)
- cell rounds differently in the two passes
- faint lines: scale blocks
W quantized for Fprop (scales along rows)
W quantized for Dgrad (scales along columns)
cells that differ
cells that differ: 91.1% | RMS error vs W: 1.87e−3 (tiles / strips: 1.23×), over the 64 × 64 matrix; its top-left 32 × 32 corner is drawn
The fourth rule follows from what backpropagation is. Given the forward pass, Dgrad and Wgrad are linear in the incoming gradient, so zero-mean noise from stochastic rounding of gradients averages out: the weight gradient stays unbiased. Round-to-nearest on gradients is biased instead. With a block scale of (block maximum)/6, every entry under 1/24 of the block’s largest rounds to zero every time. On weights or activations the noise passes through nonlinearities and the loss, where it no longer averages out, and applying stochastic rounding there “causes divergence” in a 1.2B test model.
NVIDIA’s 12B model trained on 10T tokens in NVFP4 reached 62.58% on MMLU-Pro against 62.62% for the same model trained in FP8, a difference within noise, while other benchmarks moved by up to about 4 points either way (code 56.7 against 59.5, which the paper suspects is “noisy evaluations”; GSM8k 92.3 against 89.1). Its validation loss stayed within 1% of FP8’s while the learning rate was constant, widening to “slightly above 1.5%” as it decayed. In an ablation, switching the forward pass to higher precision for the last 18% of tokens recovered most of the gap (1.5% to 0.5%), with about 6% of the run’s compute in higher precision; Ultra’s report doesn’t say it did this. On Ultra, NVIDIA branched BF16 runs from the NVFP4 run at 5T, 10T and 16T tokens and continued each for 74B tokens, “a proxy for high-precision training”. The relative train-loss gap averaged below 0.4%. A short BF16 continuation measures only the part of the gap that switching precision recovers; on the 12B model that was two thirds of it, so 0.4% is probably a lower bound (our inference). Ultra’s report has no downstream comparison of the two; the only downstream claim is for an 8B-active test model in the white paper, whose evaluations were “comparable”.
Ultra’s report describes two divergences, and neither was traced to NVFP4: one was the BF16 sum above, the other “undetermined”, and switching all tensors to BF16 “did not resolve the training divergence discussed in section 2.7”. NVIDIA then cut the run to 20T tokens. A new format still makes a stage that can’t be rerun harder to debug. NVIDIA is the only lab in the map that reports pre-training in a 4-bit format.
Why did Ultra still need PTQ? Its training grid was not its serving grid. Training used 16 × 16 weight tiles and kept attention and the last 15% of layers high; the release uses 1 × 16 blocks with calibrated scales, NVFP4 for the routed experts only and FP8 for shared experts and Mamba layers; and post-training ended in a BF16 checkpoint. A model trained in FP8 and shipped in FP8 with the same block scheme is the converse case: its weights were on the serving grid from the first step. DeepSeek’s FP8 configs (128 × 128, as V3 trained) fit that pattern, and Mistral’s FP8 release may (its report gives no block size); neither lab says the release reuses the trainer’s quantizer.
Quantization-aware training
The second job is making the model good in its serving format. Post-training quantization (PTQ) rounds a finished model, as I1 describes, often after calibration: running sample text through the model to choose the scales. Quantization-aware training (QAT) keeps training while the forward pass sees the rounded weights. Jacob et al. (2017) gave the recipe that most labs cite. The weights stay in floating point “so that they can be easily nudged by small amounts”, and each forward pass replaces them with , their values on the serving grid, a step known as fake quantization or quantize-dequantize.
is a staircase, whose derivative is zero almost everywhere. Backpropagation through it would give no gradient at all. The straight-through estimator (Bengio et al., 2013) treats the rounding as the identity in the backward pass:
(With block scales set from the block’s maximum nothing is clipped, and the scale is treated as a constant.) What this computes is the gradient of the full-precision loss at the rounded weights, applied to the float weights: no variance, and a bias that shrinks with the grid spacing. Bengio et al. call it “clearly a biased estimator”. This is the master-copy structure from the start of the article. The difference from native low-precision training is in the backward pass. QAT rounds weights (and often activations) onto the serving grid for the forward pass and adds no rounding of gradients; Dgrad uses the rounded weight, and its arithmetic is whatever the trainer runs in, FP8 at DeepSeek. Native FP8 or NVFP4 training rounds the inputs of all three multiplications, to run faster. The QAD paper (Xin et al., 2026) says QAT “only quantizes the weights and activations, thus only helping forward pass”.
- rounded to the column's low-precision format
- the forward pass's rounded value, reused
- rounded in some recipes
- BF16 or FP32
To see where the straight-through gradient leads, take a quadratic loss with positive definite. The straight-through gradient is , which is zero only if the optimum happens to be on the grid. Otherwise the float weight never settles: it drifts to a rounding boundary and flips between neighbouring grid points, which Nagel et al. (2022) call oscillation. The time-average of approaches , but the network is served with one grid point, the one in use when training stops, and that need not be the best one.
- float weight w (from w*)
- rounded weight Q(w)
- visits, last 1,000 steps
- faint dashes: rounding boundaries; ellipses: equal loss
w* = (2.6, 1.3); rounding w* (PTQ) gives (3.0, 1.5), excess 0.344; the best grid point is (3.0, 1.0), excess 0.034.
step 4,000: Q(w) = (3.0, 1.5), excess loss 0.344; float w = (2.511, 1.256); time-average of Q(w) over the last 1,000 steps (2.600, 1.300). Visits in the last 1,000 steps: (3.0, 1.5) ×600, (2.0, 1.0) ×400.
A decaying learning rate shrinks the oscillation but not its rate (Nagel et al.: it “reduces the amplitude of the oscillations but do[es] not affect their frequency”), so the run ends on whichever neighbouring grid point is in use at the last step. On a larger toy, a least-squares problem with a 32-weight block in E2M1, with the block scale recomputed each step, that gave between 0.78 and 1.05 times PTQ’s excess loss with the learning rate decayed to zero, and 1.5 to 4.4 times with it constant (computed). A single block of a convex problem has nothing else to adapt; in a network, the other weights and layers shift to compensate for each one’s rounding, which is where QAT gains over PTQ. No lab says whether its learning rate decays during QAT.
The labs’ implementations differ in detail:
- DeepSeek-V4’s experts: “the FP32 master weights maintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for computation”. The FP4-to-FP8 step is lossless for DeepSeek’s weights, which it checked, so QAT runs on the existing FP8 training framework “without any modification”, and gradients go “directly … back to the FP32 master weights, equivalent to applying the Straight-Through Estimator”.
- Aleph Alpha’s trainer “applies the same FP8 rounding as the inference engine but computes in BF16, without FP8 kernels”, with the straight-through estimator in the backward pass.
- Xiaomi applies quantize-dequantize to MiMo-V2.6’s experts “after each parameter update”, following “the numeric constraints of the MXFP4 Humming GEMM kernels” used by its rollout engine (the inference server that generates RL’s training samples), “so the two engines see identical expert weights”.
- Z.ai built “a quantization kernel applicable to both training and offline weight quantization, which ensures bitwise-identical behavior between training and inference”.
The trainer’s rounding should be the engine’s, down to how the scales are chosen, or the model adapts to a slightly different grid. Exporting itself makes the weights match; activations, rounded on the fly by each side, are where they can drift. What QAT costs depends on the trainer. At DeepSeek it is almost nothing, since FP4 values are computed in its FP8 framework anyway; Aleph Alpha’s emulation gives up FP8 kernels in training; a BF16 trainer pays an extra rounding pass per weight and activation, which the SGLang RL team, reproducing Kimi K2-Thinking’s INT4 recipe, found “noticeably slower than BF16”.
When QAT starts
Five of the seven labs that describe QAT say when it begins: mid-training, SFT, RL, or, at DeepSeek, “post-training” without a stage.
| Lab | Model | Stage (lab’s words) | Format and what is quantized |
|---|---|---|---|
| Xiaomi | MiMo-V2.6 | “during mid-training”, FP32 master weights carried into RL | MXFP4, experts |
| Moonshot | Kimi K3 | “from the SFT stage onward”, “covering both SFT and RL” | MXFP4 expert weights, MXFP8 activations; attention, shared experts, routers higher |
| Z.ai | GLM-5 | “in the SFT stage” | INT4; tensors not stated |
| DeepSeek | V4 | “during the post-training stage”, teacher and reference models included | MXFP4 expert weights and the sparse-attention indexer’s query–key path |
| DeepSeek | V4.1-Flash | “QAT during post-training” | adds the main KV cache: E2M1 with one E4M3 scale per 16 channels |
| Aleph Alpha | Kolibri-1 | RL only | FP8 (E4M3) weights in 128 × 128 blocks, activations, KV cache; embeddings, norms, router in BF16 |
| Gemma 4 | not stated | Q4_0 blocks; a mobile mix of int2/int4 weights and int8 activations | |
| MiniMax | M3.1-Flash | not stated | “the routed MoE experts and the attention paths” |
OpenAI says it “post-trained the models with quantization of the MoE weights to MXFP4” for gpt-oss, without the word quantization-aware or a mechanism, so it stays out of the count. Xiaomi’s “during mid-training” is the earliest start among the seven; the report’s reason is “allowing the model to adapt to low-precision computation while preserving model quality”. When RL begins, Xiaomi carries over “the FP32 master weights and Muown’s row state from the SFT checkpoint” (Muown is the Muon variant Xiaomi uses), “to stabilize MXFP4 training”, so QAT runs from mid-training through every later stage. DeepSeek quantizes its teacher and reference models too, the models whose outputs post-training compares the student with, so every model in the loop computes in the served format.
Labs apply QAT to the routed experts first: they hold nearly all the parameters (97.9% of Kimi K3’s) and tolerate rounding better. NVIDIA’s sensitivity analysis on Nemotron 3.5 found “MoE experts are comparatively robust” while the output layer “accounts for nearly half of the NVFP4 sensitivity”, though the same report blames 4-bit experts for “runaway loop generation in reasoning failures”. The KV cache, the keys and values a model keeps for every earlier token while it generates, is a target too, for storage: FP4 in V4.1 “nearly halves the storage footprint” against V4’s FP8 cache.
An earlier start gives the model more steps on the grid and lets every later stage, including RL, optimize the model that will actually be served. It also means every SFT and RL experiment runs on a quantized model, with whatever instability that brings, and a bug in the quantizer contaminates everything after it. We found no source that compares start points on one model. The nearest evidence comes from NVIDIA’s QAD paper, which ran QAT as a single extra stage after a finished multi-stage pipeline (SFT, RL, merging) and found that “NVFP4 QAT breaks the RL model’s capabilities”: on Nemotron 3 Nano and AceReason Nemotron 1.1 7B, QAT on SFT or RL-generated data scored below plain PTQ on every benchmark listed. That is QAT after RL, not during it, and the paper has no BF16 run on the same data, so part of the drop may come from more task-loss training after RL rather than from the 4-bit grid. Either reading suggests (ours) why the labs that use QAT put it inside their own post-training rather than after it.
- quantization-aware training
- trained in a format the lab also ships
- low-precision compute, not shipped as trained
- quantization after training (PTQ)
- PTQ, then distillation (QAD)
- QAT, stage not stated
Hover, focus or tap a mark for the lab's own words and the section they come from.
Two motives that look alike
Labs give two reasons for QAT, with different requirements.
The first is serving memory and speed: a model trained against its 4-bit grid loses less when served on it. DeepSeek wants “inference acceleration and reducing memory traffic at deployment”; Moonshot “to reduce memory footprint and serving cost”; Google “to reduce their parameter memory footprint and latency”; MiniMax “reducing memory footprint and improving serving efficiency”. For this motive, QAT has to end in exactly the released format, and it can start late.
The second is consistency in reinforcement learning. RL samples from an inference engine, and if the engine serves in FP8 while the trainer doesn’t round, the rollouts come from a slightly different policy; without QAT the engine in effect re-quantizes the trainer’s weights at each sync, a fresh PTQ every time. Aleph Alpha “observed RL training diverge in this configuration”, and QAT in the trainer is its fix, the only reason it gives. For this motive QAT has to run through RL, whatever is shipped: Kolibri-1 is released in both BF16 and FP8. Moonshot and DeepSeek state both motives; Xiaomi gives quality at low precision and identical rollout weights. Mistral matches an FP8 engine with an FP8 trainer without calling it QAT, and Poolside kept BF16 inference weights for Laguna M.1’s and XS.2’s RL, then ran Laguna S 2.1’s RL in FP8 without saying how. S5 covers the evidence on what a mismatch does and the corrections that bound it.
Quantizing after training, and distilling the gap away
Four labs quantize a finished model for at least one release: Mistral, NVIDIA (both model families, whatever the training format), Poolside, and Z.ai for Huawei’s Ascend chips.
Mistral’s is the simplest: it rounded Mistral Large 4’s routed experts to NVFP4 with “naive abs-max quantization”, using “200M tokens from our SFT mixture for calibration”, and reports that the 606 GB checkpoint matches “the FP8 reference in all evaluations”, enough to serve on four GB300 GPUs. The baseline is the FP8 release, not a BF16 model. Ultra’s PTQ recipe mixes formats by sensitivity (NVFP4 routed experts, FP8 shared experts and Mamba layers, BF16 attention) at 5.03 bits per value on average, and kept a median 96.8 to 98.5% of BF16 scores on an intermediate checkpoint, depending on how the scales were chosen; raising the budget to 7.19 bits “leaves every benchmark unchanged within noise”.
Where PTQ falls short, two labs use quantization-aware distillation (QAD), an idea from 2017 and 2018 (Mishra and Marr; Polino et al.). Start from the PTQ model and train it as in QAT, with float weights, fake quantization and the straight-through gradient, but with a different loss: a KL divergence to the full-precision model’s next-token distribution, as in distillation (S6), on a fixed dataset. The student never samples, so this is not on-policy. NVIDIA’s recipe for Nemotron 3.5 Super and Lightning trains “only the weights of language-model GEMMs with enabled quantizers”, freezes everything else, keeps the calibrated scales from PTQ, and switches off the ordinary next-token loss, the MTP loss and the load-balancing loss. Poolside used the same idea for Laguna XS.2’s NVFP4 weights after “direct post-training NVFP4 quantization caused a non-negligible quality drop”.
The QAD paper’s Table 1 shows why to distil instead of continuing with the task loss, as QAT does. On Llama Nemotron Super V1, QAT and QAD reach almost the same cross-entropy on held-out SFT data (0.408 and 0.416, BF16 0.408), but QAT’s KL divergence from the BF16 model is 0.311 against QAD’s 0.004. QAT “effectively act[s] as an additional post-training stage”, moving the model to another solution that fits the data equally well, while QAD keeps the original model’s behaviour, which is what post-training produced.
- BF16
- NVFP4 PTQ
- NVFP4 QAT
- NVFP4 QAD
- bars in this order in every group
On NVIDIA’s own Nemotron 3.5 Super, in an ablation of a 4.8-bit recipe against an earlier BF16 version, PTQ kept a median 97.68% of the BF16 scores and QAD raised that to between 98.36% and 99.18%, best after 200 iterations on the SFT data alone, with 1.11% more output tokens than BF16. Mixing in RL rollouts from the teacher “provided only modest benefits in our ablations”, and more iterations “were not uniformly beneficial”. Against the final BF16 model, the released checkpoint (a 4.82-bit “aggressive” recipe, PTQ plus 200 QAD iterations) keeps a median 99.62% across 36 benchmarks.
The costs are small next to training: “about 27 GB200 GPU-hours per PTQ candidate”, and “about 2 hours (i.e., 1024 GB200 GPU-hours)” per 100 QAD iterations. QAD can’t adapt the model to the format beyond the full model’s behaviour, and matches that behaviour only on the fixed data it sees.
The choices side by side
| Approach | What runs in low precision | Backward pass | What it buys | What it costs | Labs (stated) |
|---|---|---|---|---|---|
| BF16 mixed precision | the compute copy (BF16) | BF16, FP32 sums where sensitive | the baseline | care with sums across GPUs | Poolside, Aleph Alpha |
| FP8 training | all three multiplications, fine-grained scales | FP8 | up to 2× peak arithmetic, half the linear layers’ saved activations | FP32 accumulation, sensitive parts kept high | Mistral, Xiaomi (V2-Flash); DeepSeek implied |
| NVFP4 training | most linear layers, all three multiplications | NVFP4 with stochastic rounding, Hadamard on Wgrad | 2–3× FP8’s peak arithmetic | 15% of layers kept high; loss gap ~1% vs FP8; a new format in the longest run | NVIDIA |
| QAT | forward weights (often activations) on the serving grid | gradients unrounded; trainer’s own arithmetic | the model adapts to the format it is served in; RL rollouts match | extra rounding per step; oscillation; every later stage runs on a quantized model | Xiaomi, Moonshot, Z.ai, DeepSeek, Aleph Alpha, Google, MiniMax |
| PTQ | nothing in training | none | cheapest; no training risk | an accuracy gap, larger at 4 bits | Mistral, NVIDIA, Poolside, Z.ai (for Ascend) |
| PTQ + QAD | quantized weights trained against the full model | gradients unrounded | closes most of PTQ’s gap without moving the model | ~1,000 GPU-hours per 100 iterations at NVIDIA; matches only on fixed data | NVIDIA, Poolside |
Where the models in the map sit
From each lab’s own documents; a blank means we searched and found nothing.
| Lab | Pre-training compute | Master weights | Serving format enters | Release format (config or card) | Motive stated |
|---|---|---|---|---|---|
| NVIDIA (Nemotron 3 Ultra) | NVFP4, 15% of layers higher | PTQ after training | NVFP4 / FP8 / BF16 mix, 5.03 bits | serving | |
| NVIDIA (Nemotron 3.5 Super, Lightning) | Lightning card: “an NVFP4 recipe” | PTQ, then 200 QAD iterations | NVFP4 / FP8, 4.82 bits (Super) | serving | |
| Xiaomi (MiMo-V2.6) | not stated (V2-Flash: FP8) | FP32 | QAT from mid-training | MXFP4 experts, FP8 rest (store_dtype: mxfp4) | quality; identical rollout weights |
| Moonshot (Kimi K3) | QAT from SFT | MXFP4 experts (mxfp4-pack-quantized) | serving; RL consistency | ||
| Z.ai (GLM-5) | QAT in SFT; PTQ for Ascend | BF16, FP8 (no INT4 file found) | accuracy at low precision | ||
| DeepSeek (V4, V4.1) | FP8 framework (implied) | FP32 | QAT in post-training | FP4 experts, FP8 rest (expert_dtype: fp4; V4.1’s FP4 experts shown by config; its report doesn’t restate expert QAT) | serving; rollout = deployment |
| Aleph Alpha (Kolibri-1) | BF16 parameters, FP32 reductions (card) | QAT in RL only | BF16 and FP8 | RL consistency | |
| Mistral (Large 4) | FP8 | PTQ after training | FP8; NVFP4 experts, 606 GB | serving | |
| Poolside (Laguna XS.2, from its report; S 2.1 is in the map) | BF16 | FP32 (one RL section says BF16) | PTQ; QAD for NVFP4 | FP8, INT4, NVFP4 | low-VRAM devices |
| Google (Gemma 4) | QAT, stage not stated | Q4_0, mobile mix | memory and latency | ||
| MiniMax (M3.1-Flash) | QAT, stage not stated | NVFP4 experts, MXFP8 rest | memory and serving | ||
| OpenAI (gpt-oss) | “post-trained … with quantization” | MXFP4 MoE weights | memory |
Qwen, StepFun, Tencent and Thinking Machines state no training precision and release FP8 or NVFP4 checkpoints without saying how they made them. Liquid AI states BF16 with FP32 master weights only for an earlier model’s fine-tuning.
Easy to get wrong
- Reading a checkpoint’s format as its training precision. DeepSeek’s cards say “FP4 + FP8 Mixed”; that describes the files. Nemotron 3 Ultra was pre-trained in NVFP4 and its NVFP4 release was still made by PTQ from BF16.
- Taking “trained in FP8” or “in NVFP4” to mean the whole state. Every recipe here keeps FP32 master weights or FP32 sums, and the 4-bit copy saves at most 8% of the per-parameter training state.
- Treating the NVFP4 loss gaps as one number. 0.4% is Ultra against short BF16 continuations, probably a lower bound; 1 to 1.5% is the 12B model against FP8; neither is a BF16 model trained from scratch.
- Counting a lab’s QAT as evidence about when QAT should start. Each lab reports one start point, and the only quantified comparison of QAT against alternatives (the QAD paper) ran it after RL, not during.
- Reading “matches” across baselines. Mistral’s NVFP4 matches its FP8 checkpoint; NVIDIA’s recovery percentages are against BF16; the QAD paper’s comparisons pick checkpoints by benchmark score.
Check yourself
A weight in BF16 equals 1.5. The optimizer adds 0.003. What does the weight become, and what would it become with an FP32 master copy?
In BF16 numbers are apart, so the update must be at least half that, 0.0039, to change the weight. 0.003 is smaller: the weight stays 1.5. With an FP32 master copy, the master becomes 1.503, and its BF16 copy still rounds to 1.5 (the nearest BF16 numbers are 1.5 and 1.5078). After two such steps the master is 1.506 and the copy rounds to 1.5078: the updates were kept, and reach the forward pass once they add up to more than half a spacing.
Kimi K3’s routed experts hold 2.723T parameters. How many bytes do they take in BF16, in MXFP4, and in a Muon trainer’s state at 14 bytes per parameter?
BF16: 2.723T × 2 = 5.45 TB. MXFP4 at 4.25 bits: 2.723T × 4.25 / 8 = 1.45 TB. Training state: 2.723T × 14 = 38.1 TB. The served 4-bit experts are about 4% of what the trainer holds for the same parameters.
Why does NVIDIA’s NVFP4 recipe use 16 × 16 scale tiles for weights but 1 × 16 strips for activations?
A block’s scale runs along the dimension a product sums over. The weight’s products run along its rows in Fprop and down its columns in Dgrad, so strips would round it differently in the two, and the backward pass would differentiate a different matrix. A square tile gives one rounded matrix for both, at a somewhat higher rounding error. Activations have the same mismatch between Fprop and Wgrad, and keep the finer strips because training tolerates inconsistent activations better than inconsistent weights.
A lab says its QAT keeps the model’s validation loss equal to the BF16 model’s. Why might its quantized model still behave differently?
Equal loss on the data does not mean equal output distributions. In the QAD paper, QAT matched BF16’s cross-entropy (0.408) but drifted to a KL divergence of 0.311 from it, while QAD stayed at 0.004. Training on the task loss can move the model to another solution that fits the data as well, and lose behaviour that came from earlier stages such as RL.
In the map
Methods
- NVFP4 pretraining 5 sources
- FP8 mixed-precision training 4 sources
- BF16 mixed-precision training 5 sources
- FP32 gradient reduction 1 source
- Stochastic Rounding 5 sources
- Quantization-Aware Training 10 sources
- MXFP4 Quantization-Aware Training 4 sources
- MXFP4 Weights with MXFP8 Activations 2 sources
- INT4 Quantization-Aware Training 1 source
- FP8 Quantization-Aware Training 1 source
- Post-training quantization 5 sources
- Quantization-aware distillation 4 sources
- Quantize-Dequantize Training 1 source
Taxonomy nodes
- training precision 30 methods
- quantization-aware training 17 methods
Further reading
- Micikevicius et al., Mixed Precision Training, arXiv:1710.03740: FP32 master weights, loss scaling and FP32 accumulation, with the experiments behind them.
- Gupta et al., Deep Learning with Limited Numerical Precision, arXiv:1502.02551, and Zamirai et al., Revisiting BFloat16 Training, arXiv:2010.06192: stochastic rounding and Kahan summation as fixes for lost updates.
- Jacob et al., arXiv:1712.05877: the simulated-quantization recipe QAT descends from, for 8-bit integer inference.
- Bengio, Léonard and Courville, arXiv:1308.3432: the straight-through estimator and its bias.
- Nagel et al., Overcoming Oscillations in Quantization-Aware Training, arXiv:2203.11086: why QAT weights flip between grid points, and two remedies.
- DeepSeek-V3 technical report §3.3, arXiv:2412.19437: the FP8 training framework, fine-grained scales and accumulation.
- NVIDIA, Pretraining Large Language Models with NVFP4, arXiv:2509.25149: the four-part recipe, its ablations, and NVFP4 against MXFP4.
- Xin et al., Quantization-Aware Distillation for NVFP4, arXiv:2601.20088: QAD against PTQ and QAT on SFT- and RL-trained models.
- Nemotron 3 Ultra report §2.2, §2.7, §4 and Nemotron 3.5 report §4: NVFP4 pre-training at scale, its divergences, and the PTQ and QAD recipes.
- DeepSeek-V4 report §5.2.1 and V4.1 report §2.4.4; Kimi K3 report §4.1.4; MiMo-V2.6 report §3.2, §6.4; Kolibri-1 report §3.2.1.3 and Appendix I.4.1: each lab’s QAT in its own words.
