Open Models Almanac
Methodsinference & servinginference quantization

specific method · filed under inference & serving

Mixed-precision NVFP4 routed-expert quantization

Mixed-precision NVFP4 routed-expert quantization quantizes a mixture-of-experts model's routed experts to the NVFP4 format while keeping the rest of the model at its training precision.1,2 The result is a mixed-precision checkpoint, and naive abs-max quantization with calibration matches the FP8 reference in all evaluations.3,4

Trade-off It reduces latency and throughput cost versus full-precision serving, but the NVFP4 checkpoint requires Blackwell FP4 tensor cores.5,6

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. We also quantize ML4’s routed experts into NVFP4 format while keeping other components at their training precision — technical report
  2. Only the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16 — thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx
  3. resulting in a mixed BF16–FP8–NVFP4 precision checkpoint at 606 GB — technical report
  4. naive abs-max quantization already suffices and matches the FP8 reference in all evaluations with appropriate calibration — technical report
  5. apply NVFP4 [82] quan- tization to further improve latency and throughput — technical report
  6. requires Blackwell FP4 tensor cores (B200 / GB200 — thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx

Also called NVFP4 mixed-precision quantization, NVFP4.

sources
2
models
2
labs adopt it
2
strongest
default

Documented in

Thinking Machines Lab
Mistral AI

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1default 1

Mistral-Large-4

Mistral AIadopted1 source

We also quantize ML4’s routed experts into NVFP4 format while keeping other components at their training precision, resulting in a mixed BF16–FP8–NVFP4 precision checkpoint at 606 GB.

usedunclearin Mistral Large 4Mistral AI
↑