Inkling
Thinking Machines LabOnly the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16.
specific method · filed under inference & serving
Mixed-precision NVFP4 routed-expert quantization quantizes a mixture-of-experts model's routed experts to the NVFP4 format while keeping the rest of the model at its training precision.1,2 The result is a mixed-precision checkpoint, and naive abs-max quantization with calibration matches the FP8 reference in all evaluations.3,4
Trade-off It reduces latency and throughput cost versus full-precision serving, but the NVFP4 checkpoint requires Blackwell FP4 tensor cores.5,6
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
We also quantize ML4’s routed experts into NVFP4 format while keeping other components at their training precision— technical report
Only the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16— thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx
resulting in a mixed BF16–FP8–NVFP4 precision checkpoint at 606 GB— technical report
naive abs-max quantization already suffices and matches the FP8 reference in all evaluations with appropriate calibration— technical report
apply NVFP4 [82] quan- tization to further improve latency and throughput— technical report
requires Blackwell FP4 tensor cores (B200 / GB200— thinkingmachines/Inkling — 975B / 41B active · MOE · 1024K ctx
Also called NVFP4 mixed-precision quantization, NVFP4.
2 quotes from 2 sources, one card per model, each opening with its strongest.
Only the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16.
We also quantize ML4’s routed experts into NVFP4 format while keeping other components at their training precision, resulting in a mixed BF16–FP8–NVFP4 precision checkpoint at 606 GB.