single source Β· inference & serving
MXFP8 block-scaled attention projection quantization
MXFP8 projection quantizes the attention q/k/v projections to FP8 with per-block scaling: one E8M0 exponent per block of elements instead of a single scale per tensor.1 This finer scale granularity is enough to close the accuracy gap that per-tensor FP8 leaves against BF16 in long-context retrieval.2
Trade-off It recovers long-context accuracy over per-tensor FP8, performing similarly to BF16 above 32k context.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Substituting MXFP8 for per-tensor FP8 onπ/π/π£ changes only the scale granularity with one E8M0 exponent per 32 elements instead of one float per tensor
β NVIDIA Nemotron 3.5 Super and Lightning Technical Reportwhich is enough to close the gap
β NVIDIA Nemotron 3.5 Super and Lightning Technical Reportπ/π/π£ MXFP8 projection recovers95.3%and94 .3%above 32k, similar to π/π/π£ BF16
β NVIDIA Nemotron 3.5 Super and Lightning Technical Report
Also called MXFP8 projection, MXFP8βs per-block scaling.
Filed under inference & serving βΊ inference quantization