Open Models Almanac
Methodsinference & servinginference quantization

single source Β· inference & serving

MXFP8 block-scaled attention projection quantization

MXFP8 projection quantizes the attention q/k/v projections to FP8 with per-block scaling: one E8M0 exponent per block of elements instead of a single scale per tensor.1 This finer scale granularity is enough to close the accuracy gap that per-tensor FP8 leaves against BF16 in long-context retrieval.2

Trade-off It recovers long-context accuracy over per-tensor FP8, performing similarly to BF16 above 32k context.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Substituting MXFP8 for per-tensor FP8 onπ‘ž/π‘˜/𝑣 changes only the scale granularity with one E8M0 exponent per 32 elements instead of one float per tensor β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. which is enough to close the gap β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  3. π‘ž/π‘˜/𝑣 MXFP8 projection recovers95.3%and94 .3%above 32k, similar to π‘ž/π‘˜/𝑣 BF16 β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called MXFP8 projection, MXFP8’s per-block scaling.

Filed under inference & serving β€Ί inference quantization