single mention · inference & serving
W4A4 quantization
W4A4 quantizes both the weights and the activations of a layer to four bits. On hardware whose matrix units multiply four-bit numbers, such as NVIDIA Blackwell, it has the highest peak arithmetic of the 4-bit layouts, and it puts the activations on the same coarse grid as the weights.
Written for the explainer Quantization for serving: which bits to remove, and what each removal saves.
Also called W4A4.
Filed under inference & serving › inference quantization
Explainer · 02 · 35 minServing a large reasoning model at low costA token gets cheaper as more requests share each forward pass, until the latency promise or the KV-cache budget stops the batch from growingExplainer · I1 · 36 minQuantization for serving: which bits to remove, and what each removal savesFewer bits in the weights save memory and bytes read; fewer in the activations raise peak arithmetic where the matrix units multiply that format and shrink what crosses between GPUs; fewer in the KV cache let a GPU hold more sessions; and when values share a scale, one large value decides the grid for the rest