single mention · inference & serving
W4A16 quantization
W4A16 stores weights in four bits and keeps activations in 16: the kernel expands the weights just before each multiplication, so it needs no four-bit arithmetic units. It saves weight memory and weight traffic and runs on GPUs that cannot multiply in four bits.
Written for the explainer Quantization for serving: which bits to remove, and what each removal saves.
Also called W4A16 (weights NVFP4, activations BF16), W4A16.
Filed under inference & serving › inference quantization
Explainer · 02 · 35 minServing a large reasoning model at low costA token gets cheaper as more requests share each forward pass, until the latency promise or the KV-cache budget stops the batch from growingExplainer · I1 · 36 minQuantization for serving: which bits to remove, and what each removal savesFewer bits in the weights save memory and bytes read; fewer in the activations raise peak arithmetic where the matrix units multiply that format and shrink what crosses between GPUs; fewer in the KV cache let a GPU hold more sessions; and when values share a scale, one large value decides the grid for the rest