Open Models Almanac
Methodsinference & servinginference quantization

single mention · inference & serving

W4A16 quantization

W4A16 stores weights in four bits and keeps activations in 16: the kernel expands the weights just before each multiplication, so it needs no four-bit arithmetic units. It saves weight memory and weight traffic and runs on GPUs that cannot multiply in four bits.

Written for the explainer Quantization for serving: which bits to remove, and what each removal saves.

Also called W4A16 (weights NVFP4, activations BF16), W4A16.

Filed under inference & serving › inference quantization