specific method · filed under inference & serving
FP8 KV-cache quantization
Stores key-value cache data in FP8; the evidence includes E4M3 and amax-scaled variants but does not establish them as a single more specific scheme.
Also called FP8 KV cache during RL rollouts, FP8 E4M3 key-value cache, fp8_e4m3 KV cache, FP8 key-value cache, FP8 KV Cache, --kv-cache-dtype fp8.
- sources
- 13
- models
- 10
- labs adopt it
- 7
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Quantized KV Cache - vLLM Documentation docs docs.vllm.ai
Evidence
15 spans quoted from the sources, strongest treatment first.
with FP8 KV cache — reaches the full 1M-token context on 8xB200.
FP8 precision is applied to the remaining dimensions
--kv-cache-dtype fp8
storing the KV cache in FP8 roughly doubles the number of concurrent trajectories a single replica can carry
The KV cache was quantized to FP8 using amax-based scaling with a calibration dataset of 128 long-contexts agentic traces.
Lower memory per token: KV cache quantized to FP8.
kv-cache-dtype fp8
KV Cache is quantized to FP8
requires modelopt activation and FP8 KV Cache alignment
--kv-cache-dtype fp8
--kv-cache-dtype fp8
--kv-cache-dtype fp8
On Blackwell you can add --kv-cache-dtype fp8 to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV.
--kv-cache-dtype fp8_e4m3
Filed alongside
Other methods under inference & serving :: inference quantization.