KV Cache is quantized to FP8
2 more from this model
kv-cache-dtype fp8
--kv-cache-dtype fp8
implementation detail · filed under inference & serving
FP8 KV-cache quantization stores the attention key–value cache in FP8 format instead of a higher-precision dtype.1 It is used during serving and RL rollouts to reduce the memory each token occupies.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
KV cache quantized to FP8— Model release notes - Poolside
Lower memory per token— Model release notes - Poolside
Also called FP8 KV cache during RL rollouts, FP8 E4M3 key-value cache, fp8_e4m3 KV cache, FP8 key-value cache, FP8 KV Cache, --kv-cache-dtype fp8.
13 quotes from 12 sources, one card per model, each opening with its strongest.
KV Cache is quantized to FP8
kv-cache-dtype fp8
--kv-cache-dtype fp8
storing the KV cache in FP8 roughly doubles the number of concurrent trajectories a single replica can carry
Lower memory per token: KV cache quantized to FP8.
requires modelopt activation and FP8 KV Cache alignment
--kv-cache-dtype fp8
with FP8 KV cache — reaches the full 1M-token context on 8xB200.
--kv-cache-dtype fp8
On Blackwell you can add --kv-cache-dtype fp8 to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV.
--kv-cache-dtype fp8
--kv-cache-dtype fp8_e4m3
Picked by hand, not extracted: where to read more, not evidence for anything on this page.