Open Models Almanac
Methodsinference & servinginference quantization

implementation detail · filed under inference & serving

FP8 KV-cache quantization

FP8 KV-cache quantization stores the attention key–value cache in FP8 format instead of a higher-precision dtype.1 It is used during serving and RL rollouts to reduce the memory each token occupies.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. KV cache quantized to FP8 — Model release notes - Poolside
  2. Lower memory per token — Model release notes - Poolside

Also called FP8 KV cache during RL rollouts, FP8 E4M3 key-value cache, fp8_e4m3 KV cache, FP8 key-value cache, FP8 KV Cache, --kv-cache-dtype fp8.

sources
12
models
8
labs adopt it
7
strongest
used

Documented in

Evidence

13 quotes from 12 sources, one card per model, each opening with its strongest.

How sources treat it
optional 4used 9

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted3 sources

KV Cache is quantized to FP8

usedinference servingin Nemotron 3 UltraNVIDIA
2 more from this model

kv-cache-dtype fp8

usedinference servingin Nemotron 3 UltraNVIDIA

--kv-cache-dtype fp8

optionalinference servingin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑