Open Models Almanac
Methodsinference & servinginference quantization

single mention · inference & serving

FP8 precision

FP8 precision stores KV cache entries in 8-bit floating point instead of BF16.1 It is applied to all KV dimensions except the RoPE dimensions, which stay in BF16, and this hybrid representation cuts the KV cache size by nearly half.2

Trade-off It halves the KV cache size relative to pure BF16 storage.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. FP8 precision is applied to the remaining dimensions — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  2. BF16 precision is used for the rotary positional embedding (RoPE) dimensions, while FP8 precision is applied to the remaining dimensions. This hybrid representation reduces the KV cache size by nearly half compared with pure BF16 storage — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. reduces the KV cache size by nearly half compared with pure BF16 storage — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Also called FP8 precision for KV storage.

Filed under inference & serving › inference quantization