specific method · filed under inference & serving
FP4 KV-cache quantization
Stores the main key-value cache in FP4 to reduce cache storage, with one cited implementation using E2M1 values and per-16-channel E4M3 scales.
Also called FP4 main KV cache, FP4 KV caching, FP4 main KV caching, FP4-quantized main key-value cache, 4-bit floating-point key-value caching, FP4 main KV-cache compression.
- sources
- 2
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
5 spans quoted from the sources, strongest treatment first.
we use FP4 global KV caches during training with only marginal performance degradation. Together, CSA2 and FP4 KV caching reduce global KV cache storage to approximately 1/4 of that of DeepSeek-V4-Flash.
We now extend QAT to the main KV cache, where FP4 reduces storage rather than accelerates matrix multiplication.
cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) and FP4 KV caching reduce its global KV cache footprint
Combined with FP4 main KV caching (E2M1 format, one E4M3 scale per 16 channels), these designs reduce the global KV cache footprint to 890 bytes per token — roughly 1/4 of DeepSeek-V4-Flash.
we compress the main KV cache to FP4 to further reduce storage overhead.
Filed alongside
Other methods under inference & serving :: inference quantization.