Open Models Almanac
Methodsinference & servingKV cache management

specific method · filed under inference & serving

Periodic cache checkpointing

Periodic cache checkpointing stores the recurrent state only every CC decoding steps instead of each step, so the state is quantized less often.1 To keep the lagging state current, it caches input activations and replays them on-the-fly, reducing sequential quantization steps by a factor of CC.2

Trade-off It saves cache writes and quantization steps but pays extra compute and IO for the activation replay.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Instead of storing (thus, quantizing) the state in each decoding step, we store it every𝐶𝐶steps for an integer𝐶𝐶 >1 — NVIDIA Nemotron 3 Ultra Technical Report
  2. we cache the input activations and apply activation replay to forward the state on-the-fly. This reduces the number of sequential quantization steps by a factor of𝐶𝐶 — NVIDIA Nemotron 3 Ultra Technical Report
  3. can also save time due to saving on cache writes (though there is a tradeoff with the additional compute and IO used for the activation replay — NVIDIA Nemotron 3 Ultra Technical Report

Also called cache checkpointing, Periodic checkpointing of SWA KV entries, Periodic Checkpointing.

sources
3
models
3
lab adopt it
1
strongest
optional

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 2optional 1
↑