Open Models Almanac
Methodsinference & servingKV cache management

single mention Β· inference & serving

Checkpointing replay for Mamba recurrent-state cache

Checkpointing replay is an inference method for Mamba/SSM layers that stores a checkpointed recurrent state plus a small buffer of recent per-token inputs, instead of writing out a new state for every speculative decoding candidate.1 It reconstructs the state on demand from the checkpoint and buffered inputs, amortizing state writes over the buffer length and avoiding quantization error beyond the checkpoint itself.2,3

Trade-off It reduces per-step state-related memory traffic from T+1 to roughly 1 + 1/L, at the cost of some computational overhead amortized over the buffer.4,5

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We store a checkpointed stateπ‘†π‘π‘˜π‘π‘‘ and store theπ‘˜π‘–,𝑣𝑖,𝛼𝑖 seen since the checkpoint in a small buffer β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  2. we can reconstruct the state on-demand, and with no quantization error beyond what’s inπ‘†π‘π‘˜π‘π‘‘ β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  3. This amortizes the state write over our buffer length β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  4. reducing our per-step state-related memory traffic from𝑇 + 1to roughly1 + 1/𝐿 β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report
  5. It also lets us amortize any computational overhead associated with writing the state β€” NVIDIA Nemotron 3.5 Super and Lightning Technical Report

Also called checkpointing replay.

Filed under inference & serving β€Ί KV cache management