single mention Β· inference & serving
Checkpointing replay for Mamba recurrent-state cache
Checkpointing replay is an inference method for Mamba/SSM layers that stores a checkpointed recurrent state plus a small buffer of recent per-token inputs, instead of writing out a new state for every speculative decoding candidate.1 It reconstructs the state on demand from the checkpoint and buffered inputs, amortizing state writes over the buffer length and avoiding quantization error beyond the checkpoint itself.2,3
Trade-off It reduces per-step state-related memory traffic from T+1 to roughly 1 + 1/L, at the cost of some computational overhead amortized over the buffer.4,5
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We store a checkpointed stateπππππ‘ and store theππ,π£π,πΌπ seen since the checkpoint in a small buffer
β NVIDIA Nemotron 3.5 Super and Lightning Technical Reportwe can reconstruct the state on-demand, and with no quantization error beyond whatβs inπππππ‘
β NVIDIA Nemotron 3.5 Super and Lightning Technical ReportThis amortizes the state write over our buffer length
β NVIDIA Nemotron 3.5 Super and Lightning Technical Reportreducing our per-step state-related memory traffic fromπ + 1to roughly1 + 1/πΏ
β NVIDIA Nemotron 3.5 Super and Lightning Technical ReportIt also lets us amortize any computational overhead associated with writing the state
β NVIDIA Nemotron 3.5 Super and Lightning Technical Report
Also called checkpointing replay.
Filed under inference & serving βΊ KV cache management