Open Models Almanac
Methodsinference & servingKV cache management

single mention · inference & serving

Asynchronous KV-cache offloading

Asynchronous KV-cache offloading moves cached key–value state out of HBM, e.g. to a pinned host pool, and back when needed.1 The offloading and restoration run on side CUDA streams rather than the compute stream, so they never stall generation.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. astateresidesinHBMduringGPUtime andinapinnedhostpoolduringtooltime — MiMo V2 6 technical report
  2. OffloadingandrestorationrunonsideCUDAstreams rather than the compute stream, never stalling generation — MiMo V2 6 technical report

Filed under inference & serving › KV cache management