single mention · inference & serving
Asynchronous KV-cache offloading
Asynchronous KV-cache offloading moves cached key–value state out of HBM, e.g. to a pinned host pool, and back when needed.1 The offloading and restoration run on side CUDA streams rather than the compute stream, so they never stall generation.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
astateresidesinHBMduringGPUtime andinapinnedhostpoolduringtooltime
— MiMo V2 6 technical reportOffloadingandrestorationrunonsideCUDAstreams rather than the compute stream, never stalling generation
— MiMo V2 6 technical report
Filed under inference & serving › KV cache management