single mention · inference & serving
Request-level prefix cache
A request-level prefix cache stores the KV cache and MoE routed experts from prior turns of a request for reuse in later generation steps.1 Unlike radix cache, it avoids re-prefilling and sharing cached outputs across requests, keeping the sampling of routed experts consistent.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
This cache stores KVCache and MoE routed experts from prior turns, allowing them to be reused for subsequent generation steps of the same request
— MiMo-V2-Flash Technical Reportour request-level prefix cache avoids re-prefilling or inter-request output cache sharing, ensuring sampling consistency for routed experts
— MiMo-V2-Flash Technical Report
Filed under inference & serving › KV cache management