Open Models Almanac
Methodsinference & servingKV cache management

single mention · inference & serving

Request-level prefix cache

A request-level prefix cache stores the KV cache and MoE routed experts from prior turns of a request for reuse in later generation steps.1 Unlike radix cache, it avoids re-prefilling and sharing cached outputs across requests, keeping the sampling of routed experts consistent.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. This cache stores KVCache and MoE routed experts from prior turns, allowing them to be reused for subsequent generation steps of the same request — MiMo-V2-Flash Technical Report
  2. our request-level prefix cache avoids re-prefilling or inter-request output cache sharing, ensuring sampling consistency for routed experts — MiMo-V2-Flash Technical Report

Filed under inference & serving › KV cache management