specific method · filed under inference & serving
Prefill-decode disaggregation
Prefill/decode disaggregation runs prefill and decode on separate pools of GPUs and sends each request's KV cache from the first to the second, so prefills never interrupt decode steps. It costs a transfer per request and two pools to size, and its measured gain is goodput under latency targets rather than raw throughput.
Written for the explainer Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory.
Also called Prefill–Decode (PD) disaggregation.