Open Models Almanac
Methodsinference & servingserving parallelism

specific method · filed under inference & serving

Prefill-decode disaggregation

Prefill/decode disaggregation runs prefill and decode on separate pools of GPUs and sends each request's KV cache from the first to the second, so prefills never interrupt decode steps. It costs a transfer per request and two pools to size, and its measured gain is goodput under latency targets rather than raw throughput.

Written for the explainer Disaggregation and the KV-cache system: scheduling, prefix reuse and tiered memory.

Also called Prefill–Decode (PD) disaggregation.

sources
4
models
3
labs adopt it
2
strongest
used

Documented in

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

Prefill-decode disaggregation, which runs the two phases on separate workers with their own parallelism and scheduling, is the established response, and we adopt it for Nemotron 3 Ultra.

usedinference servingin Nemotron 3 UltraNVIDIA
1 more from this model

Prefill-decode disaggregation... we adopt it for Nemotron 3 Ultra

usedinference servingin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑