single mention · inference & serving
Continuous batching
Batches inference requests continuously for serving rather than processing only fixed batches.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.
Also called Continuous batching for inference serving.
Filed under inference & serving › inference scheduling
Explainer · 00 · 35 minThe early modern LLM and what it costsLLaMA 1 as the baseline: prefill pays in FLOPs, decode pays in bytes, and nearly every later design moves one of the twoExplainer · 02 · 35 minServing a large reasoning model at low costA token gets cheaper as more requests share each forward pass, until the latency promise or the KV-cache budget stops the batch from growingExplainer · I4 · 31 minDisaggregation and the KV-cache system: scheduling, prefix reuse and tiered memoryReading a cached prefix back from another tier usually costs far less than recomputing it; the price is in keeping it: memory held for hours, a route back to it, and for hybrid models a saved state at the right boundary