Open Models Almanac
Methodsinference & servinginference scheduling

single mention · inference & serving

Continuous batching

Batches inference requests continuously for serving rather than processing only fixed batches.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.

Also called Continuous batching for inference serving.

Filed under inference & serving › inference scheduling