Open Models Almanac
Methodsinference & servinginference scheduling

single mention · inference & serving

Concurrent request batching

Improves MoE inference throughput by processing multiple requests together instead of at batch size one.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.

Also called increasing batch size for MoE inference.

Filed under inference & serving › inference scheduling