specific method · filed under inference & serving
Increasing batch size for inference
A throughput approach that processes multiple requests simultaneously rather than serving at batch size one.
Also called increasing batch size for MoE inference.
- source
- 1
- model
- 1
- labs adopt it
- 0
- strongest
- evaluated
How sources treat it
One count per evidence span, weakest treatment to strongest.
evaluated 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
while the 26B mixture-of-experts model presents unique routing challenges at a batch size of 1 on Apple Silicon, processing multiple requests simultaneously (e.g., batch sizes of 4 to 8) unlocks up to a ~2.2x speedup locally.
evaluatedinference servingin Gemma 4 26B MoEGoogle
Filed alongside
Other methods under inference & serving :: inference scheduling.
Asynchronous schedulingNUMA binding of workers to GPU-local CPU socketsComposite-key sorting for topology-aware GPU rank assignmentContinuous batchingCross-group pinning of cache-hit blocksDual-cluster consistent-hash failover for cache affinityEnvoy-based proxy with custom orchestratorExacto routingGreedy feasible-rank placement by remaining capacityHost-side scheduling optimizationLatency-sensitive execution class with priority isolationMax sequences tuningPer-node hard admission constraintPrefix-cache-aware session affinity schedulingRegistering NVLink domains as Ray custom resourcesRequest-class resource-budget admission controlRuntime-signal-based rollout concurrency auto-throttlingSub-NUMA partitioning with per-VM NUMA bindingWorkload-aware routed-expert GEMM scheduling