specific method · filed under inference & serving
Asynchronous scheduling
A scheduling mechanism that overlaps device-to-host sampling copies with preparation of the next decode step.
Also called async scheduling.
- sources
- 3
- models
- 3
- labs adopt it
- 3
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs explainer www.lmsys.orgdescribes the overlap scheduler that hides CPU/device overhead
Evidence
3 spans quoted from the sources, strongest treatment first.
--async-scheduling
usedunclearin Qwen3.6-35B-A3BQwen
Asynchronous Scheduling: Within vLLM, we implemented a mechanism to overlap the 'Device-to-Host' (D2H) sampling copies with the preparation of the next decode step
usedinference servingin GLM-5Z.ai
Filed alongside
Other methods under inference & serving :: inference scheduling.
NUMA binding of workers to GPU-local CPU socketsComposite-key sorting for topology-aware GPU rank assignmentContinuous batchingCross-group pinning of cache-hit blocksDual-cluster consistent-hash failover for cache affinityEnvoy-based proxy with custom orchestratorExacto routingGreedy feasible-rank placement by remaining capacityHost-side scheduling optimizationIncreasing batch size for inferenceLatency-sensitive execution class with priority isolationMax sequences tuningPer-node hard admission constraintPrefix-cache-aware session affinity schedulingRegistering NVLink domains as Ray custom resourcesRequest-class resource-budget admission controlRuntime-signal-based rollout concurrency auto-throttlingSub-NUMA partitioning with per-VM NUMA bindingWorkload-aware routed-expert GEMM scheduling