specific method · filed under inference & serving
Runtime-signal-based rollout concurrency auto-throttling
A scheduling method that uses runtime load signals to dynamically control the number of requests sent to the inference engine.
Also called auto-throttling mechanism.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 1
Documented in
Evidence
1 span quoted from the sources, strongest treatment first.
We therefore design an auto-throttling mechanism at the LLM request scheduling layer, using runtime signals such as active request count, queued request count, and KV cache utilization to dynamically control how many requests are sent to the inference engine.
coreinference servingin Kimi K3Moonshot AI
Filed alongside
Other methods under inference & serving :: inference scheduling.
Asynchronous schedulingNUMA binding of workers to GPU-local CPU socketsComposite-key sorting for topology-aware GPU rank assignmentContinuous batchingCross-group pinning of cache-hit blocksDual-cluster consistent-hash failover for cache affinityEnvoy-based proxy with custom orchestratorExacto routingGreedy feasible-rank placement by remaining capacityHost-side scheduling optimizationIncreasing batch size for inferenceLatency-sensitive execution class with priority isolationMax sequences tuningPer-node hard admission constraintPrefix-cache-aware session affinity schedulingRegistering NVLink domains as Ray custom resourcesRequest-class resource-budget admission controlSub-NUMA partitioning with per-VM NUMA bindingWorkload-aware routed-expert GEMM scheduling