taxonomy node · level 2

inference scheduling

20 methods filed at this node or below it, from the sources of 13 models.

inference & serving :: inference scheduling

Matching aids for the classifier: async scheduling; host-side scheduling optimization; NUMA binding; topology-aware NVLink placement; continuous batching.

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Model	techniques
DeepSeek-V4.1-Flash used	Latency-sensitive execution class with priority isolation usedPer-node hard admission constraint usedSub-NUMA partitioning with per-VM NUMA binding used—
NVIDIA-Nemotron-3-Ultra-550B-A55B used	Composite-key sorting for topology-aware GPU rank assignment usedNUMA binding of workers to GPU-local CPU sockets usedRegistering NVLink domains as Ray custom resources used—
MiMo-V2.6-Flash used	Greedy feasible-rank placement by remaining capacity used—
GLM-5.3 used	Max sequences tuning used—
GLM-5.2 used	Asynchronous scheduling used—
MiniMax-M3 used	Host-side scheduling optimization used—
Gemma 4 31B optional	Exacto routing optionalIncreasing batch size for inference evaluated—
Kimi K3 core	Cross-group pinning of cache-hit blocks coreDual-cluster consistent-hash failover for cache affinity corePrefix-cache-aware session affinity scheduling coreRequest-class resource-budget admission control coreRuntime-signal-based rollout concurrency auto-throttling coreWorkload-aware routed-expert GEMM scheduling core—
Laguna-S-2.1 used	Envoy-based proxy with custom orchestrator used—
MiMo-V2.6-Pro used	Greedy feasible-rank placement by remaining capacity used—
Qwen3.6-35B-A3B used	Asynchronous scheduling used—
Qwen3.8-Flash-Next optional	Continuous batching optional—
Step-3.7-Flash used	Asynchronous scheduling used—

inference scheduling

In this branch 20

filed here 20

By model