Open Models Almanac
Methodsinference & servinginference scheduling

methods · inference & serving

inference scheduling

35 methods filed at this node or below it, from the sources of 22 models.

In this branch 35

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 35

Asynchronous inference and execution core · 7 sources · 9 quotes
KV-cache-aware routing used · 2 sources · 2 quotes
NUMA binding for GPU-local CPU affinity used · 2 sources · 2 quotes
Provider failover routing used · 2 sources · 2 quotes
Provider routing used · 2 sources · 2 quotes
Session affinity routing used · 2 sources · 2 quotes
29 single mentions
Cross-group cache-hit block pinning core · 1 source · 1 quote
Dual-cluster consistent-hash failover core · 1 source · 1 quote
Latency-sensitive execution class core · 1 source · 1 quote
Threshold-triggered inference core · 1 source · 1 quote
Action-chunk timestep alignment used · 1 source · 1 quote
Eager adaptive depth scheduling used · 1 source · 1 quote
Envoy proxy with custom orchestrator used · 1 source · 1 quote
Greedy feasible-rank placement used · 1 source · 1 quote
Host-side scheduling optimization used · 1 source · 1 quote
Maximum-sequence tuning used · 1 source · 1 quote
Metric-driven inference-fleet autoscaling used · 1 source · 1 quote
NVLink domains as Ray custom resources used · 1 source · 1 quote
Static-shape compilation used · 1 source · 1 quote
Continuous batching optional · 1 source · 1 quote
Exacto routing optional · 1 source · 1 quote
Concurrent request batching evaluated · 1 source · 1 quote
Load balancing across providers mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash-1017 usedProvider failover routing used—
DeepSeek-V4.1-Flash coreIndependent replica placement using predicted resource availability coreLatency-sensitive execution class coreNode-local placement validation with hard admission control coreSCHED_IDLE prioritization for non-latency-sensitive tasks coreSub-NUMA partitioning with NUMA-local worker VMs core—
MiniMax-M3.1-Flash usedProvider routing usedLoad balancing across providers mentioned—
GLM-5.3 usedMaximum-sequence tuning used—
MiMo-V2.6-Flash usedGreedy feasible-rank placement used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedComposite-key topology-aware GPU rank assignment usedNUMA binding for GPU-local CPU affinity usedNVLink domains as Ray custom resources used—
GLM-5.2 usedAsynchronous inference and execution used—
Gemma 4 31B optionalExacto routing optionalConcurrent request batching evaluated—
Kimi K3 coreCross-group cache-hit block pinning coreDual-cluster consistent-hash failover corePrefix-cache-aware session affinity scheduling coreRequest-class resource-budget admission control coreRuntime-signal-based rollout concurrency auto-throttling coreWorkload-aware routed-expert GEMM scheduling core—
Kolibri-1 usedPriority-based, topology-aware GPU job scheduling used—
Laguna-S-2.1 usedEnvoy proxy with custom orchestrator used—
MiMo-V2.6-Pro usedGreedy feasible-rank placement used—
MiniMax-M3 usedHost-side scheduling optimization used—
Mistral-Large-4 usedMetric-driven inference-fleet autoscaling usedSession-sticky routing to data-parallel ranks usedProvider routing mentioned—
MolmoAct 2 usedCaching invariant action-expert intermediates usedEager adaptive depth scheduling usedStatic-shape compilation used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedKV-cache-aware routing usedSession affinity routing used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedKV-cache-aware routing usedSession affinity routing usedAsynchronous inference and execution optionalProvider failover routing optional—
Qwen3.6-35B-A3B usedAsynchronous inference and execution used—
Qwen3.8-Flash-Next optionalContinuous batching optional—
SmolVLA coreAsynchronous inference and execution coreThreshold-triggered inference core—
Step-3.7-Flash usedAsynchronous inference and execution used—
Xiaomi-Robotics-1 coreAsynchronous inference and execution coreAction-chunk timestep alignment used—
↑