Open Models Almanac
Methodsinference & servinginference scheduling

specific method · filed under inference & serving

KV-cache-aware routing

Routes requests with awareness of KV-cache state to improve cache use.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See inference scheduling for the methods filed beside it.

Also called KV-cache-aware request routing, KV-cache-aware router.

sources
2
models
2
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 1

A KV-cache-aware router sends each request to an instance.

usedinference servingin Nemotron 3.5 Super VLNVIDIA
1 more from this model

NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV cache management, and disaggregated serving.

optionalinference servingin NVIDIA DynamoNVIDIA
↑