A KV-cache-aware router sends each request to an instance.
1 more from this model
NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV cache management, and disaggregated serving.