NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV cache management, and disaggregated serving.
1 more from this model
The disaggregated deployment uses 7 GPUs for prefill only and 4 GPUs for decode only.