NVIDIA Dynamo Boosts Reasoning AI Throughput by 30x on Blackwell GPUs
NVIDIA unveils Dynamo, an open-source distributed inference framework that drastically increases throughput for reasoning models like DeepSeek-R1. The new tool uses innovative scheduling and disaggregated architecture to optimize large-scale AI deployments.
NVIDIA introduces Dynamo, an open-source inference serving framework designed to scale generative AI and reasoning models across large distributed environments. Unveiled at GTC 2025, this new tool increases the number of served requests by up to 30 times when running DeepSeek-R1 models on NVIDIA Blackwell hardware.
The framework achieves these massive performance gains through several key architectural innovations. It separates prefill and decode inference stages, dynamically schedules GPUs based on shifting demand, and utilizes LLM-aware request routing to eliminate costly KV cache recomputations. Additionally, Dynamo accelerates asynchronous data transfers between GPUs and offloads KV caches across various memory hierarchies to further reduce response times.
Developers easily integrate NVIDIA Dynamo into their existing workflows because it remains compatible with popular open-source tools like PyTorch, SGLang, NVIDIA TensorRT-LLM, and vLLM. The framework is currently available to the public on the ai-dynamo/dynamo GitHub repository, while enterprise users will soon access it through NVIDIA NIM microservices for added security and production-level support.