NVIDIA Dynamo Open-Source Software Doubles AI Reasoning Performance
NVIDIA introduces Dynamo, an open-source inference library that dramatically accelerates AI reasoning models while cutting costs in AI factories. The software doubles performance on current hardware and boosts token generation by 30x on next-generation systems.
NVIDIA unveils Dynamo, a new open-source inference software designed to accelerate and scale AI reasoning models across large GPU fleets. As the successor to NVIDIA Triton Inference Server, this tool focuses on maximizing token revenue generation for AI factories by efficiently orchestrating complex inference requests. The software addresses the growing demand for cost-effective AI operations as models increasingly require tens of thousands of "thinking" tokens for every single prompt.
The technology achieves significant performance gains through a method called disaggregated serving, which separates the processing and generation phases of large language models onto different GPUs. This approach allows each phase to run independently for maximum hardware utilization. Using the same number of GPUs, Dynamo doubles the performance of AI factories serving Llama models on NVIDIA Hopper platforms and increases token generation by over 30x when running DeepSeek-R1 on GB200 NVL72 racks.
To maintain high efficiency, Dynamo dynamically adds, removes, and reallocates GPUs based on fluctuating request volumes and intelligently routes queries to minimize response computations. The software also offloads inference data to more affordable memory and storage devices to further reduce operational costs. As a fully open-source project, Dynamo supports popular frameworks including PyTorch, SGLang, NVIDIA TensorRT-LLM, and vLLM.