Nvidia Sets New Inference Speed Record for DeepSeek-R1 in MLPerf v6.0: 8064 Tokens per GPU per Second

Recently, in the MLPerf v6.0 AI inference benchmark test, Nvidia achieved all-round leading results with its Blackwell Ultra architecture (GB300 NVL72). It not only ranked first in inference performance but also had 9 times more training wins than its closest competitor. This test version introduced multiple new generative AI models, focusing on the inference interactivity of dense large language models and vision-language models, as well as the performance of large-scale multi-node systems.

In the DeepSeek-R1 server-side test, Nvidia achieved 8064 tokens per GPU per second, a 2.77x speed increase compared to the v5.1 version. Additionally, the Llama3.1 405B model saw performance improvements of 1.52x and 1.21x in server and offline tests respectively, demonstrating strong AI inference capabilities.