NVIDIA Boosts Kimi K3 Inference Speed by 1.68x

Research firm SemiAnalysis reported that NVIDIA made an upstream vLLM optimization, allowing the 2.8-trillion-parameter Kimi K3 model to run at 423 tokens per second per user on GB300 NVL72 hardware, a 1.68x performance gain on the same setup.

The work was done by a NVIDIA CentML engineer through large-scale Megakernel fusion. A new decode-only kernel was written in Cute DSL, and the whole Megakernel contains 12,000 lines of code.

This AI-assisted way of building advanced compute kernels is seen as a future direction. NVIDIA is also developing vLLM optimizations for DeepSeek V4.1 and other models, steadily improving how hardware supports large model inference.