NVIDIA Tackles AI Memory Squeeze with Full-Stack Design, Up to 30x More Throughput per Dollar

AI models keep getting bigger, and GPU memory is under huge pressure. HBM chips are getting scarce and pricier, and this trend looks set to continue.

To break this memory bottleneck, NVIDIA proposes a "full-stack co-design" approach, treating GPU, CPU, network, storage and software as one whole. Key moves include: compressing model weights and KV cache with the NVFP4 low-precision format, spreading weights across up to 72 GPUs in NVL72 systems, using Vera CPU's large memory for nearby buffering, and letting Dynamo software move inactive contexts off costly memory.

Citing the SemiAnalysis AgentX benchmark, at 125 tokens per user per second, Vera Rubin NVL72 delivers up to 30 times more agent throughput per dollar than GB300 NVL72, with only about 3% more HBM per GPU.

This shows AI competition is shifting from memory size to how much useful output each byte and each watt can deliver.