AI BriefingNVIDIA Full-Stack Codesign Boosts Agentic Throughput per Dollar by 30x
·2026.11.06 01:00
Key point
By optimizing the memory hierarchy across GPUs, CPUs, and storage, NVIDIA achieves a 30x increase in agentic throughput per dollar with only a 3% rise in HBM capacity.
Details
- NVIDIA's full-stack codesign approach increases agentic throughput per dollar by up to 30x on the Vera Rubin NVL72 platform compared to GB300 NVL72, despite HBM capacity per GPU rising only roughly 3%.
- The strategy addresses the "memory wall" by managing data across a hierarchy: shrinking model weights using NVFP4 format, distributing weights across up to 72 GPUs in an NVL72 domain, and offloading inactive KV cache to lower-cost storage tiers.
- NVIDIA Dynamo software and NVIDIA CMX context memory storage move paused context to shared, low-cost storage (partners include Cloudian, DDN, Dell, HPE, IBM, NetApp, VAST, and WEKA) to free up high-bandwidth memory for active concurrent users.
- RAG efficiency is improved by using NVIDIA Nemotron Retriever models for smaller embeddings and the cuVS library to keep the search index on CPU/storage rather than occupying HBM.
- On the SemiAnalysis AgentX DeepSeek V4 Pro workload, GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform at 100 tokens per second per user.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Analysis of Power Density and Liquid Cooling Infrastructure Changes in North American AIDC Driven by NVIDIA GPU Evolution
Nvidia and Meta Expand AI Partnership to Deploy Vera Rubin Infrastructure for Agents