Hierarchical Memory Architecture Needed to Resolve AI Inference Bottlenecks

The focus of AI infrastructure is shifting from 'training,' centered on large-scale computation, to 'inference,' which prioritizes multi-user processing and cost efficiency. As the proliferation of long contexts, RAG, and AI agents causes a surge in KV Cache capacity, GPU HBM alone can no longer resolve bottlenecks.

The Need for a Hierarchical Memory Structure

Optimization of the entire data movement path is required, rather than simply improving GPU performance. Technologies like NVIDIA Dynamo propose a structure that expands KV Cache capacity by utilizing CPU RAM and disk storage outside of GPU memory. To achieve this, the following hierarchical memory structure is essential.

• GPU HBM: Stores model weights and Hot KV Cache (highest performance, capacity constrained)
• Host DRAM: Buffers Warm KV Cache, session states, and request queues (reduces HBM burden)
• NVMe SSD: Offloads Cold KV Cache and stores long contexts/RAG source documents (large capacity, latency management required)
• Shared Storage: Stores large documents and shared KV Cache (requires high bandwidth)

Role Changes by Component and Architectural Transition

Host DRAM functions as a buffer layer for inference services rather than a GPU replacement, handling session state maintenance and cache offloading. The CPU has transformed from the main actor in model computation to a hub responsible for request scheduling and memory hierarchy management. The NVIDIA Vera Rubin NVL72 is a rack-scale platform integrating GPU, CPU, DPU, and network, suggesting that systemic harmony is important beyond individual server optimization.

NVMe SSD has moved beyond a simple storage device to become an inference performance layer. Samsung Electronics also explains that storage has changed from a supplementary component to an active layer enabling system scalability. DPUs and NICs minimize GPU data supply latency through distributed I/O processing.

Key Metrics and Competitiveness of Inference Infrastructure

Criteria for purchasing and operating AI servers have shifted from GPU specifications to data movement paths and memory efficiency. In addition to traditional CPU/memory metrics, inference-specific metrics such as Time to First Token (TTFT), KV Cache Usage, and Cache Hit Ratio must be monitored. CSP competitiveness will be determined by platform capabilities that provide low latency and predictable cost structures, rather than GPU holdings.