Resolving AI Inference Memory Bottlenecks: Shifting Server Design to Hierarchical Memory Architectures

The proliferation of AI inference services has caused KV Cache growth to create bottlenecks across the entire data movement path, extending beyond GPU HBM.
Data access efficiency must be optimized through a hierarchical memory structure composed of HBM, Host DRAM, NVMe SSD, and high-performance shared storage.
The CPU serves as a hub for request scheduling and service control, while DPUs and NICs handle distributed I/O processing, coordinating GPU computation and data supply.
Server procurement and design criteria are shifting from a focus on GPU specifications to a focus on workload-specific memory usage patterns and data movement path efficiency.

Key Memory Hierarchy Roles

GPU HBM: The top-tier layer for storing model weights and Hot KV Cache; it offers high speed but faces significant capacity and cost constraints.
Host DRAM: A buffer layer that stores Warm KV Cache and session states, distributing the load on HBM.
NVMe SSD: A high-capacity auxiliary layer for Cold KV Cache offloading and RAG source data storage, supporting inference scalability.

Platforms such as NVIDIA Dynamo, CMX, and Vera Rubin NVL72 propose a structure that integrates and manages storage and networks outside of GPU memory.
Bottleneck locations must be identified by comprehensively monitoring the relationship between Time to First Token, KV Cache Hit Ratio, and GPU and Memory Utilization.
The capability to operate AI service platforms that optimize latency, cost, and concurrent request scale together, rather than simply providing GPUaaS, is becoming increasingly important.