The 'New Golden Age' of AI Chip Architecture: The Evolution of Domain-Specific Hardware like GPU, TPU, and LPU, and Strategies to Solve the Memory Wall

As the limitations of Moore's Law and Dennard Scaling caused single-thread CPU performance growth to plummet from 52% to 3% annually, the need for Domain-Specific Architectures (DSA) for AI computing emerged. The 'New Golden Age of Computer Architecture' predicted by Hennessy and Patterson has become a reality, with dozens of new architectures such as GPU, TPU, LPU, NPU, and wafer-scale engines being developed and deployed.

The Essence of AI Computing and the Memory Wall

AI computing is dominated by matrix multiplication (GEMM) and vector multiplication (GEMV). While GEMM operations with high arithmetic intensity are primarily performed during training and the Prefill stage, the Decode stage is dominated by GEMV operations, which are processed token by token, causing a sharp drop in arithmetic intensity. At this point, the Memory Wall, arising from insufficient memory bandwidth relative to compute speed, is the core bottleneck. Each architecture attempts to solve this data movement problem through data placement, movement methods, compute unit structures, and inter-chip communication schemes.

Key Architectures and Market Trends

The key architectures that have successfully achieved actual deployment and their characteristics are as follows.

• NVIDIA GPU: The strongest leader based on the CUDA ecosystem and programmability. It has secured supply contracts totaling 6GW from OpenAI and Meta.
• TPU (Google): A systolic array-based accelerator used for training Gemini, with plans to provide up to 1 million chips to Anthropic.
• Trainium (AWS): Anthropic is currently using over 1 million chips to run Claude models.
• Cerebras Wafer-Scale Engine: Utilized in OpenAI's inference services.
• Groq LPU: Absorbed into the NVIDIA ecosystem through NVIDIA's $20 billion acquisition.

This hardware competition demonstrates that beyond simple compute performance, memory bandwidth optimization and large-scale scaling (scale-up/out) strategies have emerged as the core competitive advantages of AI infrastructure.