The Performance Engine of AI Chips, Systolic Array

Systolic Array is the core processing unit of AI chips, handling matrix multiplication (matmul) operations with extreme efficiency. Since most of the computation in AI models depends on matrix multiplication, executing it quickly determines overall performance.

Matrix multiplication requires two input matrices to interact according to a geometric pattern. The Systolic Array implements this pattern in physical hardware, moving data only to nearby nodes to minimize long-distance communication. This allows it to reduce data movement and maximize parallel computation.

The Systolic Array's operating methods are divided into two types: the Accumulator Stationary method, where each node handles one element of the output matrix while streaming inputs, and the Weight Stationary method, where each node holds an input matrix element while streaming the other input.

The difference in strategy between NVIDIA and Google TPU is notable. NVIDIA GPUs started with small arrays (8×8) and gradually expanded (now 128×128), relying on runtime scheduling. Google TPU, on the other hand, chose larger arrays from the start (evolving from 128×128 to 256×256) and leverages compiler-based pre-scheduling. Recently, NVIDIA has also been shifting toward the TPU approach, adding larger arrays and special function units like TMA.

System software is key to fully utilizing the Systolic Array. Incorrect data movement scheduling or synchronization can leave the array idle. Compilers like Luminal minimize data movement and maximize instruction schedule overlap, allowing each hardware unit to pipeline independently.