On AI Hardware (7 min read)

While the computational power of modern GPUs is extremely strong, in the autoregressive decoding process of LLMs, the bottleneck becomes HBM (High Bandwidth Memory) bandwidth rather than compute power. In cutting-edge GPUs like H100, computational performance (PFLOP/s) is growing much faster than memory bandwidth, causing the 'operational-intensity gap'—where the speed of transferring data from memory to compute units fails to keep up with compute speed—to widen increasingly.

To address this problem, various companies are attempting to overcome memory constraints in their own distinct ways.

• Groq: Uses SRAM built directly into the chip instead of HBM, eliminating data transfer latency and achieving deterministic execution.
• Cerebras: Provides a massive amount of on-chip SRAM and overwhelming internal memory bandwidth through a single wafer-scale chip.
• MatX: Utilizes software-managed scratchpad memory optimized for transformer inference patterns.
• d-Matrix: Adopts an in-memory compute approach that performs computation directly within the memory array itself.

On the software side, there are also attempts to shift the bottleneck point from memory bandwidth to compute power by adjusting batch size using the roofline heuristic. Through this, systems are designed to achieve target latency while maintaining maximum throughput.