The Inference Shift

The center of gravity for AI inference is splitting from a single general-purpose GPU into purpose-specific architectures. This is because the core bottleneck differs across training, answer inference, and agentic inference.

The GPU era emerged because training is massive parallel computation, which required HBM together with chip-to-chip networking. For inference too, prefill is compute-bound while decode is a bandwidth-bound stage that alternates between reading the KV cache and model weights, so GPUs remain valid there.

• Training is likely to continue being led by GPUs going forward.
• Answer inference favors ultra-fast chips like Cerebras or Groq, since token generation speed matters.
• Agentic inference needs to maintain context, state, and history over long periods, so a memory hierarchy combining host memory, SSD, DB, logs, embeddings, and object store becomes more important.

The Cerebras WSE-3 offers 44GB of on-chip SRAM and 21 PB/s of bandwidth, giving it roughly 6,000 times higher bandwidth than the H100's 80GB HBM at 3.35 TB/s, though with smaller memory capacity. However, once you exceed on-chip memory, the performance advantage drops off sharply, and the structure of using an entire wafer as a single chip creates significant yield and cost burdens.

Ultimately, true agents are evolving away from coding assistants where humans check the results, toward completing tasks without human intervention. This market's scale is determined not by the number of humans but by compute, and Nvidia is responding with Dynamo and CPU/memory racks, but in the long run, a cheaper and simpler memory hierarchy could become more advantageous. For the same reason, slower chips can be designed to be simpler and cooler, making them well-suited even for environments like space data centers.