AI Economics Part 2 (11-minute read)

AI demand splits into human and agent. Humans send short, bursty requests, but agents perform long-running tasks that run 24/7. Agents presuppose multiple tool calls and long memory, and even small errors, once accumulated, can easily cause the entire task to fail.

Training and inference also show different utilization patterns. Training is flat and continuous, human inference is jagged like a commute pattern, but agent inference is continuous while also being interrupted at every tool call.

Training is a massive computation where thousands of GPUs stay synchronized without stopping for weeks, and the bottleneck is GPU-to-GPU communication. Inference splits into short, explosive requests for humans and long, sustained requests for agents, with the latter using 10 to 50 times more compute than a human session. As people start attaching web search, external APIs, and document lookups to their usage, human inference also comes to rely more heavily on CPU and DRAM.

At the chip level, the roles of CPU, GPU, HBM, and DRAM vary by workload. In training, GPU and HBM are the core while CPU and DRAM sit mostly idle. In human inference, HBM is the key factor in reducing first-token latency, while in agent inference, context and tool results spill over into DRAM and the CPU comes to the forefront, putting pressure on every layer at once.

There are four key points.

• Training: Adding more GPUs does not scale performance linearly. Less compute, better data selection, and better networking and software matter, and NVIDIA's advantage comes from an ecosystem tied together all the way through CUDA and NVLink.
• Human inference: The low utilization during night and early morning hours is a significant opportunity cost. Dedicated inference chips built for short requests such as Groq and AWS Inferentia, spot pricing, and dynamic reallocation create room for improvement.
• Agent inference: Because of long context, CPU-GPU handoffs, and spillover into DRAM, existing hardware built for human use is insufficiently optimized.
• HBM: This is a bottleneck dominated by SK hynix, Samsung, and Micron, with high barriers to entry in fabrication and packaging. Agent AI demand is creating a second HBM super-cycle, and the most promising near-term solution is CXL 3.0. However, large-scale commercialization appears to be 2 to 3 years away.