Two Bets on Fixation and a Dark Horse

Taalas and Groq have adopted different approaches to fix models close to the chip in order to reduce weight movement, the biggest bottleneck in AI inference. While GPUs fetch weights from HBM for every token and layer, dedicated chips place weights in on-chip memory to lower dependence on memory bandwidth.

Taalas etches model weights into a mask-ROM fabric during the manufacturing stage. By combining read-only memory with compute capabilities, this approach ensures data flows along physical layers within the chip without leaving for external memory. The trade-off is a lack of flexibility: once a model is fixed, it cannot be rewritten.

According to vendor-published figures, Taalas processes 16,960 tokens per user per second on Llama 3.1 8B, with 0.015J per token and $0.0075 per million tokens. However, these figures are all vendor-measured and have not been independently verified.

Groq has chosen a compromise that keeps weights on the chip while retaining rewriteability. The difference between the two methods lies in how much of the model is permanently fixed, which determines not only speed and efficiency but also model replacement cycles and hardware flexibility.

This trend parallels the shift in Bitcoin mining from CPUs to GPUs, FPGAs, and ASICs. When a specific workload becomes sufficiently stable, it moves away from general-purpose hardware to dedicated silicon, at which point the cost and accessibility of performing the same task change significantly.

The proliferation of dedicated chips for AI inference may partially disperse HBM demand. However, since GPUs used alongside these chips still require HBM, it remains to be seen how much memory supply pressure will be alleviated.