Rethinking AI TCO: Why Cost Per Token Is the Only Metric That Matters

Traditional data centers stopped at storing and processing data, but in the era of generative and agentic AI, they have transformed into AI token factories. Now the economics of infrastructure must be evaluated not by GPU specs or FLOPS per dollar, but by cost per token — the actual cost of producing a single token.

The key lies in the denominator of the equation, not the numerator. Cost per GPU hour is a clearly visible input, but the value a company actually captures is determined by tokens per second and the software, network, memory, storage, and ecosystem optimizations that raise it. The more tokens you can extract with less power and higher utilization, the lower the cost per million tokens falls, and the greater the profitability gained from the same infrastructure.

This piece describes this as the "inference iceberg." On the surface, metrics like cost per GPU per hour, peak PFLOPS, HBM capacity, and FLOPS per dollar are visible, but below the surface are the factors that actually affect token output.

• Scale-up interconnect that handles all-to-all traffic for MoE models
• FP4 support while maintaining accuracy
• Latency-reduction techniques such as speculative decoding and multi-token prediction
• Disaggregated serving, KV-aware routing, KV-cache offloading
• Ultra-low latency, high throughput, and support for long input sequences in agentic AI
• Full lifecycle support spanning training, post-training, and large-scale inference

The DeepSeek-R1 case demonstrates the same conclusion. NVIDIA's Blackwell costs roughly 2x more in compute cost compared to Hopper, but in practice shows over 50x tokens per watt and 35x lower cost per million tokens. This gap, which surface-level metrics fail to explain, is why the standard for choosing infrastructure must shift to an output-centric focus.

The final conclusion is clear. When choosing AI infrastructure, you should look at cost per token and actual token output rather than comparing specs. NVIDIA claims that through extreme codesign — jointly optimizing hardware, software, and the partner ecosystem — it lowers token costs, and that even on existing infrastructure, improvements to the open-source inference stack continue to drive efficiency higher over time.