How AI Tokens Travel Through Data Centers

Every AI service is essentially an Inference engine that predicts the next token. As of 2026, the mainstream of AI computing has fully shifted from Training to Inference, and the scale has grown so large that Google now processes 3.2 quadrillion tokens per month.

This article traces 15 physical and logical points inside a data center, from the moment a prompt is entered to the moment the answer is output, to dig into how inference infrastructure works. The key stages are as follows:

• Gateway & Scheduler: Receives requests and allocates them to appropriate resources
• KV Cache: Stores computed values from previous tokens to prevent redundant computation
• GPU & Network: Where actual model computation is performed and data moves between nodes

Inference infrastructure is no longer just an abstract concept—it has become a core engineering domain where you must calculate Latency and compute cost per token to assess investment efficiency.