Prefill-as-a-Service: Moving Next-Gen Models' KVCache Across Data Centers

In large-scale LLM serving, prefill-decode(PD) disaggregation has become the standard architecture, but the actual deployment boundary is still determined by KVCache transfer. Traditional dense-attention models generate KVCache traffic so large that prefill and decode had no choice but to be tied together within a single high-bandwidth network domain, which in turn limited heterogeneous deployment and elastic resource operation.

Recent hybrid-attention family models significantly reduce KVCache size, opening up the possibility of cross-datacenter transfer. However, a smaller KVCache doesn't immediately make this practical — real workloads also add burstiness, variance in long request lengths, uneven distribution of prefix cache, and fluctuating inter-cluster bandwidth. Blindly externalizing all prefill leads to congestion, queueing instability, and low resource utilization.

In response, the proposed Prefill-as-a-Service(PrfaaS) takes the approach of offloading only long-context prefill to a separate compute-dense prefill cluster, and delivering the generated KVCache to the local PD cluster over commodity Ethernet. The key is not to rely solely on model-level KV efficiency, but to jointly apply selective offloading, bandwidth-aware scheduling, and cache-aware request placement at the system level.

The mechanism works as follows.

• length-based threshold routing sends only sufficiently long requests to the prefill cluster.
• A bandwidth-aware scheduler detects link status fluctuations and distributes load before congestion builds up.
• A global KVCache manager and hybrid prefix-cache pool jointly consider request length, cache location, and cross-cluster bandwidth.

This design allows heterogeneous accelerators to scale prefill and decode capacity independently of each other, without needing to share the same low-latency RDMA fabric. In other words, instead of forcibly binding everything into one tightly coupled cluster, roles can be separated across loosely connected clusters, data centers, or even regions.

In the case study, a 1T-parameter hybrid model was evaluated following the Kimi Linear architecture. A heterogeneous deployment combining a standalone PrfaaS prefill cluster with a conventional PD decode cluster achieved 54% higher serving throughput than homogeneous PD, and 32% higher than a naive heterogeneous baseline, while cross-datacenter bandwidth usage remained at a modest level.

In conclusion, cross-datacenter serving of next-generation models does not hold up simply because a model has reduced its KVCache. To make it practical, KVCache reduction must be paired with system design that also covers request selection, bandwidth-aware scheduling, and cache placement.