Building the foundation for running massive LLMs

Cloudflare redesigned its infrastructure to run massive open-source models like Moonshot Kimi K2.5 faster and more efficiently on Workers AI. Following the release of the public model, it re-analyzed usage patterns and shifted to a structure optimized for input token processing and tool calling.

The biggest change is prefill decode disaggregation (PD disaggregation). prefill is the stage that processes input tokens and fills the KV cache, while decode is the stage that generates output; the former is compute-bound and the latter is memory-bound, so handling both on a single GPU ties up resources inefficiently. So Cloudflare set up separate inference servers that split prefill and decode, and uses token-aware load balancing to distribute the number of in-flight tokens across each endpoint.

This structure not only forwards requests but also has to write cached token information from the prefill server back into the decode server's response and streaming SSE, making the load balancer more complex as well. But the results were clear. After moving traffic to the new PD disaggregated architecture, p90 Time to First Token dropped, and even as request volume increased, tail latency variance was greatly reduced with the same number of GPUs. p90 time per token also dropped from around 100 ms to about 20~30 ms, achieving roughly a 3x improvement.

For agentic workloads, Prompt Caching was also strengthened. Given the tendency to repeatedly send long contexts, when a client sends an x-session-affinity header, it routes to the region that previously computed the input tensor, reducing prompt recomputation. Internally, KV-aware routing is used, while externally discounted cached tokens are offered to encourage use of the header, raising the cache hit ratio for core users from 60% → 80%.

As models grew larger, a single instance began to span multiple GPUs, and KV-cache sharing became another bottleneck. To address this, Cloudflare leveraged Moonshot AI's Mooncake Transfer Engine and Mooncake Store to support direct memory transfer between GPUs, combining them with LMCache or SGLang HiCache to share cache across the entire cluster. This reduced reliance on session-aware routing and allowed the cache to be extended out to NVMe, keeping it alive longer.

To capture both inference quality and speed, speculative decoding was also applied. A small draft model proposes candidate tokens first, and the larger target model verifies them—a method that's especially effective for outputs with predictable structure, such as JSON-form tool calls. For Kimi K2.5, an NVIDIA EAGLE-3 draft model is used, adjusting the number of future tokens to raise tokens per second while maintaining high quality.

The core engine, Infire, was also significantly expanded for massive models. Infire is an inference engine written in Rust to match Cloudflare's distributed global network, and this time it added multi-GPU support, now handling pipeline parallelism, tensor parallelism, and even expert parallelism. Memory overhead was further reduced, so that Llama 4 Scout secures over 56 GiB of KV-cache headroom on 2 H200s, and Kimi K2.5 was reported to run on 8 H100s while still leaving over 30 GiB free.

Boot speed was also improved, so even the largest models can start handling requests within 20 seconds, with load time now effectively bounded by drive speed. As a result of these optimizations, Cloudflare achieved a 20% improvement in tokens per second throughput on unconstrained systems, and is now able to run the latest models on hardware combinations that were previously not possible.