Smaller, Faster, Safer: Running Kimi and GLM at Scale

Cloudflare Workers AI serves large-scale, long-context, Mixture-of-Experts models like Moonshot's Kimi K series and Z.ai's GLM on GPUs in data centers close to users. Both models perform excellently, but their large GPU memory requirements make efficient serving difficult.

Cloudflare applied three optimizations to its SGLang-based inference environment.

• KV cache quantization: Using FP8 instead of BF16 halved the KV cache size. For Kimi K2.6, the context that can be held in memory increased from about 686,000 tokens to 1.37 million tokens.
• Model weight compression: GLM 5.2's weights were reduced from FP8 to INT4, shrinking the checkpoint from 705GB to 421GB.
• Shared cache protection: A method was applied to protect the shared cache while accommodating more requests on the same hardware.

In a disaggregated H200 decoding environment for Kimi K2.6, FP8 KV cache showed slightly lower per-request throughput than BF16, but while BF16 ran out of memory at 32 concurrent requests, FP8 handled up to 64 requests. As a result, maximum throughput reached 2,192 tokens per second, about 41% higher than BF16's peak, and cost per token dropped by about 30%.

Given that prefill is compute-bound and decode is memory-bound, Cloudflare kept BF16 cache in the prefill pool and applied FP8 to the decode pool. Evaluation results showed virtually no performance difference between FP8 and BF16 KV cache, and accuracy was maintained on key benchmarks including GSM8K, ARC, MMLU, and tool-calling validity.

All experiments and production traffic ran on SGLang, the open-source inference serving framework. Cloudflare is collaborating with the SGLang team to upstream related patches and features into the open source project.