Combining KV Cache Offloading and EPP Routing Boosts LLM Inference Throughput by 2.58x
·2026.11.16 21:14
Key point
In a 32-GPU environment, processing time was reduced from 121.7 minutes to 47.2 minutes.
Details
- Achieved 2.58x throughput (1.219 req/s) in a 32x A100 80GB environment by combining KV cache offloading (LMCache) and EPP routing
- With EPP routing, the GPU cache hit rate increased from 27.7% to 90.5%, and recomputed tokens decreased from 25.3 million to 3.4 million
- Applying offloading alone yielded 1.19x throughput, while routing alone yielded 1.15x, showing that the combined approach is more effective than either individually
- E2E median latency improved from 141.35 seconds to 48.42 seconds, and TTFT median improved from 22.33 seconds to 10.26 seconds
Technical Configuration and Experimental Results
- Experiment 1 (Single Node): Applying CPU/disk cache and MP method improved throughput by 3.56x compared to GPU cache, and reduced the recomputed input token ratio from 81% to 13.5%
- Experiment 2 (Multi-Pod): In 6 pods with 2x H100 80GB each, applying EPP routing achieved 1.87x throughput compared to Round Robin
- Experiment 3 (Combined): In 4 pods with 32x A100 80GB each, combining EPP approximate + offloading achieved 2.58x throughput, and TPOT median decreased from 224.91ms to 65.48ms
Bottleneck Analysis and Improvements
- Applying O_DIRECT and 8 threads for disk reads improved I/O speed from 2.6 GB/s to 4.74 GB/s, and fixing NUMA sockets improved copy speed by 1.90x
- PCIe placement issue: NVMe and specific GPUs (2, 3) are located under the same PCIe root port, causing up to a 91% reduction in GPU KV copy speed when writing to memory on the opposite socket
- EPP routing methods: The approximate method achieved 2.58x through index capacity expansion, while the precise method was limited to 1.30x due to not reflecting CPU/disk status
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.