Predictive Speculative KV Replication for Bursty LLM Inference

LLM inference services can experience sudden bursts of requests sharing the same long prefix, such as in data labeling or multi-agent fan-out scenarios. The authors point out that public traces do not sufficiently reflect this pattern, and that existing GPU routing policies are inefficient in such environments.

The GitHub project Biting the Bullet (BTB) predicts large-scale request bursts and then proactively replicates the already-computed prefix KV cache into another GPU's HBM via RDMA. Instead of moving the cache or re-running prefill after a request arrives, it places the cache in GPU memory in advance to reduce initial response latency.

For Llama-3.3-70B, the KV cache is about 320KiB per token, so a 1,000-token prefix corresponds to about 320MiB. Since cache reuse requires an exact match of the same prefix, even a single differing token early in the prefix breaks the sharing.

The authors report the following processing times for an 8,000-token prefix on a 4×H100 node setup:

• GPU HBM: about 0.20ms
• Host RAM: about 11.9ms
• RDMA remote GPU: about 13.1ms
• NVMe: about 374ms
• Recomputation (prefill): about 571ms

Compared to SGLang's default cache-aware router, BTB reported reducing average time to first token by 10-60%, and under optimal conditions, reducing p95 time to first token by up to 80-82%. The key premise is the ability to detect repeated long prefixes and bursty arrivals in advance.