KV Cache Research: Limitations of LRU Alternatives and Failure in Capacity-bound Environments

Failure of LRU Alternatives and Performance Degradation

According to new research, several alternative policies proposed for KV cache management failed to outperform the existing LRU (Least Recently Used) or even degraded performance in real agent workloads. Simulation results based on 393 Claude Code sessions and Mooncake trace data showed that three alternative policies—Hazard-based (H), Cost-based (HC), and Coherent-based (HCG)—all recorded lower hit rates compared to LRU. Specifically, at the 8,000-block threshold, LRU's hit rate was 83.48%, whereas applying the composite policy (HCG) caused it to drop sharply to 68.63%.

LRU Advantage in Capacity-bound Scenarios

The experiments were conducted in Capacity-bound environments with capacity limits, where LRU-based eviction occurred before TTL (Time-To-Live) expiration, rendering TTL-based policies ineffective. The primary cause of re-computation was not TTL expiration of idle sessions, but rather short tool-calling loops with a median interval of 2.1 seconds. The original text emphasizes that these tight loops are the dominant cause of cache misses, accounting for a much higher proportion than requests with longer intervals.

Research Limitations and Implications

The study was a cache-policy-only simulation that did not consider GPU execution time or latency, and there are generalization limitations due to the data scale (393 sessions). The researchers suggested that compression or tiering strategies may be more effective than liveness prediction in capacity-bound environments.