Pie Released: LLM Serving System That Lets User Code Control KV Cache and Generation Loops

Pie, unveiled by Yale University researchers at SOSP 2025, is an open-source LLM serving system where user programs (Inferlets) directly control the KV cache, forward passes, and sampling, rather than relying on fixed generation loops. It was designed to address the limitation of existing systems like vLLM and SGLang, which manage the KV cache solely through system-wide policies, preventing direct application-level control.

Architecture and Performance

Pie provides 42 APIs that separate the forward pass into embed, forward, and sample stages, consisting of a WebAssembly-based application layer and a GPU execution inference layer. Testing the Llama 3 1B model on NVIDIA L4 GPUs showed that Pie reduced latency by up to 15% and increased throughput by up to 30% for agent workloads with frequent external I/O (such as ReACT). While specific optimizations (encoding application knowledge in code) improved throughput by 3.5x compared to vLLM, this is distinct from the baseline performance (approximately 1.5x that of vLLM).

Programmability and Overhead

For general text generation, Pie exhibits an additional latency of approximately 1.5–1.9ms per token compared to vLLM (the "programmability tax"). To address this, version 0.4.0 introduced Speculative Execution to pre-prepare the next forward pass, claiming to eliminate 90% of the overhead, though specific TPOT improvement figures were not disclosed. The current version is 0.5.2; the project FAQ defines Pie as a research prototype not yet ready for production, and multi-node inference has not yet been announced as a release.