Open-Source Reproduction of KV-Cache Compression Techniques

Single-GPU reproduction implementations of Cartridges and STILL, KV-cache compression techniques for improving memory efficiency in long-context inference, have been released as open source. Rather than just covering the theoretical content of the papers, this implementation focuses on providing readable implementations and comparison code so that researchers and engineers can directly review the code and run benchmarks.

The key implementation details are as follows:

• Cartridges: A compressed KV-cache implementation optimized for a specific corpus.
• STILL: A reusable neural KV-cache compaction implementation.

In particular, the STILL repository includes comparisons of full-context inference, truncation, and the Cartridges approach, allowing for analysis of the practical system trade-offs involved in KV-cache reuse and memory compression.