InfiniteKV: Open-Source KV Cache for Infinite Context

Existing LLM serving methods use a sliding window approach that deletes old tokens to solve VRAM shortage problems, but InfiniteKV takes a completely new approach to this.

This technology separates memory into two regions. The most recent 256 tokens are pinned in GPU memory to maintain performance, while earlier tokens are converted into 104-byte compressed records and stored in regular RAM or on disk.

Key Features and Performance:

• Memory Efficiency: For processing 1 million tokens, the existing float16 approach requires about 122GB of VRAM, but InfiniteKV uses only about 3GB of records, making it possible to run on a regular PC.
• Retrieval-Based Attention: Every time a new token is generated, it retrieves the most relevant past tokens from the stored records and includes them in the model's context.
• Verified Performance: In tests on the Mistral-7B model, it was confirmed to generate accurate answers even at the 76,747th token, far exceeding the trained context window (32,768).
• Data Integrity: All results are provided in the form of JSON recipes, ensuring reproducibility of the experimental environment and results.