AI BriefingShard Released, Compressing KV Cache by 10x
Key point
Shard has been released, compressing the KV cache memory of Llama-3.1-8B by about 10x without performance degradation.
Details
Shard is a drop-in replacement for HuggingFace Cache that dramatically reduces KV cache memory usage for the Llama-3.1-8B model.
It provides a compression ratio of about 10x at 8K context and about 11x at 32K context, and operates without meaningful performance degradation on benchmarks such as NIAH (Needle In A Haystack) and LongBench.
The key technical features are as follows:
- K (Key) matrix: After removing RoPE, PCA (Principal Component Analysis) and int4 quantization are applied to process it as a low-dimensional matrix.
- V (Value) matrix: Hadamard rotation and Vector Quantization are applied.
- Direct computation: During attention computation, operations are performed directly on the compressed K states, without a reconstruction step back to fp16.
The project's source code is available on GitHub (krish1905/shard).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
'Cache-to-Cache' Framework Revealed for Direct KV-cache Communication Between LLMs Without TextPBuilding a Pinterest VLM Serving Stack Based on NVIDIA Dynamo