AI BriefingFlashMemory DeepSeek-V4 Retriever (GitHub Repo)
Key point
A lightweight retriever that compresses DeepSeek-V4's CSA KV-cache to maximize GPU memory efficiency has been released.
Details
FlashMemory DS-V4 Retriever is a lightweight retriever that sparsifies DeepSeek-V4's Compressed-Sparse-Attention (CSA) KV-cache.
Based on the hidden state of the decode token, it predicts which CSA KV-cache chunks the next approximately 64 tokens will reference. Only the top-scoring chunks are kept on the GPU, while the rest are offloaded to CPU or disk to reduce memory usage. This allows performance similar to full-attention methods to be maintained while keeping only 10-15% of the KV-cache on-device.
Key Features and Architecture:
- Ensemble Approach: Sigmoid scores computed from 3 independent CSA layers (l10, l12, l20) are combined via
maxormeanmethods to make the final decision - Data Format: Each chunk consists of a 132-byte
uint8format, including a 128-byte float8_e4m3 quantized key value and a 4-byte dequantize scale value - Efficient Inference: Precise scoring is performed using queries (q) that have undergone YaRN-based RoPE and Hadamard transforms
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
vLLM v0.28.0 Released with Major Optimizations for Kimi-K3 and DeepSeek V4
Underdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLXLLMTraceFX Introduces Independent Auditor to Verify That Cache Hits Do Not Prove Computation Omission
KV Cache Research: Limitations of LRU Alternatives and Failure in Capacity-bound Environments