AI Briefing8x Less Memory Than vLLM, FastDMS Released
Key point
FastDMS has been released, optimizing the KV-cache compression technology DMS to achieve less memory usage and faster speed than vLLM.
Details
FastDMS, an optimized version of the Dynamic Memory Sparsification (DMS) technology announced by NVIDIA and researchers from several universities, has been released under the MIT license.
DMS is a technology that can compress KV-cache by up to 8x through a learned per-head token eviction method. The existing reference implementation had the drawback of being slow, but FastDMS resolves this through kernel optimization.
Key Performance and Features:
- Memory Efficiency: At 8K context, it uses 5-8x less KV memory compared to vLLM BF16.
- Inference Speed: Provides 1.5-2x faster decoding compared to vLLM.
- Physical Memory Reclamation: Rather than just theoretical compression, it uses the Compact DMS method to physically reclaim the memory of evicted slots, saving actual device memory.
This project has been validated on Qwen 3 8B and Llama 3.2 1B models, and the source code and trainer are available via GitHub.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jevper Released: A Jev Classification Wrapper for OpenAI-Compatible Models with Automatic Logprobs Detection
Rootless Linux Sandbox 'Drop' Released for Isolating AI Agents and Third-Party Programs