AI BriefingCachyLLama Supports SSD-Based KV Caching
·2026.07.25 22:11
Key point
CachyLLama, a fork of llama.cpp, has dramatically improved prompt processing speed with SSD-based KV caching.
Details
This is an llama.cpp fork project developed to solve the Prompt Evaluation bottleneck that occurs when running local agents (Aider, Claude Code, etc.).
Key features are as follows:
- Persistent On-Disk KV Cache: Saves conversation state to SSD, preserving state even after server restarts or power loss, and instantly restores from disk on cold start.
- Dedicated System Prompt Cache: Maintains a global cache for Static Prefixes, skipping repeated system prompt re-evaluation.
- Hybrid MoE/SSM Support: Accurately tracks and restores recurrent state and attention cells for the latest architectures such as Qwen 3.5/3.6, Gemma 4, and DeepSeek-V3.
- Multi-Tiering: Keeps active states in RAM, offloads idle sessions to disk, and overlaps disk I/O with computation via kernel readahead.
Benchmark results showed that for a prompt of about 15,700 tokens, a task that previously took 143.1 seconds (Cold Start) was reduced to 0.99 seconds (Warm) when the cache was utilized.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.