AI Briefing#kv-cache
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The cause of KV cache INT4 collapse in some models was identified and fixed with a 4-line correction.
Ran Gemma 4:26B on a Mac Studio M4 Max using OC + LM Studio, achieving 40 tok/s.
A high-resolution CLI profiler for vLLM that catches batching, KV, and cache issues.
It argues that storing only the residual stream, without a KV cache, can produce identical outputs.
Splitting core/scratch compresses reasoning KV cache smaller and more accurately.
At the Seoul meetup, participants confirmed the trend of vLLM growing into standard infrastructure for LLM serving.
KV cache compression enables memory sharing across multi-agents, cutting tokens by up to 49%.
TriAttention compresses the KV cache by 10.7x and improves throughput by 2.5x through trigonometric-based compression without loss of accuracy.
Mistral AI shared the technical process of tracking down a memory leak that occurred in a distributed serving environment of vLLM.
Lowering AI inference costs requires optimizing every layer, from the model to scheduling.
This explains the mechanisms and key performance metrics of Prefill and Decode, the core stages of LLM inference.
NVIDIA has released KVPress, a KV cache compression toolkit for handling long context in LLMs.
KV Cache quantization reduces LLM memory usage and enables longer context generation.
Llama-2-70B is cheap for prompt processing, but it's at a disadvantage compared to GPT-3.5 for completion-heavy tasks.
Separating Encoder, Decoder, and LLM and applying llm-d and USP optimization increased throughput by 2.1x.