AI Briefing#kv-cache
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The latest AI and developer news about #kv-cache, with the original source and a short summary.
While an LLM context window represents the total token capacity for a single request, the 'Lost in the Middle' phenomenon causes retrieval rates for middle information to drop with longer inputs, making selective management crucial.
LLMTraceFX has introduced an independent auditor to verify the reliability of cache hits, establishing evidentiary standards for whether computation is omitted.
DeepSeek released the 552B MoE model V4.1-Flash, replacing V4-Pro and optimizing KV cache to 890 bytes per token.
Pinterest built a VLM serving stack using NVIDIA Dynamo and Blackwell GPUs, improving TTFT by 85x.
Research results have been published demonstrating that no KV cache policy outperforms LRU in simulations of real agent workloads.
Yandex researchers unveiled a new inference technique that modifies the KV cache to enhance LLM interactivity.
Salesforce has released Random Attention, which applies random sampling instead of signal-based methods for KV cache eviction in reasoning models.
The NInfer fork supports a 555k token context window on the RTX 5090 via NVFP4 KV cache and YARN extension.
The 'Declarative Attention' technique has been released, allowing language models to declare areas requiring attention during their reasoning process, reducing computational costs by up to 52%.
oMLX, a native LLM inference server for Mac, has been released, reducing agent wait times to 5 seconds using SSD-based KV caching.
Ollama's KV cache quantization settings are applied globally, which can cause performance degradation for specific models.
PagedAttention applies virtual memory concepts to significantly reduce KV cache memory waste during LLM inference.
vLLM's non-standard KV cache structure is reported to hinder inference interoperability across heterogeneous hardware such as Cerebras.
Test-time training updates weights during inference to reduce memory and increase speed, but incurs the computational cost of per-user model replication.
Analyzed the KV cache capacity and memory requirements of the Qwen3.8-27B model based on context length.
WorldTrace is a framework that helps video world models accurately remember and utilize past visual information even when the generation length exceeds the training scope.
Quantization, KV cache compression, and inference chips reduce the cost and latency of agentic AI.
Cloudflare reduced GPU memory usage and increased throughput for Kimi and GLM.
BTB predicts repeated-prefix bursts and proactively replicates KV caches into GPU HBM.
BeeLlama.cpp, a fork of llama.cpp, has released v0.4.1, which improves KV cache quantization efficiency.
CachyLLama, a fork of llama.cpp, has dramatically improved prompt processing speed with SSD-based KV caching.
BeeLlama v0.4.0, a fork of llama.cpp with improved KV cache quantization performance, has been released.
A new KV cache technique that stores verified knowledge as KV states and restores it without recomputation was announced.
It's an LLM server that maximizes inference efficiency on low-spec CPU environments through KV cache reuse and n-gram archives.