AI Briefing#kv-cache
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The latest AI and developer news about #kv-cache, with the original source and a short summary.
A ByteDance Seed paper reveals that DeepSeek V4's chunked KV-cache compression creates periodic retrieval failures, with base model accuracy varying by 40.2 percentage points depending on token position.
vLLM Router provides routing functionality that distributes inference requests while considering sessions and caches.
To resolve memory bottlenecks caused by increasing KV Cache, a hierarchical structure utilizing layers from HBM to NVMe SSD is required.
The model resolves slow-channel glare and write sensitivity bottlenecks to train 2.7x faster than Splash Attention at 32K context.
Modular has released a free 57-page handbook covering LLM inference metrics, optimization, and operational costs.
By optimizing the memory hierarchy across GPUs, CPUs, and storage, NVIDIA achieves a 30x increase in agentic throughput per dollar with only a 3% rise in HBM capacity.
The method identifies that FP4 errors stem from low-entropy tokens and uses trace-aligned training to reach 65.94% AIME-120 accuracy on R1-Qwen-14B.
The method maintains near-full-KV accuracy while supporting 700B scale models at 96-99% of DSA throughput.
A known configuration mismatch and an SGLang bug caused 155 GLM-5.3 responses to contain data from other customers' requests.
The system prioritizes full-attention KV cache admission to reduce SSD wear without sacrificing cache hit rates.
VHA uses linear operations to halve physical query heads, achieving 4x KV cache reduction and +0.70% quality improvement over MLA on a 30B MoE model.
The study benchmarks KV cache limits and conversation state management strategies for agent model switching.
The open-source DeepInfer framework supports NVIDIA and Huawei Ascend platforms with a single-cluster throughput of 155 trillion tokens per day.
Pie is an open-source LLM serving system where user programs (Inferlets) directly control the KV cache and generation loops, reducing latency by up to 15% for agent workloads like ReACT.
While performance is similar under default settings, the optimal engine varies depending on output length and prefix reuse.
The update delivers up to 138% token generation per watt improvements for MoE models and introduces experimental hierarchical KV cache offloading.
The new variants achieve a 6.58% increase in SWE performance and use 20% fewer reasoning tokens while compressing the KV cache by 4x.
The new NIXL connector-based push method reduces Time To First Token by up to 3x and Time Per Output Token by up to 30% compared to pull-based approaches.
The Apache-2.0 licensed model uses a hybrid architecture with 54 KDA layers and 17 BCSA layers to minimize KV cache usage while supporting a 262K context window.
The platform processes approximately 1 trillion tokens daily internally and offers serverless and reserved capacity options.
Transitioning to a hierarchical architecture utilizing HBM, Host DRAM, and NVMe SSD is essential to resolve AI inference memory bottlenecks caused by the proliferation of long contexts and RAG.
The open-source tool 'phantom-kv' has been released, bypassing LLM censorship by injecting approximately 18MB of KV cache without modifying model weights.
The 'Gewell' inference engine, featuring lossless KV cache reduction for the Gemma 4 31B model, has been released, improving VRAM efficiency and throughput in Blackwell GPU environments.
The C2C framework has been released, directly connecting KV-caches between LLMs without text generation to reduce latency and improve accuracy.