AI Briefing#kv-cache
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The latest AI and developer news about #kv-cache, with the original source and a short summary.
The center of gravity for AI inference is moving from GPUs to the memory hierarchy.
Triggered by GitHub Copilot's billing shift, local LLM hardware is proposed as a solution for AI self-sufficiency.
Random cross-layer attention enables layer-wise KV cache sharing.
FastDMS has been released, optimizing the KV-cache compression technology DMS to achieve less memory usage and faster speed than vLLM.
In mixed traffic, class-aware vLLM routing showed the best balance.
In Tesla P40 benchmarks, the saw-family KV quantization was the most stable.
SGLang has been found to have FP8 KV cache corruption and a memory leak in image requests.
Per-GPU KV cache locality determined LLM serving cost and TTFT.
During large-scale GLM-5 serving, a KV Cache race condition fix and LayerSplit optimization were disclosed.
Re-measured the quality and long-context speed of Qwen 3.6-35B-A3B KV cache quantization on M5 Max.
The Recurrent Transformer boosts both effective depth and decoding efficiency together.
Measured Qwen 3.6 KV cache performance differences by format from 0 to 1M context on M5 Max.
A llama.cpp regression increased the Qwen3.6-27B IQ4_XS size from 14.7GB to 15.1GB.
DeepSeek V4's FP16 KV cache usage was recalculated based on a 1M context.
Tested KV cache quantization for Qwen3.6-27B on a 3090 eGPU with a 200k context.
They released new benchmarks and a harness tailored to the multi-turn, tool-calling patterns of agentic agents.
A Metal kernel implementation combining INT3 model compression with INT2 KV cache has been released.
FlashDrive optimizes the entire autonomous driving VLA inference pipeline, cutting latency from 716ms to 159ms.
Reproduction code and benchmarks for Cartridges and STILL on a single GPU have been released.
Open-source reproduction code for Cartridges and STILL, KV-cache compression techniques for long-context inference, has been released.
Compressed the KV cache on Qwen 3.6's 1M context, reducing it from 10.7GB to 6.9GB.
By sharing Wasm memory with the GPU, inference runs zero-copy on Apple Silicon.
Qwen3.6 adds preserve_thinking, improving reasoning context and KV cache efficiency.
Cloudflare accelerated inference for massive LLMs through PD disaggregation, prompt caching, and KV-cache optimization.