AI Briefing
KOSign in

Prefixwise: Open-source tool diagnoses vLLM and SGLang prefix cache misses with 19 specific causes

·2026.11.11 10:39

Key point

The MIT-licensed tool analyzes KV-cache events from vLLM and SGLang to pinpoint why requests miss the prompt cache, achieving 32/32 agreement with Anthropic's diagnostics.

Details

  • Open-source tool Prefixwise identifies 19 specific causes for prompt cache misses in LLM inference engines, distinguishing between waste and legitimate misses.
  • Diagnoses match Anthropic's internal miss diagnostics 32 out of 32 times and confirm engine evictions 5 out of 5 times for vLLM and SGLang.
  • Automated fixes via --optimize sort drifting tool lists and add cache markers, reducing prompt costs by 85.6% on Anthropic's API and achieving 99.9% predicted saving on Claude.
  • Integrates with vLLM and SGLang by reading KV-cache events, and supports CI prevention via prefixwise check to fail builds when cache-breaking changes are detected.
  • Overhead is 3 ms per request; data stored locally includes hashes and token counts, with no prompts sent to external services.
  • Limitations include unverified fixes for production traffic and lack of support for OpenAI's diagnostics and DeepSeek's hosted API cache fields.
open-sourceprompt-cachingvllmsglanginference-optimizationllm-infrastructure
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

PyTorchKRFlashStates Improves GLM-5.3-Flash Prefill Throughput by 3.14xRedditMACaQ: Open-source quantizer with custom vLLM kernels and lower low-bit KLDRedditKittenML releases inferencekit to optimize LLM inference on local hardware via coding agentsHacker NewsHuawei releases MACaQ, a model-agnostic quantizer for local inferencePyTorchKRllama.cpp v1.0.0, First Major Release in Three and a Half YearsGeekNewsGitHub Next releases Ace, a local collaborative coding environment for humans and AI agents