AI BriefingSGLang fixes FP8 KV corruption and memory leak
·2026.05.01 21:29
Key point
SGLang has been found to have FP8 KV cache corruption and a memory leak in image requests.
Details
AI Router Switzerland summarized two defects found in SGLang across 2 PRs.
- PR #24198 fixed an issue in the ragged+paged path of
forward_extend()wherek_scale/v_scalewere dropped when a radix cache prefix hit occurred. In FP8 KV cache, this could silently produce incorrect output, and the April 2025 fix applied to the non-ragged and decode paths was missing from the ragged+paged branch. - This bug can affect FP8 models that store per-layer KV cache scales in the checkpoint, such as Qwen3.6-27B-FP8, DeepSeek-V4, and Gemma 4.
- PR #24182 fixed a leak in Qwen-VL-family image requests where
release_features()only freedpixel_valuesand missedmrope_positions,mrope_position_delta, andmrope_position_delta_repeated_cacheon the GPU, causing memory to accumulate. This PR is under review. - With repeated image requests, GPU memory can keep growing and eventually lead to OOM.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Predictive Speculative KV Replication for Bursty LLM Inference
Hierarchical Memory Architecture Needed to Resolve AI Inference Bottlenecks