AI BriefingvLLM Token Waste: Cancelled Requests and Retries Leak Up to 175 Extra Tokens
·2026.10.23 10:49
Key point
Unclosed connections after user cancellation or retries cause vLLM to generate up to 175 wasted tokens per task, reducing on-time answer rates to 11%.
Details
- vLLM continues generating tokens after a user presses stop if the application does not hang up the connection, wasting compute resources.
- Leaving the connection open for 5 seconds after cancellation generates approximately 44 extra tokens, while reading to the end and discarding the output generates about 53 extra tokens.
- On longer natural answers, the token leak rises to approximately 175 extra tokens per cancelled request.
- Retries without closing the initial connection produce about 148 wasted tokens per task and drop on-time answer rates from 38% to 11%.
- The primary fix is to close the stream when the application is done with it, preventing duplicate generation during retries.
- Testing was conducted on vLLM using L4 and A10G GPUs with two models and synthetic prompts; llama.cpp, Ollama, TGI, and consumer GPUs were not tested.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
llm-d 1.0 releases production-ready LLM inference stack with multi-tenant QoS and stable APISpeculative Decoding Speeds Up LLMs by Shifting GPU Work from Memory-Bound to Compute-Bound