Fixing a Race Condition Bug Found During Large-Scale GLM-5 Serving — Scaling Pain in Coding Agent Inference Infrastructure

While serving a GLM-5-based Coding Agent at a scale of hundreds of millions of requests per day, output anomalies that only appeared under high-concurrency, long-context conditions were tracked down.

The anomaly signal was caught using Speculative Decoding metrics. After generated tokens exceeded 128, if spec_accept_length < 1.4 or spec_accept_rate > 0.96, the request was aborted and retried on the load balancer.

• Bug Fix #1: In the PD (Prefill-Decode) disaggregated architecture, a race condition was fixed where the abort signal was not propagated to Prefill, causing the reclaimed KV Cache to be overwritten by a new request.
• This fix reduced the anomalous output rate from 0.1% to below 0.03%.
• Bug Fix #2: In HiCache, a missing synchronization between the Load Stream and Forward Stream, which caused a read-before-ready issue, was fixed.
• This change was contributed to the community as SGLang PR #22811.
• LayerSplit: In a Context Parallelism environment, per-layer KV Cache is distributed across storage to reduce per-GPU memory burden, and broadcast and indexer operations are overlapped to hide communication overhead.
• Under a 90% cache hit rate condition, throughput for requests of 40K~120K tokens improved by 10%~132%.

In conclusion, this article emphasizes that in large-scale LLM serving, beyond simple throughput and latency optimization, accuracy of generated results must also be guaranteed at the system level.