The Heap Lies: Debugging a vLLM Memory Leak

While testing the Mistral Medium 3.1 model using vLLM, the Mistral AI team discovered that memory usage was increasing linearly by 400 MB per minute in a disaggregated serving environment. This issue only occurred under specific conditions where graph compilation was enabled.

The investigation revealed that the problem only appeared in the Prefill/Decode (P/D) disaggregation setup. This approach splits query processing into two stages:

• A Prefill instance computes the request's KVCache.
• The router passes the KVCache metadata to the Decode instance.
• KVCache transfer begins via NIXL, and token generation takes place on the Decode instance.

The memory leak was observed only on the Decode side, suggesting that the issue lay in the KVCache transfer process via NIXL. NIXL relies on the UCX (Unified Communication X) library for data exchange in distributed systems, which supports high-performance technologies such as Infiniband.

Initially, Python profiling tools such as Memray and Guppy 3 were used, but they failed to detect the leak. GDB crashed the process, and Valgrind was unusable due to the heavy load of the vLLM environment.