Removing CUDA checks with graph_reused
Key point
llama.cpp's ggml added graph_reused to reduce CUDA's O(n) change checks.
Details
A reused state was added to ggml's ggml_cgraph, allowing backends to more directly determine whether a graph is being reused.
The key point is bypassing the O(n) props change check that the CUDA backend performed every time graph_reuse was called to check for graph changes. In situations where graph reuse is guaranteed, this check is redundant, so skipping it reduces some overhead.
Actual measurement results were also presented.
- Based on RTX 5090
- Tested on
gemma4 ?B Q4_0,gpt-oss 20B MXFP4 MoE,qwen35 27B Q4_K_M,qwen35moe 35B.A3B Q4_K_S, etc. - Generally showed improvements of 1.02x~1.07x
- The largest case was
qwen35moe 35B.A3B Q4_K_Swith 206.56 t/s vs 194.52 t/s
During the review process, discussion continued on whether graph GUID / versioning would be more appropriate than a simple reuse flag, and how to identify split graphs versus the main graph. It was also pointed out that, as with tensor-overrides, multiple splits can occur within a single backend, requiring separate identifiers per split.
Ultimately, this patch is not just a simple flag addition, but an optimization effort that also encompasses internal llama.cpp/ggml design discussions about how to define graph identification and reuse contracts.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.