Caching for Agentic LLM Pipelines

Agentic workflows require caching for evaluation experiments due to their complexity, dynamic paths, long execution times, and high cost, but LLMs have non-determinism, producing different results even with the same prompt. So a simple cache toggle alone can neither reproduce experiments quickly nor deliberately introduce variability.

The problem was that researchers wanted the cache to exhibit two opposite behaviors at once: reproducibility and variability. On re-runs, previous results should be reused to run quickly, but when trying multiple times, as in best-of-N sampling, to measure performance variance and Success@K, each branch needs to produce different results.

Existing cache key designs failed at both. Using only the step name causes different Generate branches to share the same key, while the step name + counter approach depends on the order in which executions finish, mixing up input-output pairs. Especially in a composition structure where LLM calls take each other as inputs, the completion order differs every time, so the correspondence between the same prompt and response is not guaranteed.

The solution was to make the cache key encode graph position rather than execution order. That is, the key includes where each LLM call is located in the pipeline, which branch (run index) it belongs to, and which upstream outputs it consumed, so that even on re-runs, only calls at the same position share the same cache, regardless of completion order.

With this approach, branches like gen_0 and gen_1 keep different results, and validation/fix steps built on top of them inherit their upstream output identity and connect consistently. As a result, it became possible to run experiments where only one change is made and the rest is reused from cache, run A/B tests based on the same upstream output, and reproduce best-of-N without the branches collapsing into the same answer.