LangChain tests OpenWiki and evidence graphs for agent memory, finding 5x accuracy gain on cross-source questions

• OpenWiki, a compiled wiki memory layer, made agents 5x more accurate on connected questions than keyword search alone, raising accuracy from 0.10 to 0.52.
• Adding a lightweight evidence graph that materializes wiki citations as typed edges improved accuracy by a further 27%, doubled citation consistency, and reduced cost per correct answer by 58%.
• The evidence graph was built deterministically from existing document references and wiki citations without additional model calls, using locate and expand tools for navigation.
• For simple lookup questions, the wiki provided no accuracy benefit over search (0.83 vs 0.84) and increased latency, making memory layers most valuable for repeated cross-source reasoning.

Experiment Setup

• Tested on a synthetic library of 111 insurance documents with a ledger of 972 facts to control cross-document dependencies.
• Evaluated 80 simple lookup questions and 50 connected questions, each asked three times to measure run-to-run variation.
• Metrics included accuracy (checked against the fact ledger), consistency (Jaccard similarity of citations), and cost/latency per question.

Cost and Limitations

• Compiling the 111-document library into a wiki cost $4.15; adding 17 new documents later cost about $0.09 each.
• Reported answer costs exclude compilation and update costs, focusing only on retrieval and answering.
• The evidence graph improves retrieval consistency but does not make the agent fully deterministic; it relies on the accuracy of the wiki's model-generated citations.