Evaluating OpenWiki Using WikiBench

LangChain developed WikiBench to quantitatively measure the quality of its open-source code documentation agent, OpenWiki. This benchmark evaluates from two perspectives: how much the generated wikis help actual coding agents with their tasks, and whether updates to OpenWiki improve performance.

WikiBench runs on Harbor, an agent evaluation framework, and provides repository environments at fixed commits. The core of the evaluation, the Verifier, uses a 'leader agent' to answer repository-related questions by referring to the wiki. Questions are automatically generated by analyzing the topic areas of the codebase and are divided into 'coverage' and 'retrieval' types.

Evaluation scores are calculated by an LLM judge that verifies the accuracy of facts and the supporting evidence in the answers. This allows for comparing quality differences between wiki versions and measuring the performance gap between providing only the original source code and providing the wiki, thereby demonstrating the practical value of the wiki.