Evaluating Code Review Agents with ReviewBench

ReviewBench is a benchmark that evaluates the performance of code review agents based on issues found in real code reviews. The team collected feedback from trusted reviewers on merged PRs in the LangSmith monorepo, then selected only substantive defects and converted them into reproducible Harbor tasks.

The original review comments included minor nitpicks and questions mixed in, so they couldn't be used as answer keys as-is. The team applied an LLM gate and manual review to keep only comments that were newly introduced issues from the change and were verifiable.

The benchmark includes codebase-context-based issues such as the following:

• SQL queries that look up or delete database resources by ID alone, without a tenant condition
• Endpoint migrations that changed behavior by omitting filters from the existing API
• Defects that require inferring implicit system contracts from surrounding code, rather than just looking at the changed code

ReviewBench currently consists of 59 tasks and 64 baseline issues. Each task provides a fixed PR context and a local GitHub stub, and the agent inspects the full repository before submitting a structured review that includes the issue location, title, and description.

Evaluation is conducted using coverage, which measures how many baseline issues are found, and precision, the proportion of correct items among the submitted findings. A hidden verifier compares results using an LLM-as-a-judge approach, and the final score is an F1 that weights both metrics equally. Model comparisons were performed using the same Deep Agents harness with 3 attempts per task, without using a review-specific system prompt.