Enterprise Internal RAG Benchmark Released

EnterpriseRAG-Bench is an open benchmark that aims to evaluate RAG performance on messy documents resembling real internal company knowledge environments.

• They created a synthetic company called Redwood Inference and generated about 500,000 documents.
• The document sources are Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, Confluence.
• They first define a common company setting, then align teams, products, markets, and internal terminology.
• Next, they build scaffolding such as initiatives, an employee directory, folder structures, and agents.md.
• Finally, they generate PRDs, Slack discussions, meeting notes, tickets, PRs, and customer notes that are interconnected, reproducing dependencies and reference relationships between documents.

The key point is that it presents an evaluation standard closer to a real company's internal search and reasoning environment, rather than benchmarks centered on public data.