RAG Evaluation, the Current Approach Is Wrong

RAG evaluation points out that current evaluation criteria fail to properly reflect real-world usage environments. Retrieval quality diverges depending on how documents are split, yet existing evaluations often skip this core decision and simply compare scores.

A representative pitfall is document chunking. Splitting documents into large chunks preserves context but blurs details, while splitting them too finely captures facts well but breaks the narrative flow. The setting that produces the best evaluation scores and the setting that actually works well in a real service can be different.

Another problem is the assumption that an answer exists neatly in a single document. Real information is scattered across multiple documents, and sometimes an answer only emerges when you read relationships, frequency, and overall context together. For example, for a Seinfeld fan, George's alias is Art Vandelay, but the answer isn't revealed in a single sentence—it emerges from context accumulated across multiple episodes.

So RAG evaluation needs to examine the following:

• How well document chunking is handled
• How well scattered information is woven together
• Whether relationships between documents are understood
• Whether the big picture is captured without losing it to details

As the team of Niv Granot at AI21 Labs observed, even raising evaluation scores alone may not satisfy real users. Approaches like Microsoft's GraphRAG point toward a direction beyond simple text splitting, but unless the evaluation criteria themselves are adjusted to match reality, the same failures will repeat.