Evaluating Search Quality for AI Agents with OpenSearch (Part 1)

In RAG (Retrieval-Augmented Generation) systems, search quality is a core factor determining overall answer performance. Since poor retrieved documents lead to incorrect answers regardless of LLM performance, objective evaluation through quantitative metrics rather than "gut feeling" is essential.

Search for AI agents differs in nature from search performed directly by humans. Agents generate queries themselves, directly utilize search results as LLM context, and search results determine the upper limit of answer performance. Therefore, search for agents must be measured more coldly and rigorously.

The following two metrics are used as key evaluation indicators:

• NDCG@10: Measures the quality of the top 10 results by simultaneously considering the ranking and relevance of search results.
• Recall@10: Measures the proportion of correct answers included in the top 10 out of all ground truth documents to check for search omissions.

The system utilizes Amazon OpenSearch Service as the search engine and Amazon Bedrock as the embedding and scoring tool. It adopts a strategy of first validating the evaluation pipeline through the BEIR Benchmark before applying it to actual service data.

The scoring process introduces the LLM-as-a-Judge approach. By using Amazon Bedrock (Claude) to automatically score the relevance of search results on a scale of 0 to 1, it solves the cost and time issues associated with manual labeling and performs consistent evaluations.