Improving OpenSearch Search Quality for AI Agents (Part 2)

Experiments with Rerank and Weight tuning to improve search quality for AI Agents revealed that Rerank actually degrades performance when existing search performance is high. In particular, applying Rerank to Hybrid search, which already includes semantic information, resulted in NDCG@10 drops of -3.8% on the SciFact dataset and -8.9% on the Touche-2020 dataset.

The Dual Nature of Rerank Effectiveness

Rerank is a technique that reorders initial search candidates. It showed a significant performance improvement of +18.9% on the SciFact dataset for BM25-only search, which has weak semantic understanding. However, in states where semantic information is already reflected, such as in Hybrid search, performance degraded because only the order was changed without additional information. This is because the Cohere Rerank model reorders based on semantic similarity.

Weight Tuning and Statistical Significance

Weight tuning is a method of adjusting the normalization weights of BM25 and k-NN scores without additional API call costs. Grid Search results on the Touche-2020 dataset showed that the default 0.5:0.5 ratio (NDCG@10 0.8039) achieved the highest performance among 12 tested ratios. Since the performance difference between BM25 (0.778) and k-NN (0.757) was not significant, the default ratio was analyzed as already being at the optimal point. Paired t-test results showed a p-value of 0.64, indicating no statistically significant difference between the tuned settings and the default settings.

Automation and Evaluation Systems in Production Environments

To address search quality fluctuations, a periodic evaluation loop was built using Amazon EventBridge and Lambda. Representative query sets are queried weekly to calculate NDCG@10, and alerts are sent via Amazon SNS if the score falls below the threshold (0.6). Additionally, a quality gate was inserted into the deployment pipeline to automatically halt deployment if NDCG falls below the baseline (0.65). While improvements to real-time pipelines such as Self-RAG were proposed, it was emphasized that measurement-based application is essential considering increased latency and costs.