Hugging Face Unveils RTEB, a Retrieval Benchmark

RTEB (Retrieval Embedding Benchmark), a new standard for accurately measuring retrieval quality—a key factor determining the performance of RAG, AI agents, and recommendation systems—has been unveiled.

Existing benchmarks have suffered from a Generalization Gap problem, where models memorize publicly available evaluation data and score higher than they actually should. To address this, RTEB uses the following hybrid strategy:

• Open Datasets: Datasets are fully disclosed for transparency and reproducibility.
• Private Datasets: Private datasets are used to verify a model's true generalization ability, with evaluations conducted directly by MTEB maintainers to prevent bias.

RTEB is designed to reflect the complexity of real-world enterprise environments. By comparing performance differences between open and private data, developers can clearly identify how robust a model is on new data or whether it has overfitted to specific data.