MTEB: Text Embedding Performance Evaluation Benchmark Released

MTEB (Massive Text Embedding Benchmark) is a benchmark framework designed to comprehensively measure the performance of text embedding models across various tasks.

Key features are as follows:

• Massive scale: Includes 56 datasets across 8 tasks, providing results from over 2,000 models on the leaderboard.
• Multilingual: Supports up to 112 languages, enabling evaluation of model performance across diverse language environments.
• Extensible: New tasks, datasets, and metrics can be added, and users can submit their own models to register them on the leaderboard.

Models are classified into three categories depending on their intended use:

• Maximum speed: Models with low contextual understanding but very fast processing speed (e.g., Glove)
• Balance of speed and performance: Models that offer both efficiency and performance (e.g., all-mpnet-base-v2)
• Maximum performance: Models delivering high performance using billions of parameters (e.g., ST5-XXL, GTR-XXL)

Developers can directly benchmark their own models using the MTEB library and reflect the results on the public leaderboard.