A Latest Standard for Retrieval Evaluation

TL;DR – We’re excited to introduce the beta version of the Retrieval Embedding Benchmark (RTEB), a brand new benchmark designed to reliably evaluate the retrieval accuracy of embedding models for real-world applications. Existing benchmarks struggle to measure true generalization, while RTEB addresses this with a hybrid strategy of open and personal datasets. Its goal is straightforward: to create a good, transparent, and application-focused standard for measuring how models perform on data they haven’t seen before.

The performance of many AI applications, from RAG and agents to advice systems, is fundamentally limited by the standard of search and retrieval. As such, accurately measuring the retrieval quality of embedding models is a standard pain point for developers. How do you really know the way well a model will perform within the wild?

That is where things get tricky. The present standard for evaluation often relies on a model’s “zero-shot” performance on public benchmarks. Nonetheless, that is, at best, an approximation of a model’s true generalization capabilities. When models are repeatedly evaluated against the identical public datasets, a spot emerges between their reported scores and their actual performance on recent, unseen data.

To handle these challenges, we developed RTEB, a benchmark built to offer a reliable standard for evaluating retrieval models.

Why Existing Benchmarks Fall Short

While the underlying evaluation methodology and metrics (akin to NDCG@10) are well-known and robust, the integrity of existing benchmarks is usually set back by the next issues:

The Generalization Gap. The present benchmark ecosystem inadvertently encourages “teaching to the test.” When training data sources overlap with evaluation datasets, a model’s rating can develop into inflated, undermining a benchmark’s integrity. This practice, whether intentional or not, is clear within the training datasets of several models. This creates a feedback loop where models are rewarded for memorizing test data moderately than developing robust, generalizable capabilities.

Due to above, models with a lower zero-shot rating[1] may perform thoroughly on the benchmark, without generalizing to recent problems. For that reason, models with barely lower benchmark performance and the next zero-shot rating are sometimes beneficial as an alternative.

Misalignment with Today’s AI Applications. Many benchmarks are poorly aligned with the enterprise use cases that developers are constructing today. They often depend on academic datasets or on retrieval tasks derived from QA datasets, which, while useful in their very own right, weren’t designed to guage retrieval and might fail to capture the distributional biases and complexities encountered in real-world retrieval scenarios. Benchmarks which don’t possess these issues are sometimes too narrow, specializing in a single domain like code retrieval, making them unsuitable for evaluating general-purpose models.

Introducing RTEB

Today, we’re excited to introduce the Retrieval Embedding Benchmark (RTEB). Its goal is to create a brand new, reliable, high-quality benchmark that measures the true retrieval accuracy of embedding models.

A Hybrid Strategy for True Generalization

To combat benchmark overfitting, RTEB implements a hybrid strategy using each open and personal datasets:

• Open Datasets: The corpus, queries, and relevance labels are fully public. This ensures transparency and allows any user to breed the outcomes.
• Private Datasets: These datasets are kept private, and evaluation is handled by the MTEB maintainers to make sure impartiality. This setup provides a transparent, unbiased measure of a model’s ability to generalize to unseen data. For transparency, we offer descriptive statistics, a dataset description, and sample (query, document, relevance) triplets for every private dataset.

This hybrid approach encourages the event of models with broad, robust generalization. A model with a big performance drop between the open and the private datasets would suggest overfitting, providing a transparent signal to the community. That is already apparent with some models, which show a notable drop in performance on RTEB’s private datasets.

Built for Real-World Domains

RTEB is designed with a selected emphasis on enterprise use cases. As a substitute of a fancy hierarchy, it uses easy groups for clarity. A single dataset can belong to multiple groups (e.g., a German law dataset exists in each the “law” and “German” groups).

• Multilingual in Nature: The benchmark datasets cover 20 languages, from common ones like English or Japanese to rarer languages akin to Bengali or Finnish.
• Domain-Specific Focus: The benchmark includes datasets from critical enterprise domains like law, healthcare, code, and finance.
• Efficient Dataset Sizes: Datasets are large enough to be meaningful (not less than 1k documents and 50 queries) without being so large that they make evaluation time-consuming and expensive.
• Retrieval-First Metric: The default leaderboard metric is NDCG@10, a gold-standard measure for the standard of ranked search results.

An entire list of the datasets might be found below. We plan to repeatedly update each the open in addition to closed portion with different categories of datasets and actively encourage participation from the community; please open a difficulty on the MTEB repository on GitHub should you would love to suggest other datasets.

Launching RTEB: A Community Effort

RTEB is launching today in beta. We consider constructing a sturdy benchmark is a community effort, and we plan to evolve RTEB based on feedback from developers and researchers alike. We encourage you to share your thoughts, suggest recent datasets, find issues in existing datasets and help us construct a more reliable standard for everybody. Please be at liberty to hitch the discussion or open a difficulty within the MTEB repository on Github.

Limitations and Future Work

To focus on areas for improvement we would like to be transparent about RTEB’s current limitations and our plans for the long run.

• Benchmark Scope: RTEB is targeted on realistic, retrieval-first use cases. Highly difficult synthetic datasets aren’t a current goal but might be added in the long run.
• Modality: The benchmark currently evaluates text-only retrieval. We plan to include text-image and other multimodal retrieval tasks in future releases.
• Language Coverage: We’re actively working to expand our language coverage, particularly for major languages like Chinese and Arabic, in addition to more low-resource languages. In case you know of high-quality datasets that matches these criteria please tell us.
• Repurposing of QA dataset: About 50% of the present retrieval datasets are repurposed from QA datasets, which could result in issues akin to a powerful lexical overlap between the query and the context, favoring models that depend on keyword matching over true semantic understanding.
• Private datasets: To check for generalization, we utilize private datasets which can be only accessible to MTEB maintainers. To keep up fairness, all maintainers commit to not publishing models trained on these datasets and only testing on these private datasets through public channels, ensuring no company or individual receives unfair benefits.

Our goal is for RTEB to develop into a community-trusted standard for retrieval evaluation.

The RTEB leaderboard is accessible today on Hugging Face as an element of the brand new Retrieval section on the MTEB leaderboard. We invite you to ascertain it out, evaluate your models, and join us in constructing a greater, more reliable benchmark for all the AI community.