MTEB Leaderboard

MTEB evaluates models across 8 distinct task categories to prevent overfitting to a single metric. This holistic approach ensures a model's embedding quality is robust across diverse real-world applications.

• Classification: Linear probing on sentiment and topic datasets.
• Clustering: Measuring the purity of embedded groups.
• Pair Classification: Semantic textual similarity and paraphrase detection.
• Reranking: Assessing the ability to re-order retrieved documents by relevance.
• Retrieval: Evaluating both symmetric and asymmetric search scenarios.
• STS (Semantic Textual Similarity): Directly measuring cosine similarity against human-labeled gold standards.
• Summarization: Scoring machine-generated summaries against human references.
• Bitext Mining: Aligning parallel sentences across different languages.

The benchmark has expanded beyond its initial English-only focus to provide a unified evaluation framework for multilingual models. This allows developers to select models that perform well on cross-lingual tasks.

• English: The original and most comprehensive set of tasks.
• French & German: Dedicated task suites for high-resource European languages.
• Multilingual: A specific track for evaluating models on bitext mining and cross-lingual retrieval across 50+ languages.

MTEB normalizes scores across tasks to produce a single, comparable leaderboard. The primary ranking metric is the weighted average of normalized scores.

• Normalization: Raw scores for each task are normalized between 0 and 1 based on the historical minimum and maximum values for that task.
• Weighting: Each of the 8 task categories contributes equally to the final score, preventing high-dataset-count tasks like Retrieval from dominating the ranking.
• Reproducibility: The framework uses fixed, public datasets and evaluation scripts, ensuring that leaderboard submissions are verifiable.

The leaderboard is not restricted to a single architecture. It benchmarks a wide variety of model paradigms to guide architectural choices.

• Dense Models: Standard bi-encoders that output a single fixed-size vector per input.
• Sparse Learned Models: Models like SPLADE that generate high-dimensional but sparse lexical vectors.
• Multi-Vector Models: Architectures like ColBERT that produce token-level embeddings for fine-grained late interaction.
• Commercial APIs: Closed-source models from providers like OpenAI and Cohere are benchmarked alongside open-source models.

While the overall ranking is useful, MTEB allows practitioners to filter models based on their specific use case. A model optimized for semantic search may not be the best for clustering.

• Retrieval Focus: Filtering by the Retrieval task suite reveals models that excel at finding relevant documents in a large corpus.
• Clustering Focus: Isolating the Clustering score helps identify models that best represent underlying semantic groups.
• Speed Benchmarks: A companion benchmark measures embeddings per second, allowing users to trade off accuracy for throughput.