Mean Reciprocal Rank (MRR)

The core calculation is the multiplicative inverse of the rank position of the first correct answer.

• Formula: For a single query, the reciprocal rank is 1 / rank_i, where rank_i is the position of the first relevant item.
• Example: If the first relevant document is at position 1, the score is 1. If it is at position 5, the score is 0.2. If no relevant document is found, the score is 0.
• Key Insight: The metric heavily penalizes systems that bury the correct answer deep in the list, as the score drops hyperbolically.

MRR is uniquely suited for tasks where a user needs exactly one correct answer and will stop looking once they find it.

• Use Case: Common in Question Answering (QA) systems, FAQ retrieval, and known-item search.
• Contrast with Recall: Unlike Recall@K, MRR does not care about the total number of relevant documents in the list, only the rank of the earliest one.
• User Model: It assumes a user with a high precision requirement who browses the list sequentially from top to bottom.

The 'Mean' in MRR refers to averaging the reciprocal ranks over a complete test set of queries.

• Calculation: MRR = (1 / |Q|) * Σ (1 / rank_i), where |Q| is the total number of queries.
• Sensitivity: A single query with a very poor rank (e.g., 100) contributes a tiny value (0.01), while a top-1 hit contributes a full 1.0. This makes the metric robust to outliers but highly sensitive to top-3 performance.
• Interpretation: An MRR of 0.5 implies that, on average, the first relevant document appears at rank 2.

MRR operates strictly on binary relevance judgments (relevant or not relevant) and ignores graded relevance scores.

• No Partial Credit: A partially relevant document at rank 1 is treated identically to a perfectly relevant document at rank 1.
• Single Relevant Document: It does not reward systems that retrieve multiple distinct relevant documents for a single query, making it unsuitable for recall-oriented tasks like legal discovery.
• Alternative: For graded relevance, consider Normalized Discounted Cumulative Gain (NDCG).

In modern Dense Passage Retrieval (DPR) and Retrieval-Augmented Generation (RAG) stacks, MRR is a primary metric for evaluating the first-stage retriever.

• Pipeline Role: It validates that the retriever places the correct context chunk high enough for a downstream Cross-Encoder or Large Language Model (LLM) to utilize it.
• Training Signal: Bi-encoders are often optimized to maximize MRR by using contrastive loss functions that explicitly push the correct passage to the top of the ranking.
• Benchmarking: Standard benchmarks like Natural Questions (NQ) and TriviaQA report MRR to compare retrieval efficiency.