Mean Reciprocal Rank (MRR)

The core calculation is the multiplicative inverse of the rank position of the first relevant document. If the first correct answer is at rank 1, the reciprocal rank is 1. If it is at rank 2, it is 1/2 (0.5). If it is at rank 5, it is 1/5 (0.2). The MRR is the average of these reciprocals across a set of queries.

• Formula: MRR = (1/|Q|) * Σ (1/rank_i)
• Range: Strictly bounded between 0 and 1
• Zero Handling: If no relevant document is found, the reciprocal rank is 0

MRR is specifically designed to penalize systems that fail to place the correct answer at the very top of the list. The score drop is non-linear and severe.

• Rank 1 to 2: Score drops from 1.0 to 0.5 (a 50% penalty)
• Rank 2 to 3: Score drops from 0.5 to 0.33 (a smaller marginal penalty)
• Rank 10 to 11: Score drops from 0.1 to 0.09 (a negligible penalty)

This makes MRR ideal for question-answering systems and FAQ retrieval where the user expects the answer in position one.

Unlike NDCG or MAP, MRR operates on the strict assumption that there is exactly one relevant item per query. It only cares about the rank of the first correct answer and ignores all subsequent relevant documents.

• Binary Relevance: Documents are either correct or incorrect; there are no graded relevance levels
• First-Result Focus: If the first relevant document is at rank 3, the system gets a score of 1/3 regardless of whether ranks 4 and 5 are also relevant
• Contrast with MAP: Mean Average Precision considers all relevant documents and their positions, making it better for navigational queries with multiple valid targets

MRR is the dominant evaluation metric for systems where a single definitive answer is required.

• Open-Domain QA: Evaluating systems like Dense Passage Retrieval (DPR) on datasets such as Natural Questions or TriviaQA
• Known-Item Search: Finding a specific document the user has seen before
• Factoid Retrieval: Answering

While both metrics prioritize the top position, they measure different things.

• Precision@1: A binary metric that is 1 if the top result is relevant, 0 otherwise. It ignores the rank of the correct answer if it is not at position 1.
• MRR: Provides a continuous score that rewards getting close to the top. A correct answer at rank 2 still contributes 0.5 to the score, whereas Precision@1 would be 0.
• Key Distinction: MRR is a softer, more informative metric for model development because it shows partial progress when the model ranks the correct answer highly but not perfectly.

MRR has specific blind spots that make it unsuitable for certain retrieval tasks.

• Ignores Recall: A system that finds one relevant document at rank 1 gets a perfect score, even if it missed 10 other relevant documents
• Insensitive to Deep Ranks: Changes beyond rank 10 have almost no impact on the final score, making it poor for evaluating recall-oriented systems like e-discovery
• Query Difficulty Bias: A single very difficult query where the answer is at rank 50 can disproportionately drag down the mean, making median reciprocal rank a more robust alternative for skewed distributions

A listwise evaluation metric that measures ranking quality by discounting relevance gains logarithmically by position. Unlike MRR, which only cares about the first relevant result, NDCG rewards systems that place multiple relevant documents near the top. The metric normalizes against an ideal ranking, producing scores from 0 to 1.

• Formula: DCG / IDCG, where gains are discounted by log₂(position + 1)
• Key difference from MRR: NDCG considers all relevant documents, not just the first
• Use case: Evaluating search engines where users browse multiple results

Measures the proportion of relevant documents within the top-K retrieved results. While MRR focuses on the rank position of the first hit, Precision@K evaluates the density of relevance in the top slice.

• P@1 is equivalent to MRR's binary judgment for a single query — either 1 or 0
• P@10 is commonly used in web search evaluation
• Limitation: Does not account for rank order within the top K

Calculates the fraction of all relevant documents in the corpus that appear in the top-K retrieved results. MRR and Recall serve complementary roles: MRR measures precision of the top result, while Recall measures completeness.

• High recall, low MRR: The system finds relevant docs but buries them
• High MRR, low recall: The system nails the first hit but misses other relevant content
• Trade-off: First-stage retrievers optimize Recall@K; re-rankers optimize MRR

A rank-aware metric that averages precision scores computed at each relevant document position across a query set. MAP provides a more holistic view than MRR by rewarding systems that cluster relevant documents early in the ranked list.

• Binary relevance assumption: Documents are either relevant or not, with no graded scale
• Heavily influenced by recall: Missing relevant documents lowers MAP significantly
• MRR vs MAP: MRR is a single-point metric; MAP integrates precision across all recall levels

A rank aggregation algorithm that combines multiple ranked lists by computing a weighted sum of reciprocal ranks. RRF shares the reciprocal rank concept with MRR but applies it as a fusion mechanism rather than an evaluation metric.

• Formula: score(d) = Σ 1 / (k + rankᵢ(d)), where k is typically 60
• Purpose: Merges results from sparse retrieval (BM25) and dense retrieval (embeddings)
• Connection to MRR: Both leverage the insight that rank position importance decays hyperbolically

The binary metric measuring whether at least one relevant document appears in the top-K results. MRR is the rank-sensitive refinement of Hit Rate — instead of just recording success or failure, MRR penalizes systems that succeed at a deeper rank.

• Hit Rate@1 is the per-query binary component that MRR averages
• Insensitive to rank depth: A hit at position 1 and position 100 are treated identically
• Use case: Question-answering systems where any correct answer in the top slice counts