Mean Reciprocal Rank (MRR)

The core of MRR is the reciprocal rank for a single query. If the first relevant document appears at rank position k, the score is 1/k. If no relevant document is found, the score is 0. This creates a steep decay curve that heavily penalizes lower-ranked correct answers.

• Rank 1: Score = 1.0
• Rank 2: Score = 0.5
• Rank 3: Score = 0.33
• Rank 10: Score = 0.1

MRR is an inverse rank metric, meaning it exclusively evaluates the position of the first relevant item. It is indifferent to any relevant documents appearing after the first one. This makes it ideal for navigational queries and question-answering systems where a single correct answer is the goal.

• Ignores recall beyond the first hit
• Perfect for factoid QA evaluation
• Insensitive to subsequent relevant results

The 'Mean' in MRR is a simple arithmetic mean of reciprocal ranks over a set of queries Q. This provides a single, interpretable score between 0 and 1 representing overall system performance.

Formula: MRR = (1/|Q|) * Σ (1/rank_i)

• Higher scores indicate better first-hit precision
• A score of 1.0 means the first result was relevant for all queries
• Sensitive to queries with zero relevant results (score contribution = 0)

MRR operates on binary relevance judgments. A document is either relevant or not relevant. It does not account for graded relevance levels. This simplification makes annotation cheaper but can be a limitation for complex information needs where some documents are more relevant than others.

• No distinction between 'partially' and 'perfectly' relevant
• Contrasts with NDCG, which uses graded scores
• Well-suited for known-item search tasks

MRR is closely related to Precision@1 but provides more granularity. While Precision@1 only cares if the top result is relevant, MRR rewards systems that place the correct answer at rank 2 over rank 50. This makes MRR a more discriminative metric for evaluating re-ranking pipelines.

• Precision@1: Binary score per query (0 or 1)
• MRR: Continuous score per query (0 to 1)
• MRR better captures 'near-miss' performance

MRR is the standard evaluation metric for tasks with a single, unambiguous correct answer. It is widely used in open-domain QA benchmarks and recommendation systems where the first suggestion is critical.

• TREC QA Track: Historical standard for factoid questions
• Chatbot Evaluation: Measuring if the correct response is retrieved first
• FAQ Matching: Finding the single correct article for a user query

A listwise ranking metric that evaluates the quality of an ordered list by discounting relevance gains logarithmically by position. Unlike MRR, which only cares about the first relevant result, NDCG accounts for graded relevance at multiple ranks.

• Formula: DCG / IDCG, where IDCG is the ideal ordering
• Key difference: Rewards placing highly relevant documents at the top, penalizes relevant documents buried deep
• Use case: Preferred when relevance judgments have multiple levels (e.g., irrelevant, somewhat relevant, highly relevant)

An unsupervised algorithm that merges multiple ranked lists into a single consensus ranking by scoring documents based on the reciprocal of their rank positions across input lists.

• Mechanism: Score = Σ 1/(k + rank_i), where k is a constant (typically 60)
• Relationship to MRR: Uses the same reciprocal rank intuition but applies it as a fusion strategy rather than an evaluation metric
• Advantage: Requires no relevance labels or score calibration between systems

A neural architecture that processes a query-document pair jointly through full self-attention to generate a fine-grained relevance score. This is the re-ranking stage where MRR is most commonly measured.

• Contrast with MRR: MRR evaluates the output of this re-ranking pipeline; the cross-encoder is the model being evaluated
• Computational cost: O(n) forward passes for n candidates, making it suitable only for top-k re-ranking
• Typical pipeline: Bi-encoder retrieval → top-100 → cross-encoder re-rank → MRR evaluation

A user engagement metric representing the ratio of clicks to impressions, commonly used as an implicit relevance signal for training learning-to-rank models.

• Relationship to MRR: CTR is an online behavioral metric; MRR is an offline evaluation metric. They often correlate but measure different things
• Position bias problem: Higher-ranked items get more clicks regardless of relevance, requiring propensity weighting for accurate interpretation
• Complementary use: MRR for offline model selection, CTR for online A/B validation

A simple metric measuring the proportion of relevant documents in the top K results. While MRR focuses on the rank of the first relevant item, Precision@K evaluates the density of relevance in the top positions.

• P@1 vs MRR: When K=1, Precision@1 is binary (0 or 1); MRR captures how close the first relevant result was even if not at position 1
• Use together: MRR for single-answer tasks (factoid QA), Precision@K for tasks expecting multiple relevant results
• Limitation: Ignores rank position beyond the K cutoff

A cascade architecture where a fast retriever (e.g., bi-encoder + ANN) selects candidate documents, and a computationally intensive re-ranker (e.g., cross-encoder) refines the ordering. MRR is the standard metric for evaluating the final output of this pipeline.

• Stage 1: High recall, low precision — retrieves 100-1000 candidates
• Stage 2: High precision re-ranking — scores top candidates with cross-attention
• MRR role: Measures whether the re-ranker successfully places the first relevant document at position 1