Mean Reciprocal Rank (MRR)

MRR is calculated by averaging the reciprocal of the rank at which the first relevant document appears for each query in a test set.

• Formula: MRR = (1/|Q|) * Σ (1 / rank_i)
• rank_i is the position of the first correct answer for query i.
• If the system fails to find a relevant document, the reciprocal rank is 0.
• The score is always between 0 and 1, with 1 being a perfect score (the correct answer is always ranked first).

MRR is highly sensitive to the position of the first relevant item and completely ignores the quality of subsequent results.

• A relevant result at rank 1 contributes a score of 1.0.
• A relevant result at rank 2 contributes only 0.5.
• A relevant result at rank 10 contributes a mere 0.1.
• This sharp decay curve heavily penalizes systems that bury the correct answer deep in a list of results.

MRR is the ideal metric for navigational queries and factoid question-answering where there is exactly one correct answer.

• Question Answering: 'What is the capital of France?'
• FAQ Matching: Finding the single correct support article for a user's question.
• Link Prediction: In knowledge graphs, predicting the single correct tail entity for a given head and relation.
• It is not suitable for recall-oriented searches where a user wants a comprehensive list of documents.

Unlike other ranking metrics, MRR does not require graded relevance judgments (e.g., 'perfect', 'good', 'fair'). It operates on a strict binary relevance model.

• vs. Precision@K: Precision@K measures the density of relevant items in the top K results. MRR only cares about the position of the first one.
• vs. NDCG: NDCG accounts for multi-level relevance (e.g., a highly relevant document is better than a somewhat relevant one). MRR treats all relevant documents equally, focusing purely on rank position.
• vs. Recall: Recall measures the total number of relevant items found. MRR is agnostic to total recall.

The calculation of MRR requires a clear protocol for edge cases to ensure consistent evaluation.

• Ties: If multiple relevant documents are tied at the same rank, the standard approach is to use the highest rank (worst-case scenario) to avoid inflating the score.
• No Relevant Document: If a query returns zero relevant results, the reciprocal rank is 0.0. This heavily penalizes total failures, making MRR a strict metric for safety-critical retrieval systems.

In Retrieval-Augmented Generation (RAG) pipelines, MRR is a critical diagnostic metric for the retrieval component. It directly measures how quickly the system can surface the correct context chunk to ground the language model.

• A high MRR indicates that the bi-encoder or hybrid search system is effectively indexing the knowledge base.
• A low MRR signals a vocabulary mismatch or poor chunking strategy, often requiring query expansion or Hypothetical Document Embeddings (HyDE) to fix.

A ranking-aware metric that evaluates the quality of an entire ordered result list, not just the first relevant item. NDCG assigns higher weight to relevant documents appearing earlier in the ranking and incorporates graded relevance judgments (e.g., 'highly relevant' vs. 'partially relevant').

• Key difference from MRR: NDCG considers all relevant documents and their positions, while MRR only cares about the first one
• Formula: DCG / IDCG, where IDCG is the ideal ordering
• Use case: Preferred when multiple relevant documents exist per query and their degree of relevance varies

Measures the proportion of all relevant documents in the corpus that are successfully retrieved within the top-K results. Recall@K answers: 'Of everything relevant that exists, how much did we find?'

• Complementary to MRR: MRR focuses on rank position of the first hit; Recall@K measures coverage
• Trade-off with Precision@K: High recall often comes at the cost of lower precision
• Typical K values: 10, 20, 100 depending on the application
• Critical for: Legal discovery, medical literature review, and any domain where missing a relevant document is costly

The fraction of the top-K retrieved documents that are actually relevant to the query. Precision@K answers: 'How much of what we showed the user was useful?'

• Relationship to MRR: A system can have high Precision@1 but low MRR if that single relevant result is inconsistently ranked across queries
• Binary relevance assumption: Unlike NDCG, Precision@K typically treats relevance as a yes/no judgment
• Common in: Search engine evaluation, recommendation systems, and question-answering pipelines where top-result quality is paramount

A broader metric measuring the overall fraction of retrieved documents that are relevant, often calculated across an entire test collection rather than per-query. Retrieval precision is the foundational measure of a system's ability to avoid returning non-relevant results.

• Distinction from MRR: Retrieval precision is position-agnostic; MRR is position-sensitive
• Used in: Evaluating the first-stage retriever in RAG pipelines before re-ranking
• Impact on downstream tasks: Low retrieval precision introduces noise that can degrade LLM generation quality, even if the first relevant document is ranked highly

A specialized metric for RAG and source-attributed generation systems. Citation precision measures the proportion of generated statements with a cited source that are fully supported by that specific source.

• Builds on MRR concepts: Like MRR, it evaluates the quality of the connection between a query and retrieved evidence
• Granularity difference: Citation precision operates at the claim level within a generated response, not at the query level
• Critical for: Enterprise AI deployments where every factual assertion must be traceable to an authoritative source
• Failure mode: A system can achieve perfect MRR but poor citation precision if it retrieves the right document but misinterprets its content

An evaluation score measuring the degree to which generated text is factually consistent with the provided source context, without adding external information or contradicting the evidence.

• Relationship to MRR: MRR evaluates retrieval quality; faithfulness evaluates generation quality given that retrieval
• Measured via: Natural Language Inference (NLI) models that classify each claim as entailed, contradicted, or neutral relative to the source
• Pipeline integration: High MRR + low faithfulness indicates a retrieval system that finds the right documents but a generation system that hallucinates despite having correct context
• Tooling: Frameworks like RAGAS and TruLens provide automated faithfulness scoring