Recall@K

Recall@K quantifies how many of the total relevant documents for a query appear in the top-K retrieved results.

• Formula: Recall@K = (Number of relevant documents in top-K) / (Total number of relevant documents)
• Range: 0 to 1, where 1 indicates all relevant documents were retrieved.
• Focus: This metric ignores the ranking order within the top-K and ignores irrelevant documents, focusing solely on coverage.
• Example: If a query has 5 relevant documents in the corpus and the system retrieves 3 of them in the top-10 results, Recall@10 is 3/5 = 0.6.

Recall@K and Precision@K are complementary metrics that trade off against each other.

• Recall@K answers: "Of all the relevant items, how many did we find?" It penalizes missed results.
• Precision@K answers: "Of the items we retrieved, how many are relevant?" It penalizes irrelevant results.
• Trade-off: A system can achieve perfect Recall@K by simply returning all documents (K = corpus size), but this destroys precision. Effective systems balance both.

In Retrieval-Augmented Generation (RAG), Recall@K is often the primary retrieval metric because the downstream LLM can filter out noise, but it cannot answer correctly if the required fact was never retrieved.

• Generative Filtering: A strong LLM can ignore irrelevant chunks in its context window, acting as a precision filter.
• Factual Ceiling: The quality of the final generated answer is strictly bounded by the recall of the retrieval step. If the fact isn't in the top-K, the LLM will likely hallucinate.
• Tuning: Teams often optimize for high Recall@K (e.g., K=20) and rely on a re-ranker to boost precision before generation.

While essential, Recall@K has critical limitations that must be understood.

• Binary Relevance Assumption: It treats all documents as either fully relevant or irrelevant, ignoring graded relevance (e.g., a partially relevant document counts the same as a perfectly relevant one).
• Total Relevant Count Problem: Calculating the denominator requires knowing the absolute number of relevant documents in the entire corpus, which is often unknown in large-scale, real-world datasets.
• K-Dependency: The metric is highly sensitive to the choice of K. Recall@5 and Recall@100 tell very different stories about system performance.

In practice, Recall@K is averaged across a test set of queries to provide a robust evaluation.

• Mean Recall@K: The arithmetic mean of the Recall@K score for each query in the evaluation set.
• Binary Vectors: For each query, a binary relevance vector is created for the top-K results (1 for relevant, 0 for irrelevant).
• Implementation: Libraries like ranx or pytrec_eval provide optimized functions to calculate Mean Recall@K against standard benchmarks like MS MARCO or BEIR.

Recall@K is closely related to Hit Rate (or Success@K) , but they measure different things.

• Hit Rate: Measures the percentage of queries for which at least one relevant document is found in the top-K. It's a binary per-query metric.
• Recall@K: Measures the proportion of all relevant documents found. A system can have a 100% Hit Rate but a low Recall@K if it only ever finds one of many relevant documents.
• Use Case: Hit Rate is useful for factoid QA (one answer needed), while Recall@K is better for summarization or multi-faceted research tasks.

The complementary metric to Recall@K, measuring the proportion of retrieved documents in the top-K set that are actually relevant. While Recall@K focuses on completeness (finding all relevant items), Precision@K focuses on purity (avoiding irrelevant results).

• Formula: (Number of relevant documents in top-K) / K
• Trade-off: High recall often comes at the cost of lower precision
• Use case: Critical in search UI where users only scan the first page of results

An evaluation metric that measures how quickly the system retrieves the first relevant document. MRR is the average of the reciprocal ranks of the first correct answer across a set of queries.

• Formula: (1/|Q|) * Σ (1 / rank_i)
• Sensitivity: Heavily penalizes systems that bury the correct answer deep in results
• Best for: Question-answering systems and featured snippet evaluation where position matters critically

A rank-aware metric that evaluates retrieval quality by considering both the relevance and the position of documents. Unlike Recall@K, NDCG assigns higher weights to relevant documents appearing earlier in the ranked list.

• Graded relevance: Supports multi-level relevance judgments (e.g., highly relevant vs. partially relevant)
• Discount function: Logarithmically reduces the contribution of documents further down the list
• Normalization: Compares against an ideal ranking to produce a score between 0 and 1

The harmonic mean of Precision@K and Recall@K, providing a single balanced score that penalizes systems which optimize one metric at the extreme expense of the other.

• Formula: 2 * (Precision@K * Recall@K) / (Precision@K + Recall@K)
• Balance: Prevents gaming by returning all documents (high recall, low precision) or very few (high precision, low recall)
• Threshold: Requires choosing K thoughtfully based on the application's tolerance for missed results

A comprehensive retrieval metric that averages precision scores at each position where a relevant document is found, then takes the mean across all queries. MAP provides a holistic view of both precision and recall behavior.

• Binary relevance: Assumes documents are either relevant or not
• Stability: Less sensitive to outliers than MRR for large result sets
• Standard benchmark: Widely used in academic information retrieval evaluations like TREC

A special case of Recall@K where K=1, measuring whether the single top-ranked result is relevant. This is the strictest retrieval evaluation, demanding perfect first-attempt accuracy.

• Binary outcome: Either the top result is relevant (1) or not (0)
• Critical for: Voice assistants and single-answer systems where only one response is surfaced
• Relationship: Recall@1 ≤ Recall@K for any K > 1, forming a lower bound on retrieval completeness