Contextual Recall

Contextual Recall measures the proportion of information from a ground truth answer that is present within the retrieved context provided to the language model. It is calculated as:

• (Number of Ground Truth Statements Retrieved) / (Total Number of Ground Truth Statements)
• A score of 1.0 indicates all required information was retrieved; a score of 0.0 indicates none was found. This metric directly evaluates the completeness of retrieval, which is critical for preventing answer omissions and ensuring factual grounding.

Contextual Recall is the RAG-specific adaptation of the classic Recall metric from information retrieval. The key difference is the reference point:

• Classic Recall: Measures retrieval of entire relevant documents from a corpus.
• Contextual Recall: Measures retrieval of specific information statements needed to formulate an answer. It is closely related to Recall@k, which measures if relevant documents are in the top-k results. High Contextual Recall requires high Recall@k, but also depends on effective document chunking to ensure key facts aren't split across chunks.

In practice, optimizing for Contextual Recall often involves a trade-off with Contextual Precision, which measures how much of the retrieved context is actually relevant.

• High Recall, Low Precision: System retrieves many chunks, ensuring all facts are present, but includes irrelevant text that can confuse the LLM, increase latency, and waste context window space.
• High Precision, Low Recall: System retrieves only highly relevant chunks but may miss key facts, leading to incomplete or hallucinated answers. Effective RAG design uses techniques like hybrid search and cross-encoder reranking to optimize the precision-recall curve for a given use case.

The metric is highly sensitive to the document chunking strategy. Poor chunking can artificially lower scores even with a perfect retriever.

• Small, Fixed-Size Chunks: Risk splitting a single fact across two chunks, causing both to be incomplete and lowering recall.
• Semantic or Topic-Aware Chunks: Chunking by paragraphs or using models to identify logical boundaries (e.g., NLTK, spaCy) helps keep coherent information together, improving recall.
• Overlapping Chunks: A common technique where consecutive chunks share some content (e.g., a 100-token overlap) to mitigate split-fact issues and boost Contextual Recall at the cost of increased index size and potential redundancy.

Accurate measurement requires a robust evaluation dataset with human-annotated ground truth.

1. Query & Ground Truth Pair: For each query, human experts define the complete, verifiable answer.
2. Statement Decomposition: The ground truth answer is decomposed into distinct, atomic factual statements.
3. Annotation: Annotators label whether each atomic statement is present in the system's retrieved context. Frameworks like RAGAS automate this by using an LLM judge to perform the decomposition and presence-check, though human validation is recommended for high-stakes evaluations. Inter-annotator agreement (e.g., Cohen's Kappa) should be measured to ensure label reliability.

Low Contextual Recall directly causes hallucinations of omission and incomplete answers. Engineers optimize it by adjusting:

• Retrieval Method: Replacing pure dense vector search (high precision) with hybrid search (combining BM25 and vectors) often boosts recall.
• Query Transformation: Using query expansion (adding synonyms) or hypothetical Document Embeddings (HyDE) can retrieve a broader set of relevant chunks.
• k Value: Increasing the number of chunks retrieved (top-k) directly improves recall but harms precision and latency.
• Index Granularity: Experimenting with chunk sizes and overlap is a primary lever for recall improvement without changing the core retriever model.

Recall is the foundational information retrieval metric that measures the proportion of all relevant documents in a collection that are successfully retrieved by a search system. It is defined as (Relevant Retrieved Documents) / (All Relevant Documents).

• Key Difference: Standard recall operates at the document level, while Contextual Recall operates at the information (token/semantic) level within a retrieved context.
• Application: Used to evaluate search engines and database queries where the goal is to miss as few relevant items as possible.

Contextual Precision is the complementary metric to Contextual Recall within RAG evaluation. It measures the proportion of information within the retrieved context that is actually relevant and useful for answering the query.

• Focus: It penalizes retrieval of irrelevant passages that waste the LLM's context window and can introduce noise.
• Trade-off: Optimizing for high Contextual Recall (retrieving all relevant info) can lower Contextual Precision (by including irrelevant text). A balanced system manages both.
• Calculation: Assesses the density of useful information in the provided context chunks.

Faithfulness is a RAG generation metric that measures the extent to which a model's final answer is factually consistent with and directly supported by the retrieved context. It is downstream of retrieval quality.

• Dependency: High Faithfulness is impossible without high Contextual Recall. If the necessary facts aren't retrieved, the LLM cannot be faithful to them.
• Measurement: Typically evaluated by checking if all atomic claims in the generated answer can be attributed to statements in the source context.
• Goal: Directly targets the reduction of hallucinations in the final output.

Mean Average Precision (MAP) is a ranking-oriented metric that calculates the mean of the Average Precision scores across a set of queries. It rewards systems that retrieve relevant documents higher in the results list.

• Ranking Sensitivity: Unlike binary recall, MAP considers the order of results. Placing all relevant docs at the bottom yields perfect recall but poor MAP.
• Relation to RAG: While Contextual Recall assesses if information is present, metrics like MAP assess how efficiently it is presented (i.e., are the most relevant chunks at the top?). Efficient ranking reduces the context length needed for high recall.

Hit Rate @ K is a pragmatic, binary metric that measures the proportion of queries for which at least one relevant document is found within the top K retrieved results.

• Operational Focus: Answers the question: "Does the system find something useful in the first page of results?" Crucial for user-facing applications.
• Connection: A high Hit Rate @ K is a prerequisite for high Contextual Recall. If no relevant document is in the top K, the information cannot be recalled. It is a simpler, coarser measure of retrieval success.