Retrieval Precision

Precision at K (P@K) calculates the proportion of relevant documents among the top K retrieved results for a single query. It is the most direct operationalization of retrieval precision, focusing on the quality of the initial results presented to a user or downstream model.

• Core Calculation: P@K = (# of relevant docs in top K) / K
• Use Case: Evaluating search engine result pages or the context window for a RAG system. A high P@5 is critical for user satisfaction.
• Trade-off: Optimizing for high P@K can sometimes come at the expense of Recall at K, as the system becomes overly conservative.

Average Precision (AP) is a single-query metric that summarizes the precision-recall curve by calculating the mean of precision values at each rank where a relevant document is retrieved. It rewards systems that retrieve relevant documents earlier in the ranking.

• Core Calculation: AP = Σ (P@k * rel(k)) / (total relevant docs), where rel(k) is an indicator (1/0) for relevance at rank k.
• Use Case: Provides a more nuanced evaluation than P@K alone by incorporating rank information. It is the fundamental component for calculating Mean Average Precision (MAP).
• Interpretation: An AP of 1.0 indicates all relevant documents were retrieved at the very top of the list with no irrelevant interleaving.

Mean Average Precision (MAP) is the standard benchmark for ranked retrieval quality across a set of queries. It calculates the arithmetic mean of the Average Precision (AP) scores for each query in the evaluation set.

• Core Calculation: MAP = (Σ AP for each query) / (number of queries)
• Use Case: The primary metric for comparing the overall effectiveness of search and retrieval algorithms in research and production. It is sensitive to the entire ranking order across all queries.
• Industry Standard: Widely reported in academic literature (e.g., on benchmarks like MS MARCO, BEIR) and used for model selection and hyperparameter tuning.

Context Precision is a reference-free evaluation metric defined within the RAGAS framework. It measures the precision of the retrieved context with respect to the generated answer, not just the query. It penalizes contexts that contain irrelevant passages, even if they were retrieved based on the query.

• Core Logic: For each sentence in the generated answer, the metric checks if it is supported by the retrieved context. The score is high only if the supporting context is concentrated and not diluted by irrelevant text.
• Use Case: Critical for evaluating RAG pipelines where the quality of the context passed to the LLM directly impacts answer faithfulness. It bridges retrieval evaluation and generation quality.
• Differentiator: Goes beyond traditional P@K by evaluating the utility of retrieved text for the specific answer generated.

Source Citation Precision measures the accuracy of citations in a generated answer. It calculates the proportion of citations (e.g., document IDs, chunk references) that correctly and accurately point to the source of the stated information.

• Core Calculation: Citation Precision = (# of correct citations) / (total # of citations in answer)
• Use Case: Essential for auditability and trust in enterprise RAG systems, legal applications, and any scenario requiring verifiable attribution. A low score indicates the model is "hallucinating" citations.
• Related Metric: Often evaluated alongside Source Citation Recall, which measures if all used source facts are cited. High precision with low recall suggests under-citation.

Reranking Precision Gain is not a standalone metric but an analysis of the improvement in precision metrics (e.g., P@K, MAP) achieved by applying a cross-encoder or other reranking model to an initial candidate set from a faster retriever (like a bi-encoder).

• Core Analysis: Compare P@K (after reranking) to P@K (before reranking). The delta represents the precision gain.
• Use Case: Quantifying the value-add of a computationally expensive second-stage reranker in a multi-stage retrieval pipeline. A significant gain justifies the added latency.
• Example: A dense retriever may have a P@10 of 0.6. After a cross-encoder reranks those 10 candidates, the new top 5 (P@5) might have a precision of 0.9, demonstrating a substantial lift for the most critical results.

Retrieval Recall measures the proportion of all relevant documents in a corpus that are successfully retrieved for a given query. It answers the question: "Did the system find all the relevant information?"

• High recall is critical for tasks requiring completeness, such as legal discovery or comprehensive research.
• It is often in tension with precision; optimizing for one can reduce the other (the precision-recall trade-off).
• Calculated as: (Relevant Items Retrieved) / (Total Relevant Items in Corpus).

Precision at K (P@K) is a core information retrieval metric that calculates the proportion of relevant documents among the top K retrieved results for a single query. It is a direct, position-aware variant of retrieval precision.

• P@1, P@5, P@10 are common benchmarks, indicating precision within the first 1, 5, or 10 results.
• Highly interpretable for user-facing systems where the first page of results matters most.
• Example: If 3 of the top 5 results are relevant, P@5 = 0.6 (or 60%).

Mean Average Precision (MAP) provides a single-figure measure of quality for a ranking system by averaging the Average Precision scores across a set of queries. It incorporates both precision and the rank order of relevant items.

• Average Precision (AP) for a single query is the average of the precision values calculated at each point a relevant document is retrieved.
• MAP is the mean of AP across all queries, giving more weight to systems that retrieve relevant documents higher in the ranking.
• It is a standard benchmark for academic datasets like MS MARCO and TREC.

Context Relevance assesses the degree to which the text passages retrieved and provided to a language model are pertinent and useful for answering a specific query. It evaluates the quality of the retrieved information before generation.

• Measures if retrieved passages are on-topic, concise, and non-redundant.
• Low context relevance forces the LLM to filter noise, increasing hallucination risk.
• Often evaluated by a separate LLM judge scoring passages on a scale (e.g., 1-5) for query-specific utility.

Answer Faithfulness measures the extent to which a generated answer is factually consistent with and supported by the provided source context. It directly targets hallucinations introduced during generation.

• A faithful answer contains only claims that can be inferred from the source context.
• Evaluated by cross-referencing atomic claims in the generated answer against the retrieved documents.
• A critical metric for enterprise RAG, as it ensures the system's output is grounded and trustworthy.