Normalized Discounted Cumulative Gain (NDCG)

NDCG moves beyond binary relevance (relevant/not relevant) to evaluate graded relevance scores. Each item in a ranked list is assigned a relevance grade (e.g., 0=irrelevant, 1=somewhat relevant, 2=highly relevant, 3=perfectly relevant). The metric's Cumulative Gain (CG) is the sum of these relevance scores for the top K results, directly rewarding systems that retrieve highly relevant items.

• Example: A search returning documents with relevance scores [3, 2, 1, 0] has a CG@4 = 6.
• This is fundamental for RAG evaluation, where some context passages are critically useful (high grade) while others are merely tangential (low grade).

A core innovation of NDCG is its logarithmic discount function, which reduces the contribution of a relevant item based on its rank position. This reflects the user experience principle that items appearing lower in a list are less likely to be seen and used.

• The Discounted Cumulative Gain (DCG) formula applies a discount: rel_i / log2(i + 1), where i is the rank position.
• Impact: A highly relevant document (score=3) at position 1 contributes 3.0 to DCG. The same document at position 4 contributes only 3 / log2(5) ≈ 1.29.
• This forces evaluation to penalize systems that bury critical context deep in the retrieved list, which is detrimental for RAG generation quality.

NDCG is normalized by the Ideal DCG (IDCG), which is the maximum possible DCG achievable for a given set of relevance judgments. This creates a score between 0.0 and 1.0, allowing for comparison across different queries with varying numbers of relevant items.

• Calculation: NDCG@K = DCG@K / IDCG@K.
• IDCG is computed by sorting all items by their true relevance score in descending order and calculating the DCG of that ideal ranking.
• A score of 1.0 represents a perfect ranking. A score of 0.7 indicates the system achieves 70% of the possible gain from an ideal ordering.
• This normalization is essential for reporting a single, interpretable metric across an entire evaluation benchmark.

Unlike Precision@K or Recall@K, which treat all top-K items equally, NDCG is explicitly rank-aware. It provides a fine-grained measure of ranking quality, making it the metric of choice when the order of results is critical to the downstream task.

• Use Case in RAG: The first retrieved passage provides the primary context for an LLM. NDCG directly measures if the most relevant passage is ranked first.
• Comparison: Two systems may have identical Precision@5, but the system with the more relevant document in position 1 will have a significantly higher NDCG@5.
• This makes NDCG indispensable for evaluating retrievers and rerankers within a RAG pipeline, where ordering dictates context quality.

NDCG is the de facto standard for evaluating web search engines and recommendation systems, and by extension, retrieval for RAG. Major academic and industry benchmarks (e.g., MS MARCO, BEIR, TREC tracks) report NDCG as a primary metric, enabling direct comparison between different retrieval architectures.

• It allows comparison between sparse retrievers (BM25), dense retrievers (DPR, Contriever), and cross-encoders used for reranking.
• Standard cut-offs like NDCG@5, NDCG@10, and NDCG@100 are used to measure performance at different depths of the ranking.
• When a paper or system claims state-of-the-art retrieval performance, it is typically validated by an improvement in NDCG on established benchmarks.

NDCG exists within a family of ranking metrics, each with specific strengths.

• vs. Mean Average Precision (MAP): MAP also considers rank order but assumes binary relevance. NDCG is preferred for graded relevance scenarios common in RAG.
• vs. Mean Reciprocal Rank (MRR): MRR only considers the rank of the first relevant item. NDCG provides a more comprehensive view of the entire ranking.
• vs. Precision/Recall at K: These metrics are rank-agnostic within the top K. NDCG incorporates rank sensitivity.
• Practical Integration: In a full RAG evaluation suite, NDCG assesses retrieval quality, while metrics like Answer Faithfulness and Answer Relevance assess the final generation. High NDCG is a strong leading indicator for good final answer quality.

Mean Average Precision (MAP) calculates the mean of the Average Precision scores across a set of queries, providing a single-figure measure of quality for a ranking system. Unlike NDCG, MAP assumes binary relevance (relevant/not relevant).

• Core Calculation: For each query, Average Precision is the average of precision values at each rank where a relevant document is found. MAP is the mean of these values across all queries.
• Use Case: Best for tasks where the primary goal is to retrieve all relevant items, and their order among themselves is less critical than simply getting them to the top.
• Comparison to NDCG: MAP does not account for graded relevance levels or the specific utility decay of lower ranks as explicitly as NDCG's discount function.

Precision at K (P@K) and Recall at K (R@K) are fundamental cut-off metrics for evaluating retrieval systems at a specific depth K.

• P@K: Measures the proportion of relevant documents among the top K retrieved results. It answers: "Of the first K results shown, how many were good?"
• R@K: Measures the proportion of all relevant documents for a query that were found within the top K results. It answers: "How much of the total relevant content did I find in the first K results?"
• Relationship to NDCG: These are simpler, position-bound metrics. NDCG provides a more nuanced single score by aggregating performance across all ranks, weighting by position and relevance grade.

Mean Reciprocal Rank (MRR) is a metric focused on the rank of the first relevant item. It is the average of the reciprocal of the rank of the first correct answer across multiple queries.

• Calculation: For each query, compute 1 / rank_of_first_relevant_result. MRR is the average of these values. If no relevant item is found, the reciprocal is 0.
• Utility: Highly interpretable and critical for use cases where the user's goal is to find a single correct answer quickly (e.g., question answering, voice assistants).
• Contrast with NDCG: MRR is insensitive to the quality of results after the first relevant one. NDCG evaluates the entire ranked list, making it more suitable for exploratory search or recommendation.

Metrics like BERTScore and Semantic Similarity evaluate the meaning-based alignment between text, often used to assess generated answers against references.

• BERTScore: Computes similarity using contextual embeddings from models like BERT. It matches words in candidate and reference texts based on cosine similarity in embedding space, producing precision, recall, and F1 measures.
• General Semantic Similarity: Often computed using sentence embeddings (e.g., from Sentence-BERT) to gauge the conceptual relatedness of two passages, such as a generated answer and a ground truth.
• Role in Evaluation: While NDCG evaluates retrieval ranking, these metrics often evaluate the final generation quality in RAG pipelines, assessing answer quality independently of the retrieval process.

RAGAS (Retrieval-Augmented Generation Assessment) is an open-source framework for reference-free evaluation of RAG pipelines. It defines several metrics that complement retrieval-focused metrics like NDCG.

• Faithfulness: Measures factual consistency between the generated answer and the retrieved context. A low score indicates hallucination.
• Answer Relevance: Assesses how directly the generated answer addresses the original query, independent of the context.
• Context Relevance: Evaluates the pertinence of the retrieved context to the query. This is closely related to the precision of the retrieval stage.
• Synthetic Data: RAGAS often uses an LLM to generate synthetic test cases and evaluations, enabling assessment without human-written references.

Reranking Effectiveness quantifies the improvement in retrieval quality achieved by applying a secondary, more computationally intensive model to an initial set of candidate documents.

• Two-Stage Process: A fast retriever (e.g., a dense vector search) fetches a broad set of candidates (e.g., top 100). A slow but accurate reranker (e.g., a cross-encoder) rescores this set to produce the final top-K results.
• Measurement: The lift in metrics like NDCG@K or MAP after reranking, compared to the initial retrieval, defines effectiveness. For example, an initial retrieval might have NDCG@10 of 0.65, which improves to 0.82 after reranking.
• System Design Goal: The core engineering trade-off is between the improved accuracy of the final ranked list and the added latency and compute cost of the reranking stage.