Normalized Discounted Cumulative Gain (NDCG)

Unlike binary metrics (relevant/not relevant), NDCG operates on graded relevance scores. Each item in the result list is assigned a relevance label, typically an integer (e.g., 0=irrelevant, 1=somewhat relevant, 2=highly relevant). This allows the metric to distinguish between a good result and a perfect result.

• Example: In a search for "machine learning tutorials," a foundational blog post might score 1, while an official university course syllabus might score 3.

NDCG incorporates the principle that users value relevant items more when they appear higher in the list. It applies a logarithmic discount to the gain (relevance score) of each item based on its rank.

The gain for an item at position i is divided by log2(i + 1). This means the utility of a relevant document drops sharply if it's not on the first page of results, modeling real user attention decay.

Cumulative Gain (CG) is the simple sum of relevance scores for the top K results. It ignores position.

Discounted Cumulative Gain (DCG) refines this by summing the positionally discounted gains.

Formula for DCG@K: DCG@K = Σ (relevance_i / log2(i + 1)) for i = 1 to K.

A variant uses an exponential gain for stronger emphasis on highly relevant items: DCG@K = Σ ((2^relevance_i - 1) / log2(i + 1)).

NDCG is normalized by the Ideal DCG (IDCG). IDCG is the maximum possible DCG achievable for a given set of queries, obtained by ranking all items in perfect descending order of relevance.

Formula: NDCG@K = DCG@K / IDCG@K

This normalization bounds NDCG between 0.0 and 1.0, where 1.0 represents a perfect ranking. It allows for meaningful comparison across different queries and datasets with varying numbers of relevant items.

NDCG provides a single, interpretable number that summarizes overall ranking quality. It is the de facto standard for evaluating web search engines, recommendation systems, and RAG retrievers.

• A score of 0.9+ indicates excellent ranking.
• Scores are averaged over a large query set to evaluate system performance.
• It directly measures the user experience of finding high-quality information quickly.

NDCG complements other retrieval metrics:

• Precision@K: Measures accuracy in the top K but uses binary relevance.
• Recall@K: Measures coverage of all relevant items.
• Mean Average Precision (MAP): Averages precision at each relevant item's rank, also for binary relevance.

NDCG is preferred when relevance is non-binary. It is more correlated with user satisfaction than precision or recall alone, as it penalizes systems that bury excellent results on page two.

NDCG is the de facto standard for evaluating the quality of web search results. It quantifies how well a search engine orders documents by their usefulness to the user.

• Graded Relevance: Unlike binary metrics, NDCG uses graded relevance scores (e.g., 0=irrelevant, 1=somewhat relevant, 2=relevant, 3=highly relevant).
• Positional Discounting: It applies a logarithmic discount, heavily penalizing a highly relevant document placed on the second page of results versus the first position.
• Industry Benchmark: Major search providers use NDCG@5, NDCG@10, or NDCG@20 to A/B test ranking algorithm changes and measure improvements in user satisfaction.

In e-commerce and media streaming, NDCG evaluates the quality of personalized recommendation lists (e.g., "Recommended for You").

• List Quality: It measures if the most engaging or likely-to-purchase items are ranked at the top of a recommendation carousel.
• Business Impact: A higher NDCG correlates directly with key metrics like click-through rate (CTR), conversion rate, and user retention.
• Example: A video platform uses NDCG to test if a new algorithm ranks movies a user is likely to watch for 80% of their duration higher than those they might abandon after 5 minutes.

Within RAG architectures, NDCG is critical for evaluating the retriever component—the system that fetches context documents for the language model.

• Retriever Benchmarking: Engineers compare different retrieval strategies (e.g., dense vs. sparse vs. hybrid search) by calculating NDCG over a test set of queries and ground-truth relevant document chunks.
• Fine-Tuning Validation: After fine-tuning a dense retriever on domain-specific data, the improvement is measured by the increase in NDCG, indicating the retriever now better understands domain-specific language.
• Precision vs. Recall Trade-off: NDCG@K (e.g., K=5) ensures the top few retrieved passages are maximally relevant, which is crucial for generator accuracy and hallucination reduction.

NDCG serves as a primary offline metric for gatekeeping model deployments. A new ranking model must demonstrate a statistically significant improvement in NDCG over the current production model before it is launched.

• Reduction of Online Risk: It provides a reliable, quantitative proxy for expected user satisfaction before costly and risky live experiments.
• Multi-Model Comparison: When choosing between several candidate algorithms, the one with the highest NDCG on a held-out validation set is typically selected for further testing.
• Monitoring Drift: Tracking NDCG on a fixed evaluation set over time can alert teams to performance degradation due to data or concept drift.

NDCG is the standard evaluation metric for public datasets and challenges in information retrieval, such as the TREC conferences or the MS MARCO leaderboard.

• Reproducible Research: It provides a consistent, normalized score (between 0.0 and 1.0) that allows for direct comparison of algorithms published by different research groups.
• Dataset-Specific: Major datasets publish official evaluation scripts and report NDCG@10 or similar as the key metric for ranking submissions.
• Driving Innovation: The pursuit of higher NDCG scores on these benchmarks has driven advancements in neural ranking models, cross-encoders, and retrieval-augmented architectures.

NDCG is used in conjunction with other metrics to provide a complete performance picture. Understanding its relationship to these metrics is essential for practitioners.

• vs. Precision/Recall: Precision@K measures accuracy at a cutoff but treats all relevant items equally. NDCG is superior as it uses graded relevance and positional discounting.
• vs. Mean Average Precision (MAP): MAP is designed for binary relevance and emphasizes recall. NDCG is preferred for multi-level relevance scales common in modern systems.
• Ideal Discounted Cumulative Gain (IDCG): This is the denominator in NDCG. It represents the score of a perfect ranking, providing an intuitive upper bound of 1.0.
• Cutoff Selection: The choice of K in NDCG@K is application-dependent: NDCG@5 for first-page web results, NDCG@100 for evaluating full retrieval sets in academic settings.

Recall@K measures the proportion of all relevant documents that are successfully retrieved within the top K results. It answers the question: "Did the system find all the relevant items?"

• Formula: (Number of relevant docs in top K) / (Total number of relevant docs).
• Focus: Completeness of retrieval. A high Recall@K means few relevant items are missed.
• Trade-off: Often inversely related to precision; retrieving more items (higher K) increases recall but can lower precision.
• Use Case: Critical in applications where missing a relevant document is costly, such as legal e-discovery or systematic literature reviews.

Mean Reciprocal Rank (MRR) is a rank-sensitive metric that evaluates systems where only the first relevant result matters. It calculates the average of the reciprocal of the rank at which the first relevant document is found.

• Formula: For a set of queries Q, MRR = (1/|Q|) * Σ (1 / rank_i), where rank_i is the position of the first relevant document for query i.
• Focus: Rank of the first hit. It heavily penalizes systems that bury the first relevant result.
• Interpretation: Scores range from 0 to 1. An MRR of 0.5 means the first relevant document is, on average, ranked 2nd.
• Use Case: Ideal for question-answering systems or web search where the user's primary need is satisfied by a single, high-quality result.

Precision@K measures the fraction of retrieved documents within the top K results that are relevant. It answers: "How much of what was retrieved is actually useful?"

• Formula: (Number of relevant docs in top K) / K.
• Focus: Purity or accuracy of the retrieved set. High precision means the user is not wasting time on irrelevant results.
• Context: Unlike NDCG, it treats all relevant items equally, regardless of their degree of relevance or specific rank within the top K.
• Use Case: Essential for user interfaces with limited space (e.g., mobile search) where showing irrelevant results degrades the user experience.

Mean Average Precision (MAP) is a single-figure metric that summarizes the quality of a ranked list across multiple levels of recall. It is the mean of the Average Precision scores for a set of queries.

• Calculation: For a single query, Average Precision is the average of Precision@K values computed at each rank where a relevant document is found. MAP averages this across all queries.
• Focus: Overall ranking quality, balancing both precision and recall. It rewards systems that retrieve many relevant documents early.
• Comparison to NDCG: MAP assumes binary relevance (relevant/not relevant), while NDCG can handle graded relevance scores (e.g., highly relevant, somewhat relevant).
• Use Case: The standard evaluation metric for many classic information retrieval benchmarks and competitions.

Discounted Cumulative Gain (DCG) is the unnormalized precursor to NDCG. It measures the cumulative gain of a search result list, applying a logarithmic discount to gains based on their rank.

• Core Principle: The utility (gain) of a document is reduced (discounted) the lower it appears in the list. This reflects the decreasing user attention with rank.
• Formula: A common formulation is DCG@K = Σ (relevance_i / log2(i + 1)), for i = 1 to K.
• Purpose: Quantifies the absolute quality of a single ranking. To compare across queries, DCG is normalized against an ideal ranking to become NDCG.
• Key Insight: DCG directly implements the core idea that highly relevant documents are more valuable when they appear earlier.

Ideal Discounted Cumulative Gain (IDCG) is the maximum possible DCG score for a given set of query results. It is calculated by sorting all available relevant documents for a query in descending order of their true relevance scores.

• Role: Serves as the normalization factor in NDCG. NDCG = DCG / IDCG.
• Function: It establishes a perfect, achievable upper bound (a score of 1.0) for the ranking metric, allowing for fair comparison across queries with different numbers of relevant documents.
• Calculation: Requires knowledge of the true relevance judgments for all documents in the corpus relative to the query.
• Outcome: An NDCG score represents the system's performance as a percentage of the ideal ranking's quality.