Normalized Discounted Cumulative Gain (NDCG)

Unlike binary metrics (relevant/not relevant), NDCG operates on graded relevance judgments. Documents are assigned relevance scores (e.g., 0=irrelevant, 1=somewhat relevant, 2=relevant, 3=highly relevant). This allows the metric to distinguish between a ranking that returns a 'highly relevant' document first versus one that returns a merely 'relevant' document first, providing a more nuanced assessment of ranking quality.

NDCG incorporates a logarithmic discount factor that reduces the contribution of a document's relevance score based on its rank. The gain from a document at position p is divided by log₂(1+p). This reflects the user experience principle that results lower on the list are less likely to be examined and therefore provide less utility. A highly relevant document at position 10 contributes significantly less to the total score than the same document at position 1.

The calculation proceeds in steps:

• Cumulative Gain (CG): The simple sum of relevance scores for the top-k results. It ignores rank position.
• Discounted Cumulative Gain (DCG): The sum of relevance scores discounted by their rank: DCG@k = Σ (relevance_i / log₂(1 + i)). DCG is the core measure of a ranking's usefulness. For example, for relevance scores [3, 2, 3, 0, 1] at k=5: DCG@5 = 3/log₂(2) + 2/log₂(3) + 3/log₂(4) + 0 + 1/log₂(6) ≈ 3 + 1.26 + 1.5 + 0 + 0.39 = 6.15

NDCG is DCG normalized by the Ideal DCG (IDCG)—the maximum possible DCG achievable for a given query, obtained by ranking all documents in perfect descending order of relevance. NDCG@k = DCG@k / IDCG@k. This normalization bounds the score between 0.0 and 1.0, where 1.0 represents a perfect ranking. It allows for meaningful comparison across queries with different numbers of relevant documents and varying relevance grade distributions.

An NDCG score is always a decimal between 0 and 1.

• 1.0: Perfect ranking (matches the ideal order).
• > 0.8: Excellent ranking performance.
• ~0.6-0.7: Good, competitive performance in many benchmarks.
• < 0.5: Indicates significant room for improvement in the retrieval or ranking model. Scores are averaged over a large query set to evaluate a system. It is the de facto standard for web search, recommendation systems, and RAG retrieval evaluation where relevance is not binary.

NDCG provides a more sensitive and informative evaluation than binary metrics like Precision@k or Recall@k.

• Advantage: It rewards systems for correctly placing higher-relevance items at top ranks. A system that swaps a 'score 3' item with a 'score 1' item will see a larger NDCG penalty than a binary metric penalty.
• Use Case: Essential for evaluating search engines and RAG retrievers where the quality of the top result is critical, and relevance is a spectrum. Binary metrics fail to capture these nuances, making NDCG the preferred choice for modern ranking evaluation.

NDCG is the primary metric for evaluating web search engines like Google and Bing, where user satisfaction depends on the order of results. It measures how well the engine surfaces the most useful pages (e.g., highly relevant official documentation vs. tangentially related blog posts) at the top of the list. A high NDCG score correlates with reduced user pagination and higher click-through rates on the first few results.

In e-commerce platforms (e.g., Amazon, Shopify), NDCG evaluates product ranking algorithms. Relevance is graded: a perfect match has high gain, a related accessory has medium gain, and an irrelevant item has zero gain. Optimizing for NDCG directly improves conversion by ensuring the most likely-to-purchase items are presented first, maximizing the discounted cumulative gain from the user's limited attention.

In Retrieval-Augmented Generation (RAG), NDCG assesses the retriever's ability to rank source document chunks by their utility for answer generation. Chunks are graded by contextual relevance to the query. A high NDCG score indicates the LLM receives the most pertinent context first, which is critical for reducing hallucinations and improving answer faithfulness. It is often used alongside metrics like Answer Relevance and Contextual Precision.

Services like Netflix, Spotify, and YouTube use NDCG to evaluate recommendation systems. Here, relevance is a graded measure of user preference (e.g., 'love,' 'like,' 'dislike'). NDCG quantifies how well the platform's algorithms prioritize content the user is most likely to enjoy, directly impacting engagement metrics like watch time and subscription retention. The discounting factor correctly models the rapid decay in user attention as they scroll.

For internal enterprise search across knowledge bases, intranets, and CRM systems, NDCG evaluates how well employees find critical information. Documents are graded by utility (e.g., exact procedure = high, related policy = medium, outdated memo = low). Optimizing NDCG reduces time-to-information and operational errors by ensuring the most authoritative and directly applicable document appears at the top of internal search results.

NDCG serves as the gold-standard metric in academic and industrial benchmarks like TREC and BEIR for comparing ranking algorithms. In production A/B tests, engineers track NDCG@5 or NDCG@10 to measure the impact of a new retrieval model, embedding, or query expansion technique. A statistically significant lift in NDCG provides a strong, user-centric signal for deploying a new ranking system to production.

Cumulative Gain (CG) is the foundational, non-normalized sum of the relevance scores of the top-k retrieved documents. It is the simplest gain metric, calculated as:

CG@k = Σ_{i=1}^{k} rel_i

• Key Limitation: CG does not account for the rank position of a relevant document. A system that returns a highly relevant document at rank 10 receives the same CG@10 score as one that returns it at rank 1, despite the latter being far more useful to a user.

Discounted Cumulative Gain (DCG) refines CG by introducing a logarithmic discount factor that penalizes relevant documents appearing lower in the ranked list. The core assumption is that a user's utility (gain) from a document diminishes the further down the list they must look.

• Standard Formula: DCG@k = Σ_{i=1}^{k} rel_i / log_2(i + 1)
• Alternative Formula: Uses an exponential gain for stronger emphasis on high relevance: DCG@k = Σ_{i=1}^{k} (2^{rel_i} - 1) / log_2(i + 1)
• Purpose: DCG provides a more realistic assessment of ranking quality than CG by modeling user patience and the decreasing value of lower-ranked results.

Ideal Discounted Cumulative Gain (IDCG@k) is the maximum possible DCG score achievable for a given query. It is calculated by taking all known relevant documents for that query, sorting them in perfect descending order of relevance, and then computing the DCG of that ideal ranking.

• Role as Normalizer: IDCG serves as the gold standard benchmark against which a system's actual DCG is compared. It represents the theoretical upper bound of performance for that specific query and set of relevance judgments.
• Calculation: IDCG@k = DCG@k of the ideally sorted list of relevant documents.

Mean Average Precision (MAP) is another core ranking metric that averages the Average Precision (AP) scores across a set of queries. AP for a single query is the mean of the precision values calculated at each point a new relevant document is retrieved.

• Comparison with NDCG:
  • MAP is binary; it treats documents as either relevant or non-relevant.
  • NDCG supports graded relevance (e.g., scores of 0, 1, 2, 3).
  • MAP strongly emphasizes recall—the retrieval of all relevant items—and penalizes systems that miss relevant documents deeply in the list.
  • NDCG emphasizes the ranking of highly relevant items at the very top of the list.

Precision@k and Recall@k are fundamental cut-off-based metrics for evaluating retrieval systems.

• Precision@k: Measures the fraction of relevant documents in the top k retrieved results. P@k = (# of relevant docs in top k) / k

• Recall@k: Measures the fraction of all known relevant documents found in the top k results. R@k = (# of relevant docs in top k) / (Total # of relevant docs)

• Relationship to NDCG: While P@k and R@k provide a snapshot at a specific depth, NDCG provides a single, comprehensive score that accounts for the entire ranked list up to k, incorporating both the presence and the graded importance of relevant documents at their specific positions.

Mean Reciprocal Rank (MRR) is a simple but focused metric for tasks where only the first relevant result matters. It is the average of the reciprocal ranks of the first relevant item for each query in a set Q.

• Formula: MRR = (1/|Q|) * Σ_{q=1}^{|Q|} (1 / rank_of_first_relevant_result_q)
• Use Case: Ideal for evaluating question-answering systems or web search where the user's primary goal is to find a single correct answer or page.
• Contrast with NDCG: MRR cares only about the first hit. NDCG provides a more nuanced evaluation for scenarios where multiple relevant documents (e.g., research papers, product listings) are valuable, and their relative order is important.