Information Gain

Unlike static relevance scores, Information Gain measures the marginal utility of a document. It quantifies the reduction in uncertainty about a query topic after reading a specific result, given the user has already processed all documents ranked above it. A document with high Information Gain introduces new entities, facts, or perspectives not present in the previously seen set, even if its raw relevance score is lower than a redundant document.

The metric is formally rooted in Shannon entropy and Kullback-Leibler divergence. It calculates the divergence between the probability distribution of relevant information in the entire corpus and the distribution of information already covered by the preceding result set. The formula effectively subtracts the information content of previously ranked documents from the candidate document's content, ensuring the search engine result page (SERP) maximizes the total unique information presented to the user.

Information Gain is not a static document property; it is a query-dependent, order-sensitive calculation. The score of a document changes dynamically based on which other documents are ranked above it. This requires a greedy or iterative re-ranking process where the system continuously recalculates the remaining information need after each document is selected, making it computationally more intensive than standard pointwise ranking models.

A primary function is to explicitly penalize near-duplicate content and information cascades. If ten documents all report the same breaking news fact, the first one receives a high gain score, while the subsequent nine receive scores approaching zero. This forces the retrieval system to surface diverse subtopics, alternative viewpoints, or deeper analysis rather than filling the SERP with identical information from different domains.

Information Gain is a theoretical evolution of the Maximal Marginal Relevance algorithm. While MMR uses a linear combination of relevance and novelty based on keyword overlap, Information Gain uses a probabilistic framework to measure true semantic redundancy. It leverages language models to determine if a new document actually teaches the user something new, rather than just using different words to describe the same concept.

Modern implementations estimate Information Gain using large language models (LLMs) as proxy knowledge bases. The system prompts an LLM to generate a comprehensive answer from the top-ranked documents. It then evaluates a candidate document by measuring the semantic overlap between the candidate's content and the already generated answer. If the candidate introduces entities or claims not in the generated answer, it receives a high gain score.