Learning to Rank (LTR)

LTR algorithms are categorized by their training objective:

• Pointwise: Treats ranking as regression or classification on individual documents. Predicts an absolute relevance score for each query-document pair independently.
• Pairwise: Learns to predict the relative order between pairs of documents. The model is trained to minimize the number of inversions in the ranked list.
• Listwise: Optimizes a ranking metric like NDCG or MAP directly over the entire list of results, capturing the interdependence of document positions.

Unlike neural re-rankers that learn representations from raw text, traditional LTR relies on hand-crafted features extracted from query-document pairs:

• Query-Dependent Features: BM25 score, term frequency, proximity of query terms in the document.
• Query-Independent Features: PageRank, document length, freshness, click-through rate.
• Query-Level Features: Query length, query intent classification. The quality of these features directly determines the ceiling of model performance.

LambdaMART is the dominant pairwise LTR algorithm, combining the MART (Multiple Additive Regression Trees) gradient boosting framework with a gradient function derived from ranking metrics. Its key innovation is that the gradient for a document pair is scaled by the change in NDCG that would result from swapping their positions. This directly optimizes for the ranking metric while maintaining the efficiency of gradient boosting on decision trees.

LTR requires ground-truth labels for training, typically obtained through human annotation:

• Binary Judgments: Relevant vs. non-relevant.
• Graded Judgments: Multi-level scales (e.g., Perfect, Excellent, Good, Fair, Bad) that capture degrees of relevance.
• Click-Through Data: Implicit feedback from user interactions, treated as noisy relevance signals. The quality and consistency of these judgments are critical; inter-annotator agreement metrics like Cohen's Kappa are used to measure reliability.

LTR typically operates as a second-stage re-ranker in a cascading pipeline:

1. Candidate Generation: A fast, high-recall method like BM25 or ANN retrieves hundreds of candidates.
2. LTR Re-ranking: The LTR model scores this candidate set using computationally intensive features, producing a final, precision-optimized ranking. This architecture balances the latency of feature extraction against the precision gains of a sophisticated ranking model.

LTR models are evaluated using list-aware metrics that account for position and graded relevance:

• NDCG (Normalized Discounted Cumulative Gain): Measures the gain of a document based on its position in the result list, normalized by the ideal ranking.
• MAP (Mean Average Precision): Computes the average precision at each relevant document across multiple queries.
• MRR (Mean Reciprocal Rank): The average of the reciprocal ranks of the first relevant result, useful for navigational queries.

A powerful pairwise-to-listwise gradient boosting algorithm that directly optimizes ranking metrics like NDCG. It combines the MART (Multiple Additive Regression Trees) framework with a gradient function derived from the lambda of a cost function defined on document pairs.

• Uses LambdaRank gradients to push relevant documents up and irrelevant ones down
• Dominated Yahoo! Learning to Rank Challenge for years
• Handles non-linear feature interactions automatically via tree ensembles

The process of extracting hand-crafted relevance signals from query-document pairs that serve as input to the ranking model. Unlike neural approaches, traditional LTR relies on explicit feature design.

• Query-level features: Query length, term frequency, intent category
• Document-level features: PageRank, document length, freshness, spam score
• Query-document features: BM25 score, TF-IDF, term proximity, click-through rate
• User behavior features: Dwell time, bounce rate, historical click probability

The three fundamental LTR training paradigms that define how the model learns from relevance judgments.

• Pointwise: Treats ranking as regression or classification on individual documents. Simple but ignores relative order
• Pairwise: Learns to predict which document in a pair is more relevant. RankNet and RankSVM are classic examples
• Listwise: Optimizes a ranking metric like NDCG or MAP directly across the entire result list. LambdaMART and ListNet fall here

Listwise approaches generally yield the best real-world performance.

The ground-truth labels that define how relevant each document is to a given query, forming the supervised training signal for LTR models.

• Typically collected via human assessors using graded scales (e.g., Perfect=4, Excellent=3, Good=2, Fair=1, Bad=0)
• Can be derived from implicit signals like click-through rate with position bias correction
• Quality of judgments directly determines model ceiling
• Inter-annotator agreement metrics like Cohen's Kappa measure label consistency

The most widely used listwise evaluation metric for ranking quality that accounts for both relevance grade and position. It normalizes against the ideal ranking to produce a score between 0 and 1.

• Discounted: Position weight decays logarithmically, valuing top results more
• Cumulative Gain: Sum of graded relevance scores across the ranked list
• Normalized: Divided by the DCG of the perfect ranking (IDCG)
• NDCG@10 is the standard cutoff for web search evaluation

A foundational pairwise neural network for LTR developed by Microsoft Research that uses a cross-entropy loss on document pairs. It introduced the concept of using gradient descent to learn ranking functions.

• Models the probability that document A is more relevant than document B
• Uses a sigmoid function on score differences to produce pairwise probabilities
• Served as the basis for LambdaRank and later LambdaMART
• Can be implemented with any differentiable scoring function, including deep networks