Learning to Rank (LTR)

LTR algorithms are categorized by how they frame the ranking problem:

• Pointwise: Treats ranking as regression or classification on individual documents. Predicts a relevance score for each query-document pair independently, ignoring relative order.
• Pairwise: Frames ranking as a binary classification problem on document pairs. The model learns to predict which of two documents is more relevant for a given query, using loss functions like RankNet or Hinge Loss.
• Listwise: Directly optimizes the entire ranked list. Loss functions like ListNet and LambdaRank evaluate the permutation as a whole, targeting list-level metrics such as NDCG.

LambdaMART is the canonical LTR algorithm and a multiple additive regression tree (MART) model. It bypasses the non-differentiable nature of ranking metrics by using lambda gradients.

• Lambda Gradient: Encodes the cost of swapping a pair of documents, weighted by the change in NDCG. This heuristic allows gradient descent to directly optimize the ranking metric.
• Ensemble Method: Combines an ensemble of weak decision trees, each trained sequentially to correct the errors of the previous trees.
• Feature Engineering: Relies on hand-crafted features like BM25, PageRank, and click-through rates, making it highly interpretable compared to neural models.

LTR models depend on a rich set of hand-engineered features that capture relevance signals:

• Lexical Features: BM25 score, TF-IDF, exact match ratios, and proximity scores.
• Semantic Features: Cosine similarity between query and document embeddings from a Bi-Encoder.
• Quality Signals: PageRank, document freshness, click-through rate, and dwell time.
• Query Features: Query length, entity type, and intent classification. These features are concatenated into a feature vector for each query-document pair, allowing the model to learn complex, non-linear combinations.

Normalized Discounted Cumulative Gain (NDCG) is the standard metric for evaluating LTR models. It measures the quality of a ranked list by:

• Graded Relevance: Uses multi-level relevance judgments (e.g., 0-4) rather than binary labels.
• Position Discounting: Divides the gain by the logarithm of the rank position, heavily weighting top-ranked items.
• Ideal Normalization: Divides the DCG by the IDCG (Ideal DCG) to produce a score between 0 and 1, enabling comparison across queries with different numbers of relevant documents. NDCG@10 is a common cutoff, focusing on the first page of results.

LTR requires labeled training data where each query-document pair has a relevance judgment:

• Explicit Judgments: Human raters assign graded relevance labels following strict guidelines.
• Implicit Feedback: Derived from user behavior like clicks, dwell time, and conversions. Requires click models to debias position bias.
• Inverse Propensity Scoring (IPS): A debiasing technique that reweights clicks by the inverse of their examination probability, correcting for the fact that top-ranked items are more likely to be clicked regardless of relevance.

In contemporary search architectures, LTR typically operates as a second-stage ranker in a cascade:

• Stage 1 (Retrieval): A lightweight model (BM25 or vector search) retrieves hundreds of candidates from millions.
• Stage 2 (LTR): A LambdaMART model re-ranks the top-k candidates using hundreds of hand-crafted features, balancing precision and latency.
• Stage 3 (Neural Re-Ranking): A Cross-Encoder may further refine the top 10-20 documents with full attention. This tiered approach ensures that computationally expensive models are only applied to the most promising candidates.

The standard listwise evaluation metric for Learning to Rank systems. NDCG measures the quality of a ranked list by:

• Discounting relevance gains logarithmically by position
• Normalizing against the ideal ranking (IDCG)
• Heavily weighting top-ranked precision

A perfect NDCG@10 score of 1.0 indicates the model placed all relevant documents in the optimal order within the top 10 positions.

A multi-stage retrieval architecture where progressively more expensive models narrow the candidate set:

• Stage 1: Fast vector search or BM25 retrieves thousands of candidates
• Stage 2: A lightweight Bi-Encoder or LTR model scores the top hundreds
• Stage 3: A computationally intensive Cross-Encoder reranks only the top-k

This design optimizes the latency-relevance trade-off, keeping LTR models in the sweet spot of moderate cost and high feature expressiveness.

A systematic error where user click feedback used to train LTR models is confounded by the tendency of users to click on top-ranked items regardless of relevance. Mitigation requires Inverse Propensity Scoring (IPS) to debias training data by weighting clicks inversely to their examination probability. Failing to correct position bias causes LTR models to reinforce existing ranking flaws rather than learning true relevance.

A rank aggregation algorithm that combines the ranked lists from multiple retrieval or ranking stages by computing a weighted sum of the reciprocal of each document's rank position. RRF effectively normalizes disparate score distributions without requiring calibration, making it ideal for fusing the output of a LambdaMART LTR model with a neural dense retriever in hybrid search pipelines.