Score Calibration

A parametric calibration method that fits a logistic regression model to the raw scores of a classifier. It learns two parameters—A (slope) and B (intercept)—to map unbounded scores to the [0,1] interval. Particularly effective when the raw score distribution is sigmoid-shaped and the calibration dataset is small. The transformation is: P(y=1|x) = 1 / (1 + exp(A * f(x) + B)), where f(x) is the raw model score. Platt scaling works well for support vector machines and neural networks but assumes a monotonic relationship between scores and probabilities.

A non-parametric calibration technique that learns a piecewise constant, monotonically increasing mapping from raw scores to calibrated probabilities. Unlike Platt scaling, it makes no assumptions about the functional form of the relationship. The algorithm solves a pool-adjacent-violators problem to find the step function that minimizes squared error while preserving monotonicity. Best suited for large calibration datasets where the empirical score-probability relationship is non-sigmoidal. Risk of overfitting increases with small datasets due to its high flexibility.

A simple yet effective calibration method for neural network classifiers that divides the output logits by a single scalar parameter T (temperature) before applying softmax. When T > 1, the probability distribution softens, reducing overconfidence. When T < 1, it sharpens. The optimal T is learned by minimizing negative log-likelihood on a held-out validation set. Critically, temperature scaling preserves the rank order of predictions—it only adjusts confidence, making it ideal when accuracy metrics must remain unchanged.

The primary metric for evaluating calibration quality. ECE partitions predictions into M equal-width bins based on confidence, then computes the weighted average of the absolute difference between accuracy and average confidence within each bin. Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|. A perfectly calibrated model has ECE = 0. Lower ECE indicates better alignment between predicted probabilities and empirical frequencies. Common bin count: M=10 or M=15.

A visual diagnostic tool for calibration assessment that plots observed frequency against predicted probability. A perfectly calibrated model follows the identity line (y=x). Deviations above the diagonal indicate underconfidence—the model predicts lower probabilities than empirical outcomes warrant. Deviations below indicate overconfidence—the model is too certain. Bars extending above and below each point show confidence intervals. Essential for identifying systematic miscalibration patterns that aggregate metrics like ECE might obscure.

A flexible calibration framework that models the relationship between raw scores and probabilities using a Beta distribution. It learns parameters for two Beta distributions—one for each class—and computes calibrated probabilities via their likelihood ratio. Beta calibration generalizes Platt scaling and handles multi-class problems through a Dirichlet extension. Particularly effective when scores exhibit asymmetric distributions or when Platt scaling's logistic assumption fails. Requires careful regularization to prevent overfitting on small datasets.

The process of scaling the uncalibrated relevance scores from disparate retrieval sources onto a common, comparable scale before they can be meaningfully merged. This is a prerequisite step that calibration renders more accurate. Common techniques include:

• Min-Max Normalization: Rescales scores to a [0, 1] range
• Z-Score Normalization: Centers scores around zero with unit variance
• Sum-to-Unity: Divides each score by the sum of all scores

Without proper normalization, a system producing scores in [0.9, 1.0] will dominate one producing scores in [0.1, 0.9] in a linear combination.

A linear combination method for merging retrieval results where each system's calibrated score for a document is multiplied by a pre-defined weight, reflecting that system's relative importance. Calibration is a prerequisite for this method to be mathematically sound.

• Requires scores to be on the same scale and well-calibrated
• Weights can be static or learned via Learning to Rank (LTR)
• A poorly calibrated system with a high weight will corrupt the final ranking
• Contrast with RRF, which avoids this sensitivity entirely

A re-ranking methodology where a transformer model processes the concatenated query and candidate document simultaneously to produce a fine-grained relevance score. The raw logit output of a cross-encoder is typically uncalibrated and requires post-processing.

• Platt Scaling or Isotonic Regression is often applied to convert logits to probabilities
• Offers higher precision than bi-encoders at the cost of computational latency
• Calibrated cross-encoder scores can be fused with first-stage retrieval scores for hybrid ranking

A supervised machine learning approach that trains a model to combine multiple relevance signals into an optimal ranking function. LTR models can implicitly learn to calibrate and weight features during training.

• Pointwise: Predicts absolute relevance scores
• Pairwise: Learns relative preferences between document pairs
• Listwise: Optimizes the entire ranked list (e.g., LambdaMART optimizing NDCG)
• Feature inputs often include calibrated scores from upstream retrievers

A specific parametric method for score calibration that fits a logistic regression model to the raw scores of a classifier. It transforms unbounded model outputs into well-calibrated posterior probabilities.

• Learns two parameters: a scalar A and a bias B
• Formula: P(y=1 | score) = 1 / (1 + exp(A * score + B))
• Originally developed for Support Vector Machines (SVMs)
• Widely used to calibrate neural network logits before fusion
• Requires a held-out calibration set to avoid overfitting