Platt Scaling

Platt Scaling learns a scalar A and B parameter to transform raw classifier scores f(x) into calibrated probabilities: P(y=1 | f) = 1 / (1 + exp(A * f + B)). The parameter A controls the slope of the sigmoid, sharpening or softening probabilities, while B shifts the decision threshold. This parametric form assumes the relationship between raw scores and empirical frequencies follows a logistic curve, making it ideal for Support Vector Machines and other maximum-margin classifiers that exhibit sigmoidal distortion.

The parameters A and B are optimized using maximum likelihood estimation on a dedicated calibration dataset, distinct from both training and test sets. Using the same data for training and calibration introduces severe overfitting bias, producing unrealistically low calibration errors. The logistic regression model is trained with the raw classifier scores as the single input feature and the binary ground-truth labels as targets. Cross-validation on the training set can generate out-of-fold predictions for calibration when data is scarce.

A critical implementation detail is ensuring the calibration set is strictly independent of the data used to train the base classifier. The standard protocol involves a three-way split:

• Training set: Fits the base classifier parameters
• Calibration set: Fits the Platt Scaling parameters A and B
• Test set: Evaluates the final Expected Calibration Error (ECE) Failing to maintain this separation leads to optimistic calibration estimates that do not generalize to production data.

Platt Scaling is a two-parameter generalization of Temperature Scaling. Temperature Scaling divides all logits by a single scalar T, which is equivalent to Platt Scaling with B fixed at 0. The additional bias term B in Platt Scaling allows it to correct systematic overconfidence or underconfidence that is not centered around zero. For modern neural networks, Temperature Scaling often matches Platt Scaling's performance while being simpler, but Platt Scaling remains superior when the base classifier exhibits asymmetric confidence distortion.

Platt Scaling assumes the raw classifier scores are monotonically related to the true posterior probabilities. It fails when:

• The score distribution is non-monotonic, where higher scores do not consistently indicate higher empirical accuracy
• The base model produces pathological score distributions with heavy tails or multi-modal behavior
• The calibration dataset is too small, causing high-variance estimates of A and B In these cases, non-parametric methods like Isotonic Regression may be more appropriate despite their higher data requirements.

John Platt introduced this method in 1999 specifically to extract calibrated probabilities from Support Vector Machines, which natively output an uncalibrated signed distance from the decision boundary. The technique transformed SVMs from pure discriminative classifiers into probabilistic models suitable for decision-theoretic applications like cost-sensitive classification and reject-option scenarios. The original paper demonstrated that the sigmoid mapping consistently improved probability estimates over naive normalization of SVM outputs.

The primary empirical metric for diagnosing miscalibration. ECE partitions predictions into M equal-width bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A lower ECE indicates better calibration.
• Limitation: Sensitive to binning strategy; does not distinguish overconfidence from underconfidence.

A powerful non-parametric alternative to Platt Scaling that learns a piecewise constant, monotonically increasing function to map raw scores to calibrated probabilities.

• Advantage: Makes no assumption about the functional form of the miscalibration; can fit any monotonic distortion.
• Disadvantage: Prone to overfitting on small validation sets due to its high flexibility.
• Often implemented via the Pair-Adjacent Violators (PAV) algorithm.
• Best suited when the relationship between scores and empirical frequency is complex and non-sigmoidal.

A strictly proper scoring rule that measures the mean squared error between the predicted probability vector and the one-hot encoded true outcome.

• Formula: BS = (1/N) Σ ||p_i - y_i||²
• Decomposes into refinement (sharpness) and calibration components.
• A lower Brier Score indicates better overall probabilistic performance.
• Unlike ECE, it jointly penalizes poor calibration and poor discrimination, making it a holistic metric.

Also known as cross-entropy loss, NLL is a proper scoring rule that heavily penalizes confident misclassifications. It is the objective function minimized by Platt Scaling's logistic regression fit.

• NLL = -log(p_y), where p_y is the predicted probability of the true class.
• A perfectly calibrated model with high confidence in correct predictions achieves low NLL.
• Sensitivity: Approaches infinity as the probability of the correct class approaches zero, making it a strict measure of calibration quality.

The standard visual diagnostic for calibration. It plots predicted probability on the x-axis against observed empirical frequency on the y-axis.

• A perfectly calibrated model follows the identity diagonal (y = x).
• Below diagonal: Overconfidence (predicted probability > observed frequency).
• Above diagonal: Underconfidence.
• Often displayed alongside a histogram of prediction counts per bin to indicate where most samples lie.
• Platt Scaling's sigmoid mapping can be overlaid to visualize the correction applied.