Platt Scaling

Platt scaling fits a logistic regression model with two parameters (A, B) to the classifier's raw outputs (logits). The mapping is defined as: P(y=1 | s) = 1 / (1 + exp(A * s + B)), where s is the classifier score. This sigmoid function ensures outputs are valid probabilities between 0 and 1. It assumes the uncalibrated scores follow a sigmoidal distribution, which is often a reasonable approximation for many discriminative models like SVMs.

The method is post-hoc, meaning it is applied after the base model is fully trained. It requires a separate calibration dataset, distinct from the training and test sets. This dataset is used solely to learn the parameters A and B via maximum likelihood estimation. Using the training data for calibration would lead to overfitting and unreliable probability estimates on new data. The size of this set directly impacts the stability of the parameter estimates.

The standard formulation is designed for binary classification. It calibrates the scores for the positive class. For multi-class problems, the standard approach is the one-vs-rest (OvR) strategy: train a separate Platt scaling calibrator for each class against all others, then normalize the resulting probabilities across classes. This can be computationally intensive and may not guarantee perfect multi-class calibration.

The calibration process is computationally inexpensive. Fitting the logistic regression model is a convex optimization problem that converges quickly. At inference time, applying the calibration is a simple affine transformation of the logit followed by a sigmoid, adding negligible latency. This makes it highly suitable for production systems where throughput and low latency are critical.

The method's main weakness is its susceptibility to overfitting when the calibration set is small. With limited data, the estimated parameters (A, B) can have high variance, leading to poorly calibrated probabilities on new data. This risk is mitigated by using a sufficiently large calibration set (typically hundreds to thousands of samples) or by applying regularization (e.g., L2 penalty) during the logistic regression fit.

Platt scaling was originally developed for Support Vector Machines (SVMs), which output uncalibrated decision values. It is equally effective for other models that produce scores interpretable as confidence measures, including:

• Linear models (logistic regression, although often already calibrated)
• Boosted trees (e.g., XGBoost, LightGBM)
• Neural networks (using pre-softmax logits) It is less commonly applied to models like naive Bayes, which may already produce distorted probability estimates.

In clinical settings, a model's predicted probability directly informs treatment decisions. A radiologist needs to know if a '90% confidence' in a tumor detection is truly a 90% likelihood. Platt scaling calibrates these scores, enabling:

• Informed patient triage based on reliable risk stratification.
• Cost-benefit analysis for invasive follow-up procedures.
• Integration into clinical decision support systems where overconfidence can lead to harmful false assurances.

Transaction fraud models output a 'fraud score.' For operational efficiency, analysts set a probability threshold to flag transactions for review. Platt scaling ensures that a score of 0.95 means a transaction has a 95% chance of being fraudulent, allowing for:

• Precise resource allocation: High-confidence alerts are prioritized.
• Accurate false positive rate control, directly impacting customer experience and operational costs.
• Regulatory reporting that requires statistically sound estimates of risk exposure.

Deep neural networks, particularly those trained with cross-entropy loss, are notoriously overconfident. Their softmax outputs are not true probabilities. Platt scaling is applied post-training to:

• Rectify miscalibration in models like ResNets, Vision Transformers, and large language models (LLMs) for classification tasks.
• Serve as a baseline method against which newer techniques like temperature scaling are compared.
• Provide a simple, effective fix without retraining the entire model, crucial for production efficiency.

Conformal prediction is a framework for generating prediction sets with guaranteed statistical coverage (e.g., 90% of sets will contain the true label). It often uses a non-conformity score. Platt scaling can be used to generate well-calibrated probabilities that serve as ideal non-conformity scores, leading to:

• Tighter, more efficient prediction sets compared to using raw model scores.
• Provable guarantees of coverage that hold in practice because the underlying probabilities are calibrated.
• Applications in high-stakes areas like autonomous driving (predicting pedestrian intent) where understanding uncertainty is safety-critical.

On edge devices, model retraining is often infeasible. Platt scaling offers a lightweight post-processing step. A simple logistic regression model can be fitted on a small calibration set and deployed alongside the main model to:

• Dramatically improve decision reliability with minimal compute overhead.
• Enable dynamic confidence-based filtering; for example, a wildlife camera trap can discard low-confidence images, saving bandwidth.
• Maintain calibration even as the primary model becomes stale, extending its useful life.

Platt scaling is not a universal solution. Its effectiveness hinges on specific conditions:

• Binary Classification Focus: It is designed for binary tasks. Multi-class problems require the One-vs-Rest strategy, fitting a separate calibrator per class.
• Parametric Assumption: It assumes the relationship between logits and true probability follows a sigmoid curve. If this assumption is violated (e.g., a complex, non-monotonic relationship), non-parametric methods like Isotonic Regression may be superior.
• Calibration Set Quality: Performance degrades if the calibration data is not i.i.d. with the test/production data or is too small, leading to poorly estimated parameters.

A single-parameter post-hoc calibration method for neural networks. It applies a scalar 'temperature' (T) to the logits before the softmax: softmax(logits / T). A T > 1 softens the output distribution (reduces overconfidence), while T < 1 sharpens it. It's a generalization of Platt scaling for multi-class settings and is often the first baseline due to its simplicity and low risk of overfitting.

A non-parametric post-hoc calibration method. It fits a piecewise constant, non-decreasing function to map raw classifier scores to calibrated probabilities. Unlike parametric methods (Platt, Temperature), it makes minimal assumptions about the shape of the calibration mapping. It is more flexible and can model complex miscalibration patterns but requires more calibration data and is prone to overfitting on small sets.

The primary scalar metric for quantifying miscalibration. It works by:

• Binning predictions based on their confidence score (e.g., 0-0.1, 0.1-0.2).
• For each bin, calculating the average confidence and the empirical accuracy.
• Computing a weighted average of the absolute difference between confidence and accuracy across all bins. A lower ECE indicates better calibration. It is a standard benchmark for comparing calibration methods.

The visual counterpart to the ECE. It is a diagnostic plot where:

• The x-axis represents the average predicted confidence within a bin.
• The y-axis represents the observed empirical accuracy within that bin.
• A perfectly calibrated model's points lie on the diagonal (y=x) line. Deviations from the diagonal reveal the nature of miscalibration: points below the line indicate overconfidence, while points above indicate underconfidence.

A proper scoring rule that evaluates the overall quality of probabilistic predictions. For binary classification, it is the mean squared error between the predicted probability and the true outcome (0 or 1). The score combines two aspects:

• Calibration: How well confidence matches accuracy.
• Refinement/Sharpness: How concentrated the predictions are near 0 or 1. A lower Brier score is better. It is used both as a training loss and an evaluation metric.

A held-out dataset used exclusively for fitting post-hoc calibration parameters. Critical requirements:

• Must be distinct from the training and test sets.
• Should be representative of the expected production data distribution.
• Size matters: Parametric methods (Platt, Temperature) need less data (~1000 samples), while non-parametric methods (Isotonic) need more to avoid overfitting. Using the test set for calibration is a methodological error that invalidates performance estimates.