Calibration via Platt

Platt scaling fits a logistic regression model with two parameters (a scaling weight and a bias term) to map the classifier's raw scores (logits) to calibrated probabilities. The transformation is defined as: P(y=1 | s) = 1 / (1 + exp(A * s + B)), where s is the classifier's score. This assumes the uncalibrated scores follow a sigmoidal distribution, which is often a valid approximation for many discriminative models like SVMs and neural networks.

The logistic regression parameters (A, B) are not learned during the original model training. They are estimated using a separate calibration set—a held-out dataset not used for training or final testing. This set should be representative of the target distribution. The method minimizes the negative log-likelihood on this set, treating the true class labels as the target for the logistic regression. Using the training data for calibration would lead to overfitting and unreliable probability estimates.

The standard Platt scaling formulation is designed for binary classification. It calibrates the scores for the positive class. For multi-class problems, the common extension is the One-vs-Rest (OvR) strategy: calibrate each class against all others independently, then normalize the resulting probabilities across classes (e.g., via softmax). However, this can be computationally intensive and may not guarantee well-calibrated multi-class probabilities as effectively as dedicated multi-class methods like temperature scaling.

Platt scaling is a post-hoc method, meaning it is applied after a model is fully trained, without modifying its internal parameters. It is also model-agnostic; it works on the scores output by any classifier, including Support Vector Machines (where it was originally developed), boosted trees, and neural networks. This decoupling allows calibration to be treated as a separate pipeline step, facilitating integration into existing MLOps workflows.

As a parametric method, Platt scaling can overfit when the calibration set is very small (e.g., fewer than 1000 instances). The logistic regression model may learn a mapping that fits noise rather than the true miscalibration pattern. In such cases, a less flexible, non-parametric method like isotonic regression may be more robust. The reliability of Platt scaling is highly dependent on the size and quality of the calibration data.

For neural networks, temperature scaling is a simpler, more constrained special case of Platt scaling. Temperature scaling uses a single parameter T (temperature) to scale all logits: logits_scaled = logits / T. Platt scaling, with its two parameters (A, B), is more flexible. However, this flexibility can be a disadvantage for neural nets, as the extra degree of freedom can lead to overfitting on the calibration set, whereas temperature scaling's single parameter often provides more reliable calibration with modern deep learning models.

Platt scaling is the specific parametric method for which 'Calibration via Platt' is a shorthand. It fits a logistic regression model with two parameters (a weight and a bias) to the logits (pre-softmax scores) of a binary classifier. This learned sigmoid function transforms the scores into well-calibrated probabilities that better reflect the true likelihood of correctness.

• Core Mechanism: Applies P_calibrated = σ(a * s + b), where s is the model's raw score, and a and b are learned on a held-out calibration set.
• Assumption: Assumes the raw scores follow a sigmoidal distribution, which often holds for outputs from models like SVMs or neural networks.

Temperature scaling is a simpler, single-parameter alternative to Platt scaling, primarily used for multi-class neural network classifiers. It applies a scalar 'temperature' (T) to soften or sharpen the logits before the softmax: softmax(logits / T).

• Key Difference: Uses one learned parameter vs. Platt's two, making it less flexible but more stable with limited calibration data.
• Primary Use Case: The de facto standard for calibrating modern neural networks with a softmax output layer, whereas Platt scaling is more common for binary outputs or scores from models like SVMs.

Isotonic regression is a powerful non-parametric calibration method that fits a piecewise constant, non-decreasing function to map scores to probabilities. It makes minimal assumptions about the underlying score distribution.

• Advantage over Parametric Methods: More flexible than Platt or temperature scaling and can model complex, non-sigmoidal miscalibration patterns.
• Disadvantage: Requires more calibration data to avoid overfitting and can be less stable than parametric methods on small datasets. 'Calibration via Isotonic' is its common implementation shorthand.

Expected Calibration Error (ECE) is the primary scalar metric for quantifying miscalibration. It works by:

1. Binning predictions based on their confidence score (e.g., 0.0-0.1, 0.1-0.2).
2. For each bin, calculating the difference between the average confidence (predicted probability) and the empirical accuracy (fraction correct).
3. Taking a weighted average of these absolute differences.

A lower ECE indicates better calibration. It is the standard benchmark for evaluating methods like Platt scaling.

A calibration set (or hold-out validation set) is a critical data partition used exclusively for fitting post-hoc calibration methods. It must be distinct from the training data (to avoid overfitting) and the test data (to ensure unbiased evaluation).

• Purpose for Platt Scaling: This set provides the (score, true label) pairs used to learn the logistic regression parameters a and b.
• Size Considerations: Typically requires hundreds to thousands of samples. Too small a set leads to poorly estimated parameters; using the test set for calibration invalidates performance metrics.

Post-hoc calibration is the overarching category of techniques that adjust a trained model's outputs without modifying its internal weights. Platt scaling is a canonical example.

• Key Principle: Treats the base model as a fixed black box that produces (potentially miscalibrated) scores. A separate, lightweight calibration function is then learned on top.
• Workflow: 1. Train model. 2. Generate scores on a calibration set. 3. Learn calibration mapping (e.g., sigmoid for Platt). 4. Apply mapping to new predictions.
• Contrast with Calibration-Aware Training: Does not change the core training loop, making it simple to deploy on existing models.