Platt Scaling

Platt scaling applies a sigmoid function to the classifier's raw scores (e.g., SVM margins or unscaled logits). It learns two parameters via maximum likelihood estimation on a held-out validation set:

• Parameter A (scale): Controls the slope of the sigmoid.
• Parameter B (location): Shifts the sigmoid along the score axis. The calibrated probability for a score (s) is: (P(y=1|s) = \frac{1}{1 + \exp(A \cdot s + B)}). This simple transformation aligns confidence with empirical accuracy.

Unlike temperature scaling which can use the training set, Platt scaling must be applied to a separate validation dataset not used for the original model training. This prevents overfitting the calibration map. The process is:

1. Train the base classifier (e.g., SVM, neural network).
2. Generate raw scores for samples in the held-out validation set.
3. Fit the logistic regression (Platt scaling) model to map these scores to the true binary labels.
4. Apply the learned scaling parameters to new test data. This separation is critical for obtaining unbiased, generalizable probability estimates.

The original method is designed for binary classification. It calibrates scores from models like Support Vector Machines (SVMs), which output uncalibrated decision function margins, and early neural networks. For multi-class problems, the standard approach is the one-vs-rest (OvR) extension:

• Calibrate scores for each class against all others.
• The resulting probabilities are then typically renormalized (e.g., via softmax) to sum to 1. This extension can be less stable than direct multi-class methods like matrix scaling.

Platt scaling is a generalization of temperature scaling. Temperature scaling, used for modern neural networks, is a special case of Platt scaling where the parameter (B) (the bias/intercept) is fixed at 0. This means:

• Temperature Scaling: (P = \frac{1}{1 + \exp(s / T)}), learns only the temperature (T).
• Platt Scaling: (P = \frac{1}{1 + \exp(A \cdot s + B)}), learns both (A) and (B). Platt scaling is more flexible and is necessary for models whose scores are not centered around zero (like SVM margins).

Proper calibration via Platt scaling directly improves metrics that depend on accurate probability estimates:

• Log Loss (Negative Log-Likelihood): A proper scoring rule that is minimized when predicted probabilities match true frequencies. Miscalibrated models have poor log loss.
• Brier Score: The mean squared error between predicted probabilities and one-hot labels. Calibration reduces this error.
• Expected Calibration Error (ECE): The primary diagnostic metric for miscalibration. Platt scaling aims to minimize ECE, bringing the reliability diagram closer to the diagonal.
• Decision-making: Enables reliable cost-sensitive classification and risk assessment where probability thresholds have real-world consequences.

While foundational, Platt scaling has key limitations:

• Data Efficiency: Requires a sufficiently large held-out set for stable logistic regression fitting. Performance degrades with small data.
• Parametric Assumption: Assumes a monotonic sigmoidal relationship between scores and true probabilities. This can fail for models with pathological score distributions.
• Multi-class Complexity: The one-vs-rest extension can lead to poorly normalized probabilities if the binary calibrators are not consistent.
• Modern Context: For deep neural networks with cross-entropy loss, temperature scaling is often sufficient and more stable. Platt scaling remains crucial for models like SVMs or when scores have a non-zero bias.

In clinical decision support, a model's predicted probability of disease must match the true prevalence and risk. An uncalibrated model might output a 90% confidence for a condition that is only correct 60% of the time, leading to harmful overtreatment. Platt scaling recalibrates these scores so a '90%' prediction is correct approximately 90% of the time. This is critical for:

• Triage systems: Accurately prioritizing patient risk.
• Informing treatment thresholds: Enabling cost-benefit analysis based on reliable probabilities.
• Radiology AI: Calibrating confidence scores for tumor detection in medical imaging.

Credit scoring and fraud detection models require probabilities that faithfully represent default or fraud risk for precise pricing and resource allocation. Platt scaling transforms an SVM's or boosted tree's decision function scores into calibrated probabilities of default. This allows financial institutions to:

• Set accurate interest rates: Based on true per-customer risk.
• Optimize capital reserves: Using probabilities that align with empirical default rates.
• Prioritize fraud investigations: By providing reliable confidence that a transaction is fraudulent, ensuring investigators focus on the highest-risk cases.

In safety-critical applications like autonomous driving or content moderation, a model must know when it is uncertain and abstain. Platt scaling provides the well-calibrated probabilities needed to implement a reliable rejection option. A threshold (e.g., 0.95) is set on the calibrated probability; predictions below this are rejected for human review. This enables:

• High-precision automation: The system only acts when confidence is both high and accurate.
• Efficient human-in-the-loop workflows: Human experts review only the low-confidence, high-stakes cases.
• Dynamic risk management: The confidence threshold can be adjusted based on the cost of an error.

When comparing different classifiers (e.g., Random Forest vs. Neural Network), raw accuracy is insufficient if their probability outputs are miscalibrated. Platt scaling creates a level playing field by calibrating all models' outputs, allowing for fair comparison using proper scoring rules like Log Loss or the Brier Score. This is essential for:

• Hyperparameter tuning: Selecting models based on true probabilistic performance, not just accuracy.
• Production model selection: Choosing the model whose confidence scores are most trustworthy.
• Auditing and compliance: Demonstrating that a model's stated confidence aligns with reality, a requirement under some regulatory frameworks for algorithmic transparency.

In applications where different types of errors have vastly different costs (e.g., in spam filtering, a false positive is worse than a false negative), decision rules rely on probability estimates. Platt scaling ensures the probability P(class | x) is accurate, allowing optimal decisions that minimize expected cost. The decision rule becomes: predict class A if P(A | x) * Cost(False Negative) > P(B | x) * Cost(False Positive). This is used in:

• Marketing churn prediction: Where the cost of incorrectly predicting churn (and offering a discount) differs from missing a churning customer.
• Manufacturing defect detection: Where the cost of a faulty product escaping differs from the cost of unnecessary inspection.

While ensembles like Random Forests or Gradient Boosting Machines often improve accuracy, their averaged class probabilities can still be poorly calibrated, especially for imbalanced data. Platt scaling (or its multi-class extension, Platt-LIBSVM) can be applied to the ensemble's aggregated scores as a final calibration step. This process:

• Decouples training from calibration: The ensemble is trained to maximize discrimination, then a separate, simple model (logistic regression) is fit on a held-out set to calibrate.
• Improves reliability diagrams: Shifts the model's calibration curve closer to the ideal diagonal.
• Works with any base learner: Can calibrate scores from SVMs, trees, or neural networks, providing a unified post-processing interface.

Calibration error measures the discrepancy between a model's predicted confidence scores and its actual empirical accuracy. A perfectly calibrated model's predicted probability of being correct (e.g., 0.8) should match its observed accuracy (80% of the time). Key metrics include:

• Expected Calibration Error (ECE): A scalar summary statistic calculated by binning predictions by confidence and averaging the absolute difference between average confidence and accuracy per bin.
• Maximum Calibration Error (MCE): The maximum discrepancy observed across all confidence bins.
• Brier Score: A proper scoring rule that decomposes into calibration loss and refinement loss. High calibration error indicates a model is systematically overconfident or underconfident, which Platt scaling directly aims to correct.

Temperature scaling is a single-parameter, post-hoc calibration technique closely related to Platt scaling. It applies a learned scalar 'temperature' (T) to a model's logits before the softmax: softmax(logits / T). Key characteristics:

• Simplicity: Learns only one parameter, making it less prone to overfitting on small calibration sets.
• Applicability: Primarily used for multi-class classification, unlike Platt scaling's binary focus (though multi-class extensions exist).
• Effect: A temperature T > 1 softens the output distribution (reducing overconfidence), while T < 1 sharpens it. It preserves the original ranking of classes (argmax prediction) while adjusting the confidence distribution. It is often the baseline against which more complex methods like Platt scaling are compared.

Conformal prediction is a model-agnostic, distribution-free framework that produces statistically valid prediction sets or intervals with guaranteed coverage, offering a frequentist approach to uncertainty. Core mechanics:

• It uses a nonconformity score (e.g., 1 - predicted probability for the true label) on a held-out calibration set.
• It calculates a data-dependent threshold based on a user-specified error rate (alpha).
• At test time, it outputs the set of labels whose nonconformity scores fall below this threshold. Guarantee: For any classifier and data distribution, the true label will be contained in the prediction set with probability 1 - alpha. Unlike Platt scaling, which outputs calibrated probabilities, conformal prediction outputs sets with explicit coverage guarantees, making it valuable for risk-sensitive applications.

Selective classification, or classification with a rejection option, is a paradigm where a model is allowed to abstain from making a prediction when its confidence is below a chosen threshold. This creates a trade-off between coverage (fraction of samples predicted on) and risk (error rate).

• A risk-coverage curve visualizes this trade-off.
• Confidence scores from methods like Platt scaling are directly used as the abstention criterion.
• The goal is to maximize coverage for a target risk level or minimize risk for a target coverage. This is critical for deploying models in high-stakes environments where incorrect predictions are costly, allowing the system to defer difficult cases to a human expert.

A Bayesian Neural Network treats its weights as probability distributions rather than fixed point estimates. This provides a principled, inherent framework for uncertainty estimation through Bayesian inference. Key aspects:

• Epistemic Uncertainty: Captured by the distribution over weights, reflecting model uncertainty due to limited data.
• Aleatoric Uncertainty: Often modeled by the network's output distribution, capturing inherent data noise.
• Prediction: Made by integrating over all possible weights (marginalization), yielding a predictive distribution. Practical approximations like Monte Carlo Dropout and Deep Ensembles are used to estimate this process. Unlike post-hoc calibration (Platt scaling), BNNs bake uncertainty quantification directly into the model's architecture and training objective.

A proper scoring rule is a function that measures the quality of a probabilistic forecast, encouraging the forecaster to report their true, honest belief. They are the theoretical foundation for training and evaluating calibrated models. Common examples:

• Negative Log-Likelihood (NLL)/Log Loss: Penalizes the model based on the negative logarithm of the probability it assigns to the true label. It is strictly proper.
• Brier Score: The mean squared error between the predicted probability vector and the one-hot encoded true label. It is also strictly proper. Importance: Minimizing a proper scoring rule during training theoretically leads to well-calibrated models. Post-hoc methods like Platt scaling are often needed because neural networks trained with NLL may still be miscalibrated due to overparameterization and other non-Bayesian effects.