Probability Calibration

Probability calibration is the post-processing adjustment of a classifier's output score so that the predicted probability reflects the true empirical frequency of the positive class. In a perfectly calibrated model, if you group all predictions with a score of 0.7, exactly 70% of those instances should be fraudulent. This is critical for risk scoring, where a raw SVM or Random Forest output is often a distorted distance metric, not a true probability. Calibration bridges the gap between a ranking model and a probability estimator.

A reliability diagram is the primary visual diagnostic for calibration. It plots the mean predicted probability against the observed fraction of positives across binned predictions.

• Perfect Calibration: The plot follows the diagonal identity line.
• Under-Confidence: The curve sits between the diagonal and the center horizontal line (predictions are too moderate).
• Over-Confidence: The curve follows an S-shape steeper than the diagonal (predictions are too extreme).

A histogram of prediction frequencies is usually included to show where the model concentrates its scores.

The Expected Calibration Error is a scalar summary statistic that quantifies miscalibration. It is calculated by partitioning predictions into M equally-spaced bins, then computing the weighted average of the absolute difference between the accuracy and the average confidence within each bin.

• Formula: ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|
• Interpretation: A lower ECE indicates better calibration. ECE = 0 is perfect.
• Limitation: ECE is sensitive to the binning strategy and can be gamed by models that predict extreme probabilities poorly but score well on average.

Platt Scaling is a parametric method for calibrating binary classifiers. It fits a logistic regression model on top of the original classifier's raw output scores (logits or SVM distances).

• Mechanism: It learns two scalar parameters (A and B) such that P(y=1|x) = 1 / (1 + exp(A * f(x) + B)).
• Best Use Case: Works best when the uncalibrated distortion is sigmoidal, which is common for Support Vector Machines.
• Caveat: It can overfit on small datasets, so it requires a held-out calibration set distinct from the training data.

Isotonic Regression is a non-parametric calibration method that fits a piecewise constant, monotonically increasing function to map raw scores to calibrated probabilities. It makes no assumptions about the shape of the distortion function.

• Mechanism: It minimizes a squared error loss subject to the constraint that the mapping is non-decreasing.
• Advantage: More flexible than Platt Scaling and can correct any monotonic distortion.
• Disadvantage: Prone to overfitting on small datasets, often resulting in a step-function mapping that doesn't generalize well to unseen scores.

The Brier Score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions. It is the mean squared difference between the predicted probability and the actual binary outcome.

• Formula: BS = (1/N) * Σ (p_i - y_i)²
• Range: 0 (perfect) to 1 (worst). A score of 0.25 represents a random guess for a balanced dataset.
• Decomposition: The Brier Score can be mathematically decomposed into Refinement (resolution + uncertainty) and Calibration loss, allowing you to isolate miscalibration from the model's inherent discriminative power.