Model Calibration

The defining property of calibration is that predicted probabilities match empirical frequencies. For a perfectly calibrated model, when it predicts a 20% chance of sepsis, sepsis occurs in exactly 20 out of 100 such patients. This is measured using Expected Calibration Error (ECE), which bins predictions and computes the weighted average of the difference between accuracy and confidence in each bin. A lower ECE indicates better calibration.

Calibrated models enable safe, deterministic decision-making at specific probability cutoffs. In clinical workflows, a sepsis predictor might trigger a rapid response team at a 30% risk threshold. Without calibration, this threshold becomes arbitrary—a miscalibrated model might output 30% when the true risk is 60%, leading to missed interventions. Calibration ensures that operational thresholds correspond to consistent, predictable clinical actions.

A post-hoc calibration method that fits a logistic regression model on top of a classifier's raw output scores. It learns two parameters (A and B) to transform uncalibrated scores into calibrated probabilities using the formula: P(y=1|x) = 1 / (1 + exp(A * f(x) + B)). Platt scaling works best when the original score distribution is sigmoidal and is commonly applied to Support Vector Machines and older neural network architectures.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing function to map raw scores to calibrated probabilities. Unlike Platt scaling, isotonic regression makes no assumptions about the shape of the calibration curve, making it more flexible for complex distortions. However, it requires more data to avoid overfitting and can introduce discontinuities. It is particularly effective for calibrating random forests and boosted trees.

A single-parameter extension of Platt scaling designed for deep neural networks. It divides all logits by a learned temperature parameter T before applying the softmax function. When T > 1, the output distribution is softened, reducing overconfidence. Critically, temperature scaling preserves the rank order of predictions (accuracy is unchanged), making it ideal for calibrating modern architectures like ResNets and transformers without degrading classification performance.

A visual diagnostic tool that plots predicted probability against observed frequency. Predictions are binned (e.g., 0-10%, 10-20%), and the true positive rate is computed for each bin. A perfectly calibrated model traces the identity line (y=x). Deviations above the line indicate underconfidence (predicted risk is lower than actual), while deviations below indicate overconfidence (predicted risk is higher than actual). These diagrams are essential for detecting calibration drift in production.

The silent degradation of a model's probabilistic accuracy over time due to shifting patient populations, evolving clinical practices, or changes in data acquisition protocols. A model that was perfectly calibrated at deployment may become systematically overconfident or underconfident as the real-world distribution diverges from the training set.

• Covariate shift: Input feature distributions change (e.g., new imaging equipment)
• Prior probability shift: Base rates of conditions evolve (e.g., post-pandemic disease prevalence)
• Detection window: Often identified via periodic Hosmer-Lemeshow goodness-of-fit tests on recent production data

Unchecked calibration drift in a sepsis predictor can lead to either alarm fatigue or missed early warnings, both of which directly compromise patient safety.

The primary visual diagnostic tool for assessing calibration quality. A reliability diagram plots the predicted probability against the observed frequency of positive outcomes across discretized bins. A perfectly calibrated model traces the identity line (y = x).

• Under the diagonal: Model is overconfident (predicts higher probabilities than observed)
• Above the diagonal: Model is underconfident (predicts lower probabilities than observed)
• Bin size trade-off: Too few bins obscure miscalibration; too many introduce noise

For clinical prediction rules estimating 10-year cardiovascular risk, a reliability diagram can reveal whether the model systematically underestimates risk in specific demographic subgroups, flagging potential bias before harm occurs.

A scalar summary statistic that quantifies the weighted average absolute difference between predicted confidence and empirical accuracy across all probability bins. ECE collapses the reliability diagram into a single actionable number.

• Calculation: ECE = Σ (|B_m| / n) × |acc(B_m) − conf(B_m)|, where B_m is the m-th bin
• Interpretation: Lower ECE indicates better calibration; an ECE of 0.05 means predictions are off by 5 percentage points on average
• Limitation: ECE is sensitive to binning strategy and can mask within-bin miscalibration patterns

Modern calibration techniques like Platt scaling and isotonic regression directly optimize to minimize ECE on a held-out validation set, making it the de facto loss function for recalibration pipelines.

A parametric recalibration method that fits a logistic regression model on top of a classifier's raw output scores to produce well-calibrated probabilities. Named after John Platt, who introduced it for Support Vector Machines in 1999.

• Mechanism: Learns two scalar parameters (A, B) such that P(y=1|x) = 1 / (1 + exp(A × f(x) + B))
• Strengths: Simple, computationally cheap, and effective when miscalibration is sigmoidal in shape
• Weaknesses: Assumes a monotonic relationship between scores and probabilities; fails when the distortion is non-monotonic

Platt scaling is the default calibration method in many production ML pipelines because it requires minimal data and generalizes well, though isotonic regression often outperforms it when sufficient calibration data is available.

A strictly proper scoring rule that jointly evaluates both discrimination and calibration by measuring the mean squared error between predicted probabilities and actual binary outcomes. Unlike ECE, the Brier score penalizes both overconfidence and poor separation.

• Formula: BS = (1/N) × Σ (p_i − y_i)², where p_i is the predicted probability and y_i is the ground truth
• Range: 0 (perfect) to 0.25 (worst for balanced classes); a naive baseline predicting 0.5 always scores 0.25
• Decomposition: Brier score = Calibration loss + Refinement loss − Uncertainty

The Brier score is preferred over accuracy for evaluating clinical prediction rules because it harshly penalizes confident wrong answers—a model predicting 99% certainty on a missed sepsis case incurs a massive penalty, aligning incentives with patient safety.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing mapping from raw model scores to calibrated probabilities. Unlike Platt scaling, isotonic regression makes no assumptions about the functional form of the miscalibration.

• Algorithm: Pool Adjacent Violators Algorithm (PAVA) finds the monotonic function that minimizes squared error
• Advantage: Can correct arbitrary non-monotonic distortions that parametric methods miss
• Risk: Prone to overfitting on small calibration sets, producing step functions that don't generalize

In practice, isotonic regression is the preferred method when abundant calibration data exists, while Platt scaling is safer for small samples. Many clinical ML pipelines use a hybrid approach, selecting between them via cross-validation on a dedicated calibration set.