Calibration Drift

The model's predicted probabilities become systematically higher than the observed event frequency. A sepsis predictor that outputs an 85% risk score may only correspond to a 60% true incidence rate.

• Root Cause: Shifts in patient acuity or changes in clinical coding practices inflate the model's internal certainty without a corresponding increase in actual event rates.
• Clinical Impact: False reassurance leads to delayed intervention; clinicians learn to distrust the system's high-confidence alerts.

The model's predicted probabilities become systematically lower than the true event frequency. A readmission model predicting a 20% risk may face a 45% actual readmission rate.

• Root Cause: Introduction of a new, higher-risk patient subpopulation not represented in the training distribution.
• Clinical Impact: High-risk patients are not flagged for intensive care transitions, leading to preventable adverse outcomes and readmission penalties.

The primary quantitative metric for detecting calibration drift. ECE partitions predictions into M bins and computes the weighted absolute difference between average confidence and observed accuracy within each bin.

• Formula: ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|
• Monitoring Threshold: An ECE exceeding 0.05 typically triggers a recalibration review in high-stakes clinical deployments.
• Visualization: Reliability diagrams plot confidence against accuracy; perfect calibration follows the identity diagonal.

Calibration drift is frequently caused by covariate shift in the input feature distribution without a corresponding change in the decision boundary.

• Demographic Migration: A hospital catchment area experiences an influx of patients with different baseline risk profiles.
• Practice Pattern Evolution: New clinical guidelines alter testing frequency, changing the lab value distributions fed into the model.
• Instrumentation Changes: Recalibration of lab equipment or a new EHR vendor shifts the numerical ranges of key predictors.

A parametric post-hoc method that fits a logistic regression model on top of the original classifier's output scores to correct systematic miscalibration.

• Mechanism: Learns two scalar parameters (A, B) to transform raw logits: P_calibrated = 1 / (1 + exp(A * logit + B))
• Validation Requirement: Requires a held-out calibration dataset distinct from both training and test sets to avoid overfitting.
• Limitation: Assumes the miscalibration is a smooth, monotonic function; fails under complex, non-monotonic drift patterns.

A non-parametric recalibration technique that learns a stepwise monotonic function mapping uncalibrated scores to calibrated probabilities without assuming a sigmoidal shape.

• Mechanism: Minimizes a squared loss subject to a monotonicity constraint, producing a piecewise constant transformation.
• Advantage: Handles irregular, non-sigmoidal calibration errors that Platt scaling cannot correct.
• Risk: Prone to overfitting on small calibration sets; requires sufficient data in each probability region to produce stable estimates.

The phenomenon where the statistical properties of the target variable change over time, invalidating the original decision boundary. Unlike calibration drift, which affects probability estimates, concept drift alters the fundamental relationship between features and labels.

• Real-world example: A sepsis predictor trained pre-COVID sees the relationship between respiratory rate and sepsis fundamentally change during the pandemic.
• Detection method: ADWIN (Adaptive Windowing) or Page-Hinkley tests on prediction error rates.

A change in the distribution of input features between training and production environments. While the conditional probability P(Y|X) remains stable, the shift in P(X) can indirectly degrade calibration if the model was poorly regularized.

• Clinical example: A radiology model trained on images from one scanner manufacturer underperforms when deployed on a different vendor's equipment.
• Monitoring metric: Population Stability Index (PSI) or Kullback-Leibler divergence between training and production feature distributions.

The process of adjusting a model's output probabilities so they accurately reflect the true empirical frequency of events. A perfectly calibrated model predicting 10% risk should observe the event in exactly 10% of cases.

• Calibration methods: Platt scaling fits a logistic regression on raw scores; isotonic regression learns a non-parametric monotonic mapping.
• Evaluation metric: Expected Calibration Error (ECE) bins predictions and measures the weighted average gap between confidence and accuracy.

The primary metric for quantifying calibration drift. ECE partitions predictions into M equal-width bins, computes the absolute difference between average confidence and observed accuracy per bin, then weights by bin frequency.

• Interpretation: An ECE of 0.05 means the model's confidence diverges from accuracy by 5 percentage points on average.
• Limitation: ECE is sensitive to binning strategy; adaptive ECE uses equal-mass bins to address class imbalance in clinical datasets.

A method for evaluating the net benefit of a predictive model across a range of clinical threshold probabilities. It quantifies whether recalibration actually improves clinical decision-making, not just statistical metrics.

• Net benefit formula: (True Positives / N) - (False Positives / N) × (threshold / (1 - threshold))
• Clinical relevance: A recalibrated model may show improved ECE but identical net benefit if the drift occurred in probability regions where clinicians never act.

A single-parameter post-hoc calibration technique that divides logits by a learned temperature parameter T before applying softmax. T > 1 softens probabilities (reducing overconfidence); T < 1 sharpens them.

• Advantage: Does not alter rank ordering or classification decisions, preserving AUC while fixing calibration.
• Drift remediation: Temperature can be continuously updated on a held-out calibration set to track and correct slow calibration drift without full retraining.