Model Calibration

The defining property of a calibrated model is the direct correspondence between predicted confidence and observed frequency. If a model assigns a probability of 0.8 to a set of lesions, exactly 80% of those lesions should be malignant.

• Reliability Diagram: A plot of predicted probability vs. observed frequency; a perfectly calibrated model lies on the diagonal.
• Expected Calibration Error (ECE): A scalar summary metric that measures the weighted average gap between confidence and accuracy across bins.
• Clinical Impact: Prevents overconfident false negatives that could delay critical biopsy decisions.

Calibration is typically achieved by learning a small auxiliary model on a held-out validation set to map raw logits to corrected probabilities without altering the original diagnostic model's feature extraction or segmentation capabilities.

• Platt Scaling: Fits a logistic regression model to the raw output scores, effectively learning a sigmoid transformation.
• Isotonic Regression: Learns a non-parametric, monotonically increasing step function, useful when the distortion is non-sigmoidal.
• Temperature Scaling: A single-parameter variant of Platt scaling that divides all logits by a learned scalar T > 1 to soften overconfident predictions.

The Brier Score is a strictly proper scoring rule that jointly measures discrimination and calibration. It decomposes into refinement (sharpness) and calibration loss.

• Formula: The mean squared error between predicted probability and binary outcome (0 or 1).
• Decomposition: Brier = Calibration Loss + Refinement Loss - Uncertainty.
• Advantage over ECE: Provides a continuous, differentiable measure sensitive to the entire probability distribution, not just binned averages.

Mammography models often output probabilities across multiple BI-RADS categories (e.g., 2, 3, 4A, 4B, 4C, 5). Calibration must extend beyond binary outcomes to ensure each category's predicted probability is accurate.

• Classwise ECE: Computes calibration error independently for each BI-RADS category.
• Confidence Calibration: Ensures the model's maximum predicted probability matches the likelihood of that class being correct.
• Distributional Calibration: Requires that the full predicted probability vector matches the empirical distribution of outcomes across all categories simultaneously.

A model calibrated on a specific vendor's FFDM system may become miscalibrated when deployed on a different population or DBT scanner. Continuous monitoring is essential.

• Covariate Shift: Changes in the input distribution (e.g., new compression paddle, different dose levels) distort logit distributions.
• Prior Probability Shift: A screening population with a different cancer prevalence rate invalidates the base rate assumptions of the calibrator.
• Recalibration Strategy: Periodic re-fitting of a temperature scalar on a small, local validation set without retraining the entire detection backbone.

These are orthogonal model properties. A model with high AUC (strong discrimination) can be severely miscalibrated, producing clinically misleading confidence scores.

• Discrimination (AUC): Measures how well the model separates malignant from benign lesions, independent of probability magnitude.
• Calibration: Measures the accuracy of the probability magnitude itself.
• Clinical Necessity: A high-AUC model that reports 99% confidence on a 50% true-positive rate is dangerous; calibration corrects this without sacrificing ranking ability.

A graphical tool for diagnosing calibration errors. It plots predicted probability against observed frequency.

• A perfectly calibrated model follows the diagonal identity line.
• A curve below the diagonal indicates overconfidence.
• A curve above the diagonal indicates underconfidence.
• The gap between the curve and the diagonal is the calibration error.

The primary scalar metric for calibration. It computes the weighted average of the absolute difference between accuracy and confidence across probability bins.

• Bins predictions into intervals (e.g., 0-10%, 10-20%).
• For each bin, calculates |accuracy - confidence|.
• A lower ECE indicates better global calibration.
• Does not penalize models that are uniformly uncertain.

A post-hoc calibration method that fits a logistic regression model to the raw output scores (logits) of a classifier.

• Learns scalar parameters A and B to optimize the log-likelihood on a held-out validation set.
• Transforms unbounded logits into well-calibrated probabilities.
• Most effective when the uncalibrated model's scores are sigmoid-shaped.
• Non-parametric alternatives like Isotonic Regression offer more flexibility.

A simple, highly effective calibration technique for neural networks. It divides all logits by a single scalar parameter T (temperature) before the softmax function.

• T > 1 'softens' the probabilities, reducing overconfidence.
• T < 1 'sharpens' the probabilities, increasing confidence.
• The optimal T is learned by minimizing the negative log-likelihood on a validation set.
• Unlike Platt scaling, it does not alter the rank order of predictions.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and the actual binary outcomes.

• Calculated as the average of (prediction - outcome)².
• A lower score indicates both better discrimination and calibration.
• Decomposes mathematically into refinement loss and calibration loss.
• More sensitive to both calibration and resolution than ECE alone.

The clinical application of calibration where a model's probability score directly informs a risk-based action.

• A well-calibrated 10% malignancy risk means exactly 1 in 10 such patients have cancer.
• Enables reliable worklist prioritization based on calibrated urgency.
• Allows setting a threshold for BI-RADS 3 (short-interval follow-up) versus BI-RADS 4 (biopsy).
• Poor calibration can lead to systematic over- or under-treatment.