Trust Calibration

The foundation of trust calibration is a model's ability to accurately report its own confidence in a prediction. This goes beyond a simple softmax probability. Techniques like Monte Carlo Dropout, Deep Ensembles, and conformal prediction produce statistically rigorous uncertainty estimates. A calibrated system maps these scores to actual likelihoods of correctness using reliability diagrams and Expected Calibration Error (ECE) . When a model reports 90% confidence, it should be correct exactly 90% of the time. Without this, clinicians cannot appropriately weight the AI's output against their own assessment.

A saliency map that looks plausible but does not reflect the model's true reasoning creates an interpretability illusion, which is a direct threat to trust calibration. Faithfulness scores quantitatively measure whether the highlighted features actually cause the prediction. Methods include:

• Perturbation-based testing: Removing attributed pixels and measuring output change
• Input marginalization: Replacing features with uninformative values
• Sensitivity-n analysis: Checking if similar inputs produce similar explanations A faithful explanation allows the clinician to verify that the model is looking at the pathological region of interest, not a confounding artifact like a chest drain or scanner annotation.

The human-machine interface is the critical bridge where calibration succeeds or fails. Effective interfaces present AI suggestions as second reads rather than primary diagnoses, avoiding anchoring bias. Key design principles include:

• Displaying confidence scores alongside saliency maps to contextualize the explanation
• Providing counterfactual explanations that show what minimal changes would flip the diagnosis
• Enabling disagreement workflows where clinicians can annotate and flag AI errors
• Avoiding automation bias by requiring active confirmation rather than passive acceptance Poorly designed interfaces can induce over-reliance even when the underlying model is well-calibrated.

Unconstrained attribution maps can highlight biologically impossible regions, eroding clinician trust. Domain-specific saliency incorporates prior anatomical knowledge to ensure explanations are physiologically plausible. Techniques include:

• Anatomical atlas-guided regularization during training
• Graph-based priors that enforce spatial coherence of attributed regions
• Organ segmentation masks that restrict attribution to relevant anatomical structures For example, a lung nodule detection model's explanation must not highlight the heart or ribs as primary evidence. Constraining explanations to known anatomical boundaries makes them clinically meaningful and directly supports appropriate trust calibration.

Trust calibration is not a one-time achievement but a continuous process. Data drift and concept drift can silently degrade model performance, causing the calibration established at deployment to become invalid. A robust system includes:

• Real-time monitoring dashboards tracking ECE and accuracy over time
• Automated alerts when calibration metrics exceed predefined thresholds
• Subgroup analysis to detect performance disparities across patient demographics, scanner types, or imaging protocols
• Feedback loops that capture clinician overrides and disagreements for periodic model recalibration Without ongoing monitoring, a perfectly calibrated system can drift into dangerous miscalibration without anyone noticing.

For Software as a Medical Device (SaMD) , trust calibration must be demonstrable to regulators. An SaMD audit trail captures the complete chain of evidence for every AI-assisted decision:

• The original input image and its metadata
• The model's raw output, confidence score, and uncertainty estimate
• The generated saliency map or explanation with its faithfulness score
• The clinician's final decision and any override rationale This audit trail supports post-market surveillance under FDA and MDR frameworks. It transforms trust calibration from a subjective feeling into an auditable, verifiable property of the system, essential for regulatory clearance and clinical adoption.

A quantitative metric that evaluates explanation accuracy by measuring how well attributed importance scores correlate with actual model output changes when corresponding features are perturbed or removed. This metric directly supports trust calibration by validating that saliency maps reflect genuine model reasoning.

• Measures correlation between attribution rank and output degradation
• Uses iterative perturbation or removal of top-attributed regions
• Exposes interpretability illusions where explanations look plausible but are unfaithful
• Essential for regulatory explainability audits

An explanation method that identifies the minimal change to an input instance's features required to alter the model's prediction to a predefined alternative outcome. In diagnostic contexts, counterfactuals answer 'what if' questions that help clinicians understand decision boundaries.

• Generates actionable 'nearest unlike' examples
• Helps prevent over-reliance by exposing model fragility
• Can show how a benign classification would shift to malignant
• Supports clinician-in-the-loop validation workflows

The specific application of feature attribution techniques to verify that a diagnostic model's decision is based on the actual pathological region of interest rather than irrelevant background artifacts or confounding variables. This is the core trust calibration checkpoint in radiology AI.

• Confirms model focuses on tumor boundaries, not scanner metadata
• Detects shortcut learning on spurious correlations
• Validated against radiologist-annotated ground truth regions
• Critical for MDR and FDA clinical validation studies