Interpretability Illusion

The central tension driving the illusion. A plausible explanation aligns with human intuition or domain knowledge—like highlighting a tumor in a lung scan. A faithful explanation accurately reflects the model's true decision boundary.

• The Illusion: Grad-CAM heatmaps often look anatomically correct, but an attribution attack can make the model highlight a benign region while still classifying correctly.
• Key Distinction: Plausibility is a subjective, human judgment. Faithfulness is an objective, measurable property verified through input perturbation.
• Clinical Risk: A radiologist may trust a plausible but unfaithful map, leading to automation bias and missed diagnoses.

A classic example of the illusion in medical imaging. A model trained to detect pneumonia may appear to use lung tissue for its prediction, but a faithfulness evaluation reveals it relies on laterality markers or scanner-specific metadata.

• Mechanism: The model exploits spurious correlations in the training data, such as portable X-rays (taken on sicker patients) having a specific metal token.
• Saliency Failure: Standard saliency maps will highlight the lung field because it correlates with the token, not because the model understands pathology.
• Mitigation: Use domain-specific saliency constraints and adversarial ablation studies to expose the true driver.

A deliberate manipulation proving the fragility of post-hoc explanations. An adversarial perturbation is crafted to change the saliency map to a target explanation while keeping the model's classification unchanged.

• The Attack: An attacker can make a correctly classified malignant lesion appear benign on the heatmap, destroying clinician trust.
• Implication: If an explanation can be arbitrarily manipulated without changing the output, it is not causally linked to the decision process.
• Defense: Requires explanation regularization during training and monitoring for adversarial robustness in the explanation pipeline.

The illusion is particularly dangerous because it exploits the human tendency toward confirmation bias. A clinician expects a diagnostic model to look at the lesion; when the saliency map confirms this, trust is inflated.

• The Trap: The explanation is evaluated based on whether it matches the human's pre-existing belief, not on its mathematical fidelity.
• Trust Calibration Failure: This leads to over-reliance on the AI system when it agrees with the clinician and under-reliance when it does not, negating the value of the assistive tool.
• Solution: Implement clinician-in-the-loop workflows that force review of the explanation's uncertainty, not just its visual appeal.

Many explanation methods are validated using sensitivity analysis, which measures how much the explanation changes with a small input perturbation. This is not the same as faithfulness.

• Sensitivity: A low-sensitivity map is smooth and visually pleasing, but a constant, uninformative map has perfect sensitivity.
• Faithfulness: Measured by Quantus metrics like Pixel Flipping or ROAR (Remove And Retrain). If removing the 'important' pixels does not degrade model confidence, the explanation was an illusion.
• Best Practice: Always pair visual inspection with a quantitative faithfulness score.

Many attribution methods, including Integrated Gradients and DeepLIFT, require a baseline input representing the 'absence' of signal. The choice of this baseline dramatically shapes the resulting explanation.

• The Illusion: Using a black image as a baseline for a chest X-ray produces a saliency map highlighting the entire lung field, not the pathology. This is an artifact of the baseline, not the model's reasoning.
• Medical Context: A blurred or 'gaussian noise' baseline may be more appropriate but introduces its own statistical artifacts.
• Regulatory Impact: An SaMD audit trail must record the exact baseline used, as changing it can produce a completely different explanation for the same prediction.

A quantitative metric that evaluates the accuracy of an explanation by measuring how well the attributed importance scores correlate with the actual change in model output when the corresponding features are perturbed or removed. A low faithfulness score is the primary diagnostic tool for detecting an interpretability illusion. If removing the pixels a saliency map highlighted as 'important' does not significantly change the model's prediction, the explanation is unfaithful and misleading. Common perturbation strategies include:

• Most Relevant First (MoRF): Removing pixels in descending order of attributed importance.
• Least Relevant First (LeRF): Removing pixels in ascending order of importance as a control.
• ROAR (RemOve And Retrain): Retraining the model on perturbed datasets to measure true reliance on features.

A malicious manipulation of an input image designed to cause a model to produce a specific, incorrect explanation or saliency map while maintaining its original, correct classification. This directly exploits the interpretability illusion by creating a veneer of plausible reasoning that is completely detached from the model's true decision process. For a medical imaging system, an attacker could subtly perturb a scan so that a correct 'malignant' diagnosis is accompanied by a saliency map highlighting healthy background tissue instead of the actual tumor, destroying clinical trust and auditability.

A scientific experiment to understand a model's behavior by systematically removing or disabling specific components—such as layers, neurons, or input features—and measuring the resulting impact on performance. In the context of explainability, feature ablation is a ground-truth check against the interpretability illusion. If a saliency map claims a specific anatomical region is critical, surgically removing that region from the input should cause a predictable and significant drop in the model's confidence. If it does not, the explanation is an illusion.

The approach of applying an interpretation method to a trained machine learning model after training is complete, without requiring any modification to the model's original architecture or process. This is the category under which most saliency map techniques fall, and it is inherently susceptible to the interpretability illusion. Because post-hoc methods approximate the model's reasoning rather than revealing its native computation, they can generate visually convincing but functionally unfaithful explanations. The alternative is ante-hoc or inherently interpretable architectures like Concept Bottleneck Models.

The process of aligning a clinician's subjective trust in an AI diagnostic tool with the tool's objective performance and reliability. The interpretability illusion directly sabotages trust calibration by artificially inflating a user's confidence through plausible-looking but incorrect explanations. A radiologist who sees a crisp, anatomically coherent saliency map may over-rely on the AI, even if the map is unfaithful. Proper trust calibration requires:

• Displaying uncertainty quantification alongside explanations.
• Training users to recognize the limits of post-hoc saliency maps.
• Validating explanations with faithfulness metrics before human review.