Post-hoc Explainability

Post-hoc methods operate independently of the model's internal architecture. They treat the model as a black box, requiring only access to inputs and outputs. This enables a single explanation framework—such as LIME or SHAP—to be applied across CNNs, vision transformers, and ensemble models without modification. In regulated clinical environments, this decoupling allows the diagnostic model to be updated or replaced without revalidating the entire explanation pipeline.

The defining operational advantage: explanations are generated after training is complete. There is no need to modify the loss function, alter the architecture, or access the original training dataset. This is critical for FDA-cleared SaMD, where any retraining triggers a new regulatory submission. Techniques like Grad-CAM and Integrated Gradients can be applied to frozen, validated models in production, preserving the costly clinical validation status.

Post-hoc methods typically provide local explanations—they explain a single prediction, not the model's overall logic. A saliency map shows why a specific chest X-ray was classified as pneumothorax, but does not reveal the model's general concept of the disease. This distinction is vital for clinician-in-the-loop workflows: the explanation supports a specific differential diagnosis but cannot substitute for a global validation study.

A plausible-looking heatmap does not guarantee a faithful explanation. Research shows that some saliency methods produce visually convincing outputs that are insensitive to both the model's parameters and the input data. This interpretability illusion poses a direct patient safety risk in radiology. Mitigation requires quantitative evaluation using metrics like faithfulness score and tools like Quantus to verify that highlighted regions genuinely drive the model's decision.

Generating explanations adds latency to the prediction pipeline. Computing SHAP values requires multiple model evaluations per input, while gradient-based methods like Integrated Gradients need dozens of forward-backward passes along an interpolation path. For real-time diagnostic applications—such as endoscopic video analysis—this overhead must be carefully managed through optimized libraries like Captum or by pre-computing baselines.

Post-hoc explanations are the primary artifact for SaMD audit trails under FDA and EU MDR requirements. Every diagnostic prediction must be accompanied by a stored, reproducible explanation that demonstrates the model relied on clinically relevant anatomy—not confounding artifacts like scanner markers or text overlays. This lesion attribution verification is the cornerstone of regulatory explainability, enabling post-market surveillance and forensic review of adverse events.