Saliency Maps

A technique for producing visual explanations from convolutional neural networks by using the gradients of a target concept flowing into the final convolutional layer. Unlike basic saliency maps, Grad-CAM generates a coarse localization map that highlights the discriminative regions used by the model for a specific class prediction.

• Key Advantage: Class-specific and does not require architectural changes or re-training.
• Diagnostic Use Case: Validating that a pneumonia classifier is focusing on lung opacities rather than laterality markers or imaging artifacts.
• Mechanism: Computes the gradient of the class score with respect to feature map activations, then globally average pools these gradients to obtain neuron importance weights.

An axiomatic attribution method that assigns importance scores to input features by integrating the gradients of the model's output along a straight-line path from a baseline input to the actual input. It satisfies key theoretical properties including Sensitivity and Implementation Invariance that simpler gradient-based saliency maps may violate.

• Baseline Selection: A critical design choice; a black image or blurred version of the input is common for vision tasks.
• Formula: Attributions are computed as the path integral of gradients, approximated via a Riemann sum over m steps.
• Regulatory Relevance: Preferred in FDA submissions because its axiomatic guarantees provide a defensible mathematical foundation for feature importance claims.

A game-theoretic approach to explain the output of any machine learning model by computing the Shapley value of each feature—its marginal contribution to the prediction averaged over all possible feature coalitions. SHAP unifies several explanation methods under a single theoretical framework.

• KernelSHAP: A model-agnostic, local explanation method using weighted linear regression on perturbed inputs.
• DeepSHAP: An efficient approximation for deep networks that combines SHAP with DeepLIFT's rules.
• Diagnostic Context: Used to identify which biomarkers (e.g., specific gene expressions, lab values) most influence a patient's risk score, enabling clinician review of model reasoning.

A technique for explaining neural network decisions by decomposing the output prediction and redistributing relevance scores backward through the network's layers using conservation rules. LRP ensures that the total relevance is preserved from the output layer down to the input pixels.

• Propagation Rules: Variants like LRP-αβ and LRP-ε handle different layer types and control the sparseness of explanations.
• Strength: Produces fine-grained, pixel-level heatmaps with strong theoretical grounding in the Deep Taylor Decomposition framework.
• Comparison to Saliency Maps: LRP provides a more complete decomposition of the prediction signal, whereas raw saliency maps reflect only local gradient sensitivity.

A method that describes how to minimally change the input features of an instance to alter its prediction to a predefined target outcome. Instead of highlighting influential pixels, counterfactuals answer the question: 'What would need to be different for the model to change its diagnosis?'

• Clinical Relevance: Provides actionable guidance—e.g., 'If this patient's hemoglobin A1c were reduced by 1.2%, the model would no longer flag them as high-risk for diabetic retinopathy.'
• Sparsity Constraints: Algorithms like Wachter's method or DiCE optimize for realistic, sparse changes that respect causal constraints.
• Complementarity: Used alongside saliency maps to provide both attribution (why this prediction?) and recourse (how to change it?).

Quantitative measures that assess how accurately an explanation method reflects the true reasoning process of the underlying machine learning model. These metrics are essential for regulatory submissions to demonstrate that saliency maps are not merely visually plausible but genuinely capture model behavior.

• Comprehensiveness: Measures the drop in prediction probability when the top-k attributed features are removed.
• Sufficiency: Measures whether the top-k attributed features alone can sustain the original prediction.
• Monotonicity: Evaluates whether adding features in order of attribution rank produces a monotonically increasing prediction score.
• FDA Context: Faithfulness metrics provide the quantitative evidence required in a Predetermined Change Control Plan (PCCP) to validate that explanation methods remain reliable after model updates.