Explanation Robustness

Faithfulness (also called fidelity) measures how accurately an explanation reflects the true causal factors or reasoning process of the underlying model for a specific prediction. A faithful explanation correctly identifies which input features the model actually used to make its decision.

• Core Principle: The explanation should be a truthful account of the model's internal mechanism, not a plausible-sounding but incorrect story.
• Quantification: Often measured via Perturbation Analysis, where features deemed important by the explanation are removed or altered, and the resulting change in the model's output is observed. A large change indicates high faithfulness.
• Example: For an image classifier predicting 'dog', a faithful saliency map would highlight the dog's features (ears, snout). An unfaithful one might highlight background grass because it is statistically correlated with dog images in the training set.

Stability (or consistency) assesses whether an explanation method produces similar attributions for similar inputs. A robust explanation should not change dramatically for two inputs that are semantically equivalent or very close in the input space.

• Why it Matters: Unstable explanations are unreliable for debugging or trust. If adding negligible noise to an image completely changes the highlighted pixels, the explanation is not actionable.
• Measurement: The Stability Score quantifies the variance in explanation outputs (e.g., the cosine similarity between attribution vectors) for a set of perturbed versions of the same base input.
• Contrast with Sensitivity: Stability relates to the explanation method's output, while sensitivity often refers to the model's prediction. A model can be prediction-stable but have unstable explanations, which is a failure of the explanation method.

Completeness evaluates whether an explanation accounts for the totality of factors that contributed to a model's prediction. A complete explanation's importance scores should sum to the difference between the model's actual output and its output on a baseline reference (e.g., a zero vector or average input).

• Theoretical Basis: Methods like SHAP and Integrated Gradients are designed to satisfy this completeness property (also called the summation-to-delta property).
• Implication: It ensures no significant contributing factor is omitted. An explanation highlighting only a single feature when ten were influential is incomplete.
• Connection to Faithfulness: A complete explanation is not necessarily faithful (it could assign importance incorrectly), but a faithful explanation should strive to be complete for the features the model actually used.

Explanation Sparsity refers to the number of features identified as important. Simulatability measures how easily a human can use the explanation to predict the model's output.

• Sparsity: Human cognitive limits favor sparse explanations that highlight a few critical features rather than hundreds. However, excessive sparsity can violate completeness. The ideal is a minimal sufficient feature set.
• Simulatability: This is an extrinsic, human-centric evaluation metric. If an explanation is highly simulatable, a person given the explanation and the input can accurately guess the model's prediction. This requires the explanation to be both correct (faithful) and comprehensible.
• Trade-off: There is often a tension between completeness (including all factors) and simulatability (presenting a simple, digestible reason). Robust explanation methods aim to optimize this balance.

Contrastive Explanations answer the question "Why prediction P rather than a specific alternative Q?" This aligns with natural human reasoning, where we often explain an outcome by contrasting it with a plausible counterfactual.

• Mechanism: Instead of attributing importance to features for the actual prediction, contrastive methods identify the features whose values are most responsible for the model choosing P over Q.
• Robustness Aspect: A robust contrastive explanation should remain valid for minor perturbations that do not change the model's preference for P over Q. It focuses on the decision boundary between classes.
• Use Case: Highly valuable in high-stakes domains like finance ("Why was this loan rejected rather than approved?") or healthcare ("Why diagnosis A rather than B?").

Infidelity and Sensitivity are two key quantitative metrics used to measure the absence of robustness in explanations.

• Infidelity: Formally quantifies the expected error between the explanation's importance scores and how the model's output actually changes when the input is perturbed. Low infidelity is desired and correlates with high faithfulness.
• Sensitivity (Explanation Sensitivity): Measures the maximum change in the explanation under infinitesimal input perturbations. Low sensitivity is desired and correlates with high stability. It checks if an explanation is locally Lipschitz continuous.
• Application: These are not properties but evaluation scores. They are used in automated testing pipelines to flag unreliable explanations before they are presented to end-users or auditors.

A stability score is a quantitative metric that measures the consistency of explanations generated for similar inputs or under small, semantically-preserving perturbations. It directly quantifies the robustness of an explanation method.

• Purpose: To ensure an explanation method is not overly sensitive to noise or minor input variations that should not change the model's reasoning.
• Calculation: Often involves measuring the variance or distance (e.g., L2 norm, rank correlation) between attribution vectors for a set of perturbed versions of the same input.
• High Stability indicates the explanation method reliably identifies the same core features for a given prediction, a key requirement for trustworthy model auditing.

Infidelity is an explanation metric that quantifies the degree to which an explanation fails to accurately reflect the model's output when the input is perturbed according to the explanation's own importance scores. It is a direct measure of explanation faithfulness.

• Core Idea: A faithful explanation should predict how the model's output changes when important features are altered. High infidelity means the explanation is a poor predictor of model behavior.
• Calculation: Involves generating perturbations (e.g., blurring, noise) weighted by the explanation's importance scores and measuring the difference between the predicted and actual change in the model's output.
• Low Infidelity is desired, indicating the explanation correctly models the local causal effect of features on the prediction.

Sensitivity analysis in explainability is a validation technique that systematically evaluates how small changes in input features affect both the model's prediction and the generated explanation. It is a foundational method for probing robustness.

• Process: Involves applying controlled perturbations (e.g., adding Gaussian noise, masking features) and observing the resulting variance in prediction scores and explanation maps.
• Two Key Axes: Analyzes prediction sensitivity (model output stability) and explanation sensitivity (attribution map stability). A robust system shows low explanation sensitivity when prediction sensitivity is also low.
• Use Case: Essential for stress-testing explanations in adversarial or noisy real-world environments where inputs are never perfect.

The randomization test, or model randomization test, is a critical sanity check for feature attribution methods. It verifies if an explanation method produces meaningfully different results when applied to a trained model versus a randomly initialized model with the same architecture.

• Rationale: A valid explanation method should attribute importance based on learned patterns, not model architecture. Explanations for a randomly initialized model (with no meaningful knowledge) should be qualitatively different and less structured.

• Procedure:

  1. Generate explanations for a set of inputs using the trained model.
  2. Randomize the model's weights (layer by layer) and generate explanations again.
  3. Compare the two sets of explanations (e.g., using correlation). A robust explanation method will show a significant drop in similarity after randomization.

Local fidelity is a property of a post-hoc explanation that measures how well the explanation approximates the behavior of the complex model in the immediate vicinity of a specific input instance. It is a prerequisite for explanation robustness.

• Definition: An explanation with high local fidelity acts as a locally accurate surrogate model. If you perturb the input slightly, the explanation's estimate of the prediction change should match the actual model's change.
• Connection to Robustness: An explanation cannot be robust if it lacks local fidelity, as its description of the model's local decision boundary would be incorrect. Techniques like LIME explicitly optimize for local fidelity by fitting a simple interpretable model around a prediction.
• Evaluation: Often measured by the accuracy of the surrogate model on perturbed samples near the instance of interest.

Perturbation analysis is a broad class of explanation validation techniques that systematically modifies or removes input features to observe the resulting changes in the model's output. It is a ground-truth mechanism for assessing explanation quality and robustness.

• Methods: Includes occlusion sensitivity (systematically blocking image regions), feature ablation (setting feature values to baseline), and counterfactual generation (finding minimal changes to flip a prediction).
• Role in Robustness: By comparing the actual impact of a perturbation (the change in model output) with the predicted impact (based on the explanation's importance scores), one can compute metrics like infidelity or faithfulness score.
• Gold Standard: Often considered a more reliable validation than gradient-based methods, as it directly probes the model's input-output function.