Faithfulness Score

Local fidelity measures how precisely an explanation approximates the complex model's behavior in the immediate vicinity of a specific input instance. It is the foundational property of a faithful explanation.

• A high-fidelity explanation acts as a local surrogate model, accurately predicting how the main model would behave if the input were slightly perturbed.
• This is distinct from global interpretability; a method can be highly faithful locally without explaining the model's overall logic.
• Techniques like LIME explicitly optimize for local fidelity by fitting a simple interpretable model (e.g., linear regression) to the complex model's predictions on perturbed samples near the instance.

The infidelity metric is a formal, perturbation-based measure that quantifies the degree to which an explanation fails to reflect the model's output. It is defined as the expected squared error between the explanation's prediction of importance and the actual change in the model's output.

• Mathematically, for an input x, model f, explanation Φ, and a meaningful perturbation I, infidelity is: E_I[(I^T Φ(f, x) - (f(x) - f(x - I)))^2].
• A low infidelity score indicates high faithfulness. The metric directly tests if the feature importance scores (Φ) predict the model's output drop when those features are removed or altered (I).
• It requires defining a relevant perturbation distribution (e.g., blurring image regions, masking tokens) that simulates 'removing' the explained feature.

These are complementary metrics that evaluate if an explanation identifies the minimal sufficient features for a prediction.

• Sufficiency measures whether the subset of top-K features identified by the explanation is, by itself, sufficient for the model to make its original prediction with high confidence. Formally, it checks if f(x_S) ≈ f(x), where x_S is the input containing only the top-K explained features.
• Comprehensiveness (or completeness) measures whether the features not highlighted by the explanation are unimportant. It evaluates the drop in the model's prediction score when the top-K explained features are removed, leaving only the supposedly unimportant ones. A large drop indicates the explanation successfully captured the critical features.
• A faithful explanation should have high sufficiency and high comprehensiveness, demonstrating it captured all and only the important features.

Explanation robustness (or stability) assesses the consistency of an explanation method when the input or model undergoes minor, semantically-preserving perturbations. A faithful explanation should not change drastically for functionally equivalent inputs.

• Input Robustness: For two inputs x and x' that are semantically similar (e.g., a rephrased sentence, a slightly rotated image), the explanations Φ(f, x) and Φ(f, x') should be similar.
• Model Robustness: For two models f and f' that achieve similar predictive performance, the explanations for the same input should be comparable.
• Low robustness can indicate the explanation method is sensitive to noise or artifacts rather than the model's true decision logic. Metrics like Top-K Intersection or Rank Correlation between explanation vectors for perturbed pairs are used to measure this.

The model randomization test is a critical sanity check to determine if an explanation method is actually sensitive to the model's learned parameters, or if it produces similar results based solely on the model's architecture.

• Procedure: Generate explanations using the trained model. Then, progressively randomize the model's parameters (starting from the output layer backwards) and generate explanations again.
• Faithful methods should produce significantly different explanations for the randomized model versus the trained model. If the explanations remain similar, the method may be relying on architectural artifacts or input statistics, not the learned function.
• This test helps filter out explanation methods that appear plausible but are not actually faithful to the specific model being explained.

Contrastive faithfulness evaluates an explanation's ability to answer 'why P rather than Q?'—a natural form of human reasoning. A faithful contrastive explanation should highlight the features most responsible for the model choosing prediction P over a specific alternative Q.

• This goes beyond standard feature attribution, which answers 'why P?'. It requires the explanation to identify features that differentiate the actual input from a counterfactual input that would lead to outcome Q.
• Evaluation: Measure if perturbing the features highlighted as contrastively important causes the model's prediction to flip from P to Q. A faithful contrastive explanation will have high precision for inducing this flip.
• This characteristic is crucial for applications like loan denials or medical diagnoses, where understanding the difference between outcomes is as important as understanding the chosen outcome.

Infidelity is a quantitative metric that directly measures the unfaithfulness of an explanation. It perturbs the input based on the explanation's importance scores and measures the difference between the model's actual output change and the change predicted by the explanation.

• Mechanism: Given an input x, explanation Φ(x), and a meaningful perturbation I, infidelity is defined as 𝔼_I[(I^T Φ(x) - (f(x) - f(x - I)))^2].
• Interpretation: A low infidelity score indicates the explanation accurately reflects how the model's output changes when important features are altered, signifying high faithfulness.
• Key Distinction: While faithfulness is the property, infidelity is a specific, implementable metric to measure it.

Sufficiency and Comprehensiveness are complementary metrics that evaluate faithfulness by measuring prediction change when using only the top-K important features.

• Sufficiency: Measures if the top-K features identified by an explanation are sufficient to produce a prediction close to the original. A faithful explanation's top features should yield a similar output.
• Comprehensiveness: Measures the drop in model prediction confidence when the top-K features are removed. A faithful explanation should identify features whose removal causes a large prediction change.
• Application: Used together, they test if the explanation correctly identifies a minimal, yet complete, set of causal features.

Perturbation Analysis is a foundational validation technique for assessing explanation faithfulness. It involves systematically modifying the input and observing the correlation between the explanation's importance scores and the resulting changes in the model's output.

• Core Principle: If an explanation is faithful, perturbing a high-importance feature should cause a large change in the model's prediction, while perturbing a low-importance feature should cause minimal change.
• Methods: Includes Occlusion Sensitivity (for images/text), feature ablation, or adding noise.
• Output: The correlation (e.g., rank correlation) between explanation scores and prediction deltas serves as a direct faithfulness metric.

Local Fidelity is the property that a post-hoc explanation model (e.g., a linear surrogate from LIME) accurately approximates the complex model's behavior in the local neighborhood of a specific prediction.

• Scope: It is a necessary condition for faithfulness. An explanation cannot be faithful to the model's reasoning if it is not a locally accurate approximation.
• Measurement: Typically evaluated by generating perturbed samples around the instance, having both the complex model and the explanation model make predictions on them, and measuring agreement (e.g., R² score, accuracy).
• Distinction from Faithfulness: Local fidelity ensures the explanation matches the model's input-output function. Faithfulness ensures it matches the model's internal reasoning or causal factors for that function.

The Randomization Test is a critical sanity check to determine if a feature attribution method is truly explaining the model's learned logic, rather than just reflecting the data or model architecture.

• Procedure: 1. Generate explanations for a trained model. 2. Randomize the model's parameters layer-by-layer (destroying learned knowledge). 3. Generate explanations for the randomized model.
• Faithfulness Indicator: A valid, faithful explanation method should produce significantly different attribution maps for the trained vs. randomized model. If the explanations remain similar, the method is not sensitive to the model's actual reasoning and fails the sanity check.
• Purpose: This test validates that the explanation method has model sensitivity, a prerequisite for producing faithful explanations.

Explanation Robustness refers to the stability and consistency of an explanation method when the input undergoes small, semantically-preserving perturbations (e.g., adding minor image noise, paraphrasing text).

• Relationship to Faithfulness: While distinct properties, they are often correlated. A non-robust explanation (highly variable for similar inputs) is unlikely to be faithfully capturing a stable model reasoning process.
• Measurement: Quantified by a Stability Score, which measures the similarity (e.g., Spearman correlation, Top-K intersection) between explanations for an original input and its perturbed versions.
• Engineering Implication: For a faithfulness score to be reliable in production, the underlying explanation method must itself be robust to minor input variations.