Infidelity

Infidelity quantifies the expected squared error between the model's true output change and the change predicted by a linear approximation based on the explanation's importance scores. Formally, for a model f, input x, explanation vector Φ(x), and a perturbation distribution I, it is defined as:

Infidelity(Φ, f, x) = E_I[(I^T Φ(x) - (f(x) - f(x - I)))^2]

• I is a random perturbation vector (e.g., masking or noise).
• Φ(x) is the feature attribution vector (e.g., from SHAP or Integrated Gradients).
• f(x) - f(x - I) is the model's actual output difference due to the perturbation.
• I^T Φ(x) is the explanation's linear prediction of that output difference. A lower infidelity score indicates a more faithful explanation.

The metric is fundamentally grounded in perturbation analysis. It validates an explanation by testing its predictive power under input changes.

• Method: Systematically apply random perturbations I to the input x and measure two things: 1) the actual change in the model's prediction, and 2) the change predicted by a linear model using the explanation's importance scores as coefficients.
• Key Insight: A faithful explanation should serve as a good local linear approximator of the complex model. If the importance scores in Φ(x) are correct, then the dot product I^T Φ(x) should closely match the true output difference f(x) - f(x - I) for many perturbations.
• Contrast with Completeness: While completeness checks if attributions sum to the total output, infidelity checks if they correctly predict output changes.

Infidelity is a direct, quantitative measure of local fidelity and is a primary method for calculating a faithfulness score.

• Local Fidelity: This property requires an explanation to accurately reflect the model's behavior in the neighborhood of a specific input. Infidelity operationalizes this by averaging over many local perturbations (I).
• Faithfulness Score: Often defined as 1 - Infidelity (after normalization) or simply the negative correlation between the explanation's prediction and the model's actual behavior. A high-faithfulness explanation has low infidelity.
• Distinction from Plausibility: An explanation can be plausible to a human (make sense) but have high infidelity if it does not match the model's true internal reasoning process. Infidelity targets faithfulness, not human agreement.

Infidelity is a model-agnostic evaluation metric. It can be applied to any explanation method (Φ) and any model (f), provided you can query the model's output.

• Black-Box Evaluation: It only requires the ability to perturb inputs and observe outputs. No internal model weights, gradients, or architecture knowledge is needed.
• Broad Applicability: It can evaluate explanations for deep neural networks, tree-based models (like XGBoost), and ensemble methods equally.
• Explanation Method Agnostic: It tests the output of any attribution method—including SHAP, LIME, Integrated Gradients, or simple gradient-based saliency—against the same ground truth: the model's perturbed behavior.

The computed infidelity score is highly sensitive to the choice of perturbation distribution I. This is both a key characteristic and a critical consideration.

• Distribution Choice: Common choices include:
  • Masking: Setting features to a baseline value (zero, mean, random).
  • Gaussian Noise: Adding small, normally distributed noise.
  • Blurring/Occlusion: For image data, applying Gaussian blur or gray patches.
• Semantic Meaning: Perturbations should be meaningful in the input domain. Adding Gaussian noise to a tabular feature might be valid, but applying it to a one-hot encoded category is not.
• Interpretation: Scores are only comparable when using the same perturbation family. A method may have low infidelity under one perturbation (e.g., masking) but high under another (e.g., noise), revealing aspects of its robustness.

Infidelity is a standard metric in explainability benchmarking suites and research libraries for quantitatively comparing explanation methods.

• Quantitative Benchmarking: In papers and toolkits, explanation methods are often ranked on datasets by their average infidelity scores across many inputs.
• Implementation Libraries: It is implemented in major XAI libraries:
  • Captum (PyTorch): infidelity function.
  • SHAP: Can be computed using the Explainer objects and perturbation.
  • TensorFlow Explainability: Can be implemented via custom loops.
• Production Monitoring: In MLOps pipelines, tracking infidelity scores over time can detect explanation drift, where an explanation method becomes less faithful as the underlying model or data distribution evolves.

Infidelity is used to benchmark and compare different explanation techniques like SHAP, LIME, and Integrated Gradients. By measuring how much each method's importance scores degrade under perturbation, engineers can identify which method produces the most faithful local approximations of the model. This is critical for selecting the right tool for model auditing.

• A low infidelity score indicates the explanation's feature weights accurately predict model output changes.
• High infidelity flags an unreliable explanation method that should not be trusted for critical decisions.

In regulated industries (finance, healthcare), demonstrating that an AI's decision-making process is understandable is often a legal requirement. Infidelity provides a quantitative, auditable metric to prove that the explanations provided to regulators or customers are not arbitrary. A model card can report infidelity scores to show that its feature attributions have been rigorously tested for faithfulness, supporting claims of transparency under frameworks like the EU AI Act.

Systematically high infidelity for certain input types can reveal model weaknesses. If an explanation claims certain features are important, but perturbing them doesn't change the output, it may indicate the model is relying on spurious correlations or is unstable in that region of the input space. This insight directs engineers to collect more representative training data or apply regularization techniques to improve model generalization and robustness.

In active learning or human-AI collaboration systems, infidelity helps prioritize which model predictions require expert review. A high infidelity score signals that the model's reasoning (as explained) is internally inconsistent, making the prediction less trustworthy. This allows human reviewers to focus their attention on the most uncertain or poorly explained cases, making the refinement process more efficient and targeted.

Infidelity is closely related to explanation robustness. A good explanation should be stable—similar inputs should yield similar explanations. By measuring infidelity across a set of semantically similar perturbations, practitioners can assess if an explanation method produces consistent results. Low and consistent infidelity builds user trust, as the explanation appears reliable and non-random, which is essential for user adoption of AI-assisted tools.

Infidelity can be calculated as a continuous validation check in production MLOps pipelines. As new model versions are deployed, their explanation infidelity scores can be monitored alongside traditional performance metrics. A significant drift in the average infidelity score can trigger alerts, indicating that the new model's decision logic may have changed in a way that makes existing explanation methods less faithful, prompting a re-evaluation of the interpretability stack.

A faithfulness score is a quantitative metric that measures how accurately an explanation reflects the true reasoning process or causal factors of the underlying model for a given prediction. It is a broader category under which infidelity falls.

• Core Concept: Evaluates if the explanation's highlighted features are genuinely important to the model's internal computation.
• Relationship to Infidelity: Infidelity is a specific, perturbation-based method for measuring faithfulness. A high infidelity score indicates low faithfulness.
• Example: If an explanation highlights a tumor in a medical scan, a faithful explanation means the model's prediction of 'malignant' changes significantly if that tumor region is altered.

Perturbation analysis is an explanation validation technique that systematically modifies or removes input features to observe the resulting changes in the model's output. Infidelity is a formal metric derived from this family of techniques.

• Methodology: Involves creating perturbed versions of an input (e.g., masking pixels, zeroing out tokens) based on an explanation's importance scores.
• Purpose: To empirically test the causal relationship between explained features and the model's prediction.
• Key Distinction: While perturbation analysis is the general approach, infidelity provides a specific, quantitative score summarizing the result of these perturbations.

Explanation robustness refers to the property of an explanation method to produce consistent and stable attributions for a given prediction when the input or model is subjected to minor, semantically-preserving perturbations.

• Focus: Assesses the reliability of the explanation method itself, not its faithfulness to the model.
• Contrast with Infidelity: Infidelity measures if an explanation is faithful to one specific model prediction. Robustness measures if the explanation method yields similar explanations for similar inputs.
• Example: A robust explanation method would highlight the same object in two slightly different photos of a scene. A method with low robustness might highlight different objects.

Sensitivity analysis in explainability evaluates how small changes in the input features affect both the model's prediction and the generated explanation. It encompasses the evaluation of both prediction stability and explanation stability.

• Dual Measurement: Tracks two outputs: 1) change in model prediction (related to infidelity), and 2) change in the explanation map.
• Broader Scope: Infidelity is specifically concerned with the first output—the change in prediction. Sensitivity analysis often studies the correlation or divergence between these two outputs.
• Use Case: Helps determine if an explanation method is sensitive to noise, which can indicate instability or overfitting to insignificant features.

Sufficiency is an explanation metric that measures whether the subset of features identified as most important by an explanation is, by itself, sufficient for the model to make its original prediction.

• Test Method: The top-k most important features (per the explanation) are isolated, and the model is run on this ablated input.
• Interpretation: A high sufficiency score means the model's prediction remains unchanged using only the explained features, suggesting the explanation captured the critical factors.
• Complement to Infidelity: While infidelity measures the impact of removing important features, sufficiency measures the predictive power of keeping only those features. A good explanation should score well on both.

A completeness score is a metric that evaluates whether an explanation accounts for all features or factors that contributed significantly to a model's prediction. It ensures the explanation is not missing critical components.

• Mathematical Basis: Often defined as the sum of the attribution scores for all features. For methods like SHAP, this sum equals the difference between the model's prediction and its baseline expectation.
• Relationship to Infidelity: An explanation can be complete (account for 100% of the prediction) but still unfaithful (infidel) if it misallocates importance among the features. Completeness is a necessary but not sufficient condition for a high-quality explanation.
• Practical Implication: Catches explanations that highlight only a dramatic but incomplete subset of reasons for a prediction.