Feature Attribution

Also known as fidelity, this is the most critical property. A faithful explanation accurately reflects the true reasoning process of the underlying model for a specific prediction. It answers: does the importance score for a feature correlate with its actual impact on the model's output?

• Quantified by metrics like Infidelity and Faithfulness Score.
• Validated via Perturbation Analysis: systematically removing or altering high-attribution features should cause a significant change in the model's prediction.
• A method lacking faithfulness is misleading and cannot be used for debugging or trust.

This property ensures an explanation accounts for the total contribution of all input features to the model's prediction. The sum of the importance scores for all features should equal the difference between the model's output for the instance and a defined baseline (e.g., the model's output for a neutral input).

• Core to additive feature attribution methods like SHAP and Integrated Gradients.
• A Completeness Score measures the deviation from this ideal.
• Incomplete explanations may omit subtly influential features, providing a fragmented view of model logic.

A robust explanation should be consistent for semantically similar inputs. Small, meaningless perturbations to the input (e.g., adding image noise) should not cause large, arbitrary swings in the assigned feature importance scores.

• Measured by a Stability Score across similar instances or perturbed versions.
• Lack of robustness indicates the explanation method is sensitive to noise rather than model logic, reducing trust.
• The Randomization Test is a key sanity check: attributions for a trained model should differ significantly from those for a randomly initialized model.

Sparsity refers to an explanation that identifies a minimal set of decisive features. Human cognitive load is limited; highlighting every feature is not interpretable. A sparse explanation isolates the few critical factors driving the prediction.

• Contrasts with dense, noisy saliency maps that highlight most of an image.
• Methods like Anchors explicitly generate sparse, high-precision rules.
• Must be balanced with completeness—over-sparsity can omit legitimately contributing features.

Many real-world explanations are inherently contrastive. We ask "Why did the model predict fraud instead of legitimate?" Contrastive explanations isolate the features most responsible for the chosen prediction relative to a specific alternative.

• Directly answers practical 'why not?' questions crucial for error analysis and recourse.
• Different from standard attribution, which explains the score for a single class.
• Enhances actionability by clarifying the decision boundary.

The ultimate test of an explanation is whether it improves human understanding or task performance. This is evaluated through extrinsic metrics beyond mathematical fidelity.

• Simulatability: Can a human use the explanation to correctly predict the model's output?
• Human-AI Agreement: Does the explanation align with a domain expert's reasoning?
• Decision-Making Speed/Accuracy: Does the explanation help a user (e.g., a loan officer) make a better or faster decision?
• This characteristic bridges technical explainability with real-world usability.

Engineers use feature attribution to diagnose model failures and improve performance. By analyzing incorrect predictions, they can identify if the model is relying on spurious correlations or data artifacts instead of meaningful signals.

• Example: A medical imaging model incorrectly classifies a tumor. A saliency map reveals it focused on a hospital bed tag in the corner of the image, not the tumor morphology. This prompts data cleaning and model retraining.
• Action: Attribution guides feature engineering and data collection strategies by highlighting which inputs the model finds predictive.

Regulations like the EU AI Act and sector-specific rules (e.g., in finance and healthcare) require algorithmic transparency. Feature attribution provides auditable evidence of a model's decision-making process.

• Example: A bank denies a loan application. SHAP values can be generated to show the exact contribution of income, debt-to-income ratio, and credit history to the denial decision, fulfilling right to explanation mandates.
• Action: Attribution scores are logged as part of the model card and decision audit trail, enabling external auditors to verify the absence of illegal discrimination.

Presenting explanations alongside predictions increases user adoption and trust, especially for high-stakes decisions. This enables human-AI collaboration where experts can validate or override model suggestions.

• Example: A radiologist using an AI diagnostic aid sees a LIME explanation highlighting the lung nodules that led to a 'high risk' prediction. The doctor can concur or note if the model focused on irrelevant scar tissue.
• Action: Interactive dashboards integrate attribution visualizations, allowing users to query 'why?' and build calibrated trust in the system's capabilities.

In research fields like bioinformatics, genomics, and material science, models are used for hypothesis generation. Feature attribution can uncover novel, non-intuitive relationships in complex data.

• Example: A graph neural network predicts a new drug compound's efficacy. Integrated Gradients applied to the molecular graph identify a specific functional subgroup as critically important, guiding chemists toward synthesizing new analogs.
• Action: Attribution acts as a feature importance filter, directing costly wet-lab experiments or simulations toward the most promising candidates identified by the model.

Security teams use attribution to reverse-engineer model vulnerabilities and develop defenses. By understanding what features a model relies on, attackers can craft adversarial examples; defenders use the same knowledge to harden models.

• Example: Perturbation analysis reveals a self-driving car's vision model is overly sensitive to specific pixel patterns on a stop sign. This insight is used to generate adversarial training data to improve robustness.
• Action: Explanation-guided red teaming systematically tests if small, imperceptible changes to important features (as identified by attribution) can cause prediction flips, quantifying explanation robustness.

Beyond model mechanics, attribution reveals actionable business insights by quantifying what factors drive key predictions, such as customer churn risk or sales forecasts.

• Example: A customer lifetime value (CLV) model uses hundreds of behavioral features. Feature attribution shows that the frequency of using a specific product feature is the strongest positive driver, while a recent support ticket is the strongest negative driver.
• Action: Product teams prioritize enhancing the high-value feature, while customer success teams develop interventions for users who file tickets, directly linking model output to business strategy.

A unified, game theory-based framework for feature attribution. It calculates the Shapley value for each feature, representing its average marginal contribution to the prediction across all possible combinations of features. SHAP provides a theoretically sound foundation for local explanations that are both consistent and locally accurate.

• Core Principle: Based on cooperative game theory, ensuring fair credit distribution.
• Output: For a single prediction, SHAP values sum to the difference between the model's actual output and its expected (baseline) output.
• Use Case: Highly valued for its mathematical rigor in finance and healthcare for auditing individual model decisions.

A model-agnostic technique that explains individual predictions by approximating the complex black-box model with a simple, interpretable surrogate model (like linear regression) trained on perturbed samples around the instance of interest.

• Method: Generates a new dataset of perturbed inputs and their corresponding black-box predictions, then fits an interpretable model to this local data.
• Key Feature: Explains why you should trust (or not trust) a prediction by showing the locally influential features.
• Limitation: The explanation is only faithful to the surrogate model, not directly to the black-box model's decision boundary.

A gradient-based attribution method that assigns importance by integrating the model's gradients along a straight-line path from a baseline input (e.g., a black image or zero vector) to the actual input.

• Theoretical Guarantees: Satisfies completeness (attributions sum to the prediction difference) and sensitivity.
• Baseline Choice: The baseline represents an 'absence of signal' and is critical; common choices include a zero vector or an average input.
• Application: Particularly effective for explaining deep networks in computer vision and structured data tasks, providing pixel or feature-level attributions.

A contrastive explanation method that answers "What is the minimal change needed to alter the prediction?" It provides actionable insights by showing a similar data point that would have received a different (desired) outcome.

• Format: "Your loan was denied because your income is $50k. If your income were $55k, it would have been approved."
• Key Property: Focuses on actionability and recourse for the end-user.
• Evaluation: Measured by proximity (how close the counterfactual is to the original) and sparsity (how few features were changed).

A visual explanation technique, primarily for convolutional neural networks (CNNs), that highlights the regions of an input image most responsible for a prediction. Common methods include computing the gradient of the output class score with respect to the input pixels.

• Visual Output: A heatmap overlaid on the original image, where 'hot' colors indicate high importance.
• Simple Example: For an image classified as 'dog', the saliency map should highlight the dog's face and body, not the background.
• Caveat: Basic gradient-based saliency maps can be noisy; advanced variants like Grad-CAM produce smoother, more coherent visualizations by using feature map activations.

Quantitative metrics for post-hoc explanation validation.

• Faithfulness Score: Measures how accurately the explanation reflects the model's true reasoning. A common test is perturbation-based: remove features deemed important by the explanation; a faithful explanation will cause a large drop in the model's prediction confidence.
• Completeness Score: Evaluates whether the explanation accounts for all significant contributing factors. If the sum of attribution scores for the top-k features equals (or is proportional to) the total prediction output, completeness is high.

These scores are essential for moving from qualitative, visual explanations to quantitatively validated ones suitable for regulatory audits.