SHAP (SHapley Additive exPlanations)

Also known as the summation property, this is the foundational axiom of SHAP. For a given prediction, the sum of the SHAP values for all features, plus the model's expected output (baseline), equals the actual model prediction. Formally: prediction = expected_value + sum(SHAP_values). This ensures the explanation perfectly reconstructs the prediction for that specific instance.

• Example: In a loan application model, if the baseline approval probability is 30%, and the applicant's features (income, credit score, debt) have SHAP values of +15%, +10%, and -5% respectively, the final predicted probability is 30% + 15% + 10% - 5% = 50%.

This property states that a feature that is not present in the current input instance must be assigned a SHAP value of zero. It ensures that the explanation is only influenced by features that were actually available to the model when making the prediction.

• Practical Implication: It guarantees that placeholder or null values do not artificially contribute to the explanation. This is crucial for handling sparse data or models with conditional feature inputs.

This is the most powerful theoretical guarantee of SHAP. If a model changes such that the marginal contribution of a feature increases or stays the same for every possible subset of other features, that feature's SHAP value will not decrease. This property ensures that explanations are faithful to model improvements.

• Consequence: It prevents explanation methods from being gamed or producing contradictory results. If you retrain a model to rely more heavily on a specific feature (e.g., credit_score), its SHAP values will consistently reflect that increased importance.

Two features that have identical contributions to all possible coalitions (subsets of other features) must receive equal SHAP values. This enforces fairness in attribution for features that the model treats as functionally equivalent.

• Example: In an image classifier, if two identical color channels (e.g., from a duplicated sensor) provide the exact same information to the model, their SHAP values for a prediction will be the same, regardless of their arbitrary ordering in the input vector.

While SHAP values are calculated for individual predictions (local explanations), they can be aggregated across a dataset to provide robust global model insights. Common aggregations include:

• Mean Absolute SHAP: The average of the absolute SHAP values for a feature across all instances, representing its overall global importance.
• SHAP Summary Plot: Displays the distribution of SHAP values for each feature, showing both impact (value) and direction (positive/negative correlation with output).
• Dependence Plots: Plots a feature's SHAP value against its actual value, revealing complex relationships and interactions.

The expected value (phi_0) is the average model prediction over the entire training dataset. It serves as the reference point from which all SHAP contributions are calculated. A feature's SHAP value answers: "How much did this feature move the prediction away from this baseline average for this specific instance?"

• Critical Role: This baseline is what makes SHAP values contrastive explanations. They explain the difference between the actual prediction and the average prediction, not the raw output in isolation. Understanding the baseline is key to correctly interpreting the magnitude and sign of SHAP values.

In regulated industries (finance, healthcare, insurance), 'right to explanation' mandates require justifying automated decisions. SHAP generates quantitative, instance-level explanations that can be documented for auditors. For example, a credit denial letter can include: "Your application was primarily influenced by: 1) Credit utilization (SHAP value: +0.32), 2) Number of recent inquiries (+0.18)." This provides actionable feedback to consumers and a defensible audit trail for the organization. It directly supports compliance with regulations like the EU's GDPR, which requires meaningful information about the logic of automated decision-making.

SHAP values serve as a rich signature of model behavior. By tracking the distribution of SHAP values for key features over time in production, teams can detect drift in model reasoning, not just in input data or output scores. A sudden decrease in the SHAP value for a previously important feature signals concept drift—the relationship between that feature and the target has changed. This is a more sensitive and actionable alert than monitoring average prediction scores alone, enabling proactive model retraining before performance degrades.

SHAP's foundation in cooperative game theory provides a principled, consistent story for technical and non-technical audiences. Visualizations like force plots and summary plots translate complex model mechanics into intuitive narratives.

• Data Scientists use it to validate model logic with peers.
• Business Analysts use global summaries to understand model drivers.
• End-Users receive clear reasons for decisions affecting them. This shared understanding builds organizational trust in AI systems and facilitates collaboration between technical builders and business stakeholders.

SHAP provides a quantitative method to pressure-test models against expert intuition. If domain experts believe 'feature X' is critically important, but SHAP shows a near-zero mean attribution, it triggers a vital investigation: Is the model flawed, or is the expert intuition outdated? Conversely, a high SHAP value for a non-intuitive feature can reveal novel, data-driven insights. This creates a feedback loop where expert knowledge informs model development, and model explanations refine expert understanding, leading to more reliable and insightful AI systems.

A model-agnostic explanation technique that, like SHAP, explains individual predictions. LIME works by:

• Perturbing the input instance to create a dataset of similar, slightly altered samples.
• Observing the black-box model's predictions on these perturbed samples.
• Fitting a simple, interpretable surrogate model (like a linear model) to this local dataset. The coefficients of this local surrogate serve as the explanation. While intuitive, LIME's explanations can be sensitive to the choice of perturbation kernel and lack SHAP's game-theoretic guarantees of consistency and fair attribution.

The overarching class of explainability methods to which SHAP belongs. Feature attribution assigns a numerical importance score to each input feature for a specific prediction. Key dimensions for comparing methods include:

• Local vs. Global: Does it explain a single prediction (local) or the model's overall behavior (global)?
• Model-specific vs. Model-agnostic: Does it require internal model knowledge (e.g., gradients) or treat the model as a black box?
• Theoretical Guarantees: Does it satisfy desirable properties (e.g., SHAP's efficiency, symmetry, dummy, additivity)? Other prominent attribution methods include Integrated Gradients (for differentiable models) and TreeSHAP (the highly efficient, model-specific implementation for tree ensembles).

An explanation format that answers "What would need to change to get a different outcome?" Instead of assigning credit, a counterfactual explanation provides a minimal, realistic change to the input features that would flip the model's prediction. For example, "Your loan was denied. If your annual income had been $5,000 higher, it would have been approved."

• Focuses on actionable insights for the subject of the decision.
• Defines a contrastive case (the 'what if' scenario).
• Can be generated using optimization techniques to find the nearest input with the desired prediction class.

Quantitative metrics for validating the quality of feature attributions like SHAP values.

• Faithfulness (Infidelity): Measures how accurately the explanation reflects the model's true function. A common test is to perturb features in order of importance and measure if the prediction change correlates with the attribution scores. Low faithfulness indicates the explanation is misleading.
• Completeness: Ensures the attribution accounts for the entire prediction. For methods like SHAP, completeness is guaranteed by the efficiency property (Shapley values sum to the difference between the actual prediction and the average prediction). For other methods, it must be explicitly verified.

A core technique for both generating and validating explanations. It involves systematically modifying model inputs and observing output changes.

• For Explanation Generation: Methods like LIME and Occlusion Sensitivity (for images) create explanations by perturbing inputs and fitting a surrogate or measuring prediction drops.
• For Explanation Validation: The Randomization Test (or Model Randomization Test) is a critical sanity check. It compares attributions from the trained model to those from a randomly initialized version. A valid explanation method should produce significantly different results, confirming it is explaining learned patterns, not model architecture. Perturbation is fundamental to assessing explanation robustness and stability.