Explanation Model

The explanation model must match the original model's output for the specific input being explained. If the original model predicts a value of $f(x)$, the sum of all feature attributions plus a baseline must exactly equal $f(x)$.

• Definition: $f(x) = \phi_0 + \sum_{i=1}^{M} \phi_i$
• Significance: Guarantees the explanation is faithful to the prediction, not an approximation
• Violation: If attributions don't sum to the prediction, the explanation is incomplete or misleading

Features that are not present in the original input must receive an attribution of zero. This ensures that absent features do not artificially influence the explanation.

• Definition: If $x_i' = 0$ (feature is missing), then $\phi_i = 0$
• Practical Impact: Prevents the model from assigning importance to features that don't exist in the instance
• Implementation: Achieved by mapping simplified binary inputs to original feature space where missing features are replaced with baseline values

If a model changes so that a feature's marginal contribution increases or stays the same regardless of other features, that feature's attribution should not decrease. This property ensures attributions track true feature importance.

• Formal Statement: If $f_x'(S \cup {i}) - f_x'(S) \geq f_x(S \cup {i}) - f_x(S)$ for all subsets $S$, then $\phi_i(f', x) \geq \phi_i(f, x)$
• Why It Matters: Prevents counterintuitive situations where a more important feature receives a lower attribution
• Unique Solution: Shapley values are the only additive feature attribution method satisfying consistency

The sum of all feature attributions must exactly equal the difference between the model's prediction and the expected model output. This property, inherited from Shapley values, ensures complete distribution of the prediction among features.

• Equation: $\sum_{i=1}^{M} \phi_i = f(x) - E[f(X)]$
• Interpretation: The total 'effect' is fully partitioned among features with no residual
• Baseline Role: $E[f(X)]$ serves as the starting point; attributions show how each feature pushes away from this average

Two features that make identical marginal contributions across all possible coalitions must receive identical attributions. This ensures the explanation model treats features fairly based solely on their model impact.

• Condition: If $f(S \cup {i}) = f(S \cup {j})$ for all subsets $S$ not containing $i$ or $j$, then $\phi_i = \phi_j$
• Fairness Guarantee: Prevents arbitrary bias in feature importance assignment
• Example: Two perfectly correlated features with identical model effects will receive equal SHAP values

If a model is a linear combination of two sub-models, the attribution for the combined model equals the same linear combination of individual attributions. This enables compositional reasoning about explanations.

• Property: $\phi_i(\alpha f + \beta g) = \alpha \phi_i(f) + \beta \phi_i(g)$
• Ensemble Application: SHAP values for an ensemble model are the weighted average of SHAP values for individual models
• Practical Use: Simplifies explanation computation for bagged or boosted ensembles by decomposing into base learners

A class of explanation models that express a prediction as a linear sum of individual feature contributions relative to a baseline value. This is the mathematical structure underlying SHAP and LIME.

• Linear function of binary variables: The simplest form of an explanation model
• Satisfies local accuracy: The sum of attributions equals the prediction difference
• Enables direct comparison of feature importance across instances

An inherently interpretable model—such as a linear regression, decision tree, or rule list—trained to mimic the predictions of a black-box model. The explanation model is a specific type of surrogate.

• Global surrogates: Approximate the entire model behavior
• Local surrogates: Approximate behavior around a single prediction
• Trade-off between fidelity (accuracy to original) and interpretability

A technique that builds an explanation model by perturbing inputs around a prediction and fitting a sparse linear model to the resulting outputs. LIME is model-agnostic and predates SHAP.

• Uses interpretable representations (e.g., presence of words) rather than raw features
• Weights perturbed samples by proximity to the original instance
• Does not guarantee the efficiency property that SHAP enforces

The expected model output across a background dataset, representing the prediction when no feature information is known. In an explanation model, this is the intercept term from which all feature contributions deviate.

• Computed as E[f(x)] over the background distribution
• The sum of all SHAP values plus the baseline equals the actual prediction
• Choice of background dataset critically impacts explanation semantics

A fundamental axiom requiring that an explanation model exactly matches the original model's output for the specific instance being explained. This ensures the explanation is faithful to the prediction.

• Also called efficiency in the Shapley context
• Guarantees no unexplained residual in the attribution
• Distinguishes additive feature attribution methods from heuristic approaches

A simplified, human-understandable mapping of raw features used as inputs to an explanation model. Binary vectors indicating feature presence are the most common representation.

• Super-pixels for images: Grouped pixel regions
• Bag-of-words for text: Individual token presence
• Binned features for tabular data: Discretized continuous values
• The mapping function between raw features and interpretable representations is critical to explanation quality