Feature Attribution

An attribution method satisfies completeness when the sum of all feature importance scores equals the difference between the model's output for the actual input and a predefined baseline input. This property, also known as efficiency, ensures that the explanation fully accounts for the model's prediction. Methods like Shapley Additive Explanations (SHAP) are designed to guarantee this property, providing a closed-form budget of relevance. Without completeness, an explanation might omit critical contributing factors, leading to an incomplete audit trail.

• Mathematical form: Σ attributions = f(x) - f(baseline)
• Violation example: A saliency map that highlights only 80% of the contributing pixels leaves 20% of the decision logic unexplained.
• Key differentiator: Gradient-based methods like Vanilla Gradients often violate completeness due to saturation effects in deep networks.

Two functionally equivalent models should always receive identical attributions, regardless of their internal architecture. This property mandates that feature attribution must depend solely on the mathematical function mapping inputs to outputs, not on incidental implementation details like the number of layers or activation functions. Integrated Gradients and SHAP are designed to be implementation-invariant by operating on the function's gradients or marginal contributions. Methods that rely on inspecting internal weights, such as Grad-CAM, are inherently implementation-dependent.

• Test case: A decision tree and a neural network trained to produce identical outputs for all inputs must yield identical feature importance scores.
• Failure mode: Explaining two functionally identical networks differently breaks trust in the explanation method itself.
• Practical relevance: Essential for comparing explanations across different model versions or compressed variants.

If a feature is mathematically irrelevant to the prediction—meaning the model's output never changes regardless of that feature's value—it must receive an attribution of exactly zero. This property, also called the null player axiom in game theory, prevents explanation methods from hallucinating importance for variables the model ignores. A common violation occurs with perturbation-based methods that accidentally corrupt the input distribution, causing the model to extrapolate and produce non-zero gradients for dummy features.

• Axiom: f(x) constant w.r.t. feature i → attribution_i = 0
• Violation example: Adding Gaussian noise to a dummy feature changes the input manifold, causing a LIME explainer to assign non-zero weight.
• Robustness check: Concatenate a random noise channel to an image input; a valid explanation must assign zero importance to that channel.

For an ensemble model that averages the outputs of multiple sub-models, the attribution for the ensemble must equal the weighted average of the attributions for each sub-model. This property ensures logical consistency when explaining model averaging or mixture of experts architectures. Shapley values naturally satisfy linearity due to their additive construction. Gradient-based methods satisfy this only if the gradient computation is linear, which holds for standard backpropagation but not for non-differentiable ensembling.

• Formal definition: attr(f_ensemble) = Σ w_k * attr(f_k)
• Use case: Explaining a Random Forest by averaging SHAP values across all trees.
• Counterexample: A black-box explainer that treats the ensemble as a monolith may produce attributions that don't decompose correctly.

Two features that play identical functional roles in the model must receive identical attribution scores. If swapping the values of feature A and feature B leaves the model output unchanged for all possible inputs, then A and B are symmetric and must be assigned equal importance. This property prevents arbitrary bias in the explanation. Shapley values enforce symmetry by construction, as they average marginal contributions over all possible feature orderings.

• Test: In a model where f(x1, x2) = x1 + x2, both features must receive equal attribution.
• Violation: A saliency map computed via input gradients may assign higher importance to x1 if its numerical scale is larger, even though the function is symmetric.
• Mitigation: Normalizing input features before gradient computation can partially address scale-induced asymmetry.

Local attribution explains a single prediction by identifying which features drove that specific output, essential for debugging individual errors or providing user recourse. Global attribution aggregates importance across the entire dataset to reveal the model's overall learned dependencies, useful for regulatory compliance and feature selection. Methods like SHAP offer both via instance-level Shapley values and global bar plots of mean absolute SHAP values. LIME is inherently local, while permutation feature importance is inherently global.

• Local example: Explaining why a specific loan application was denied by highlighting the applicant's debt-to-income ratio.
• Global example: Identifying that 'age' is the most influential feature across all loan decisions, prompting a fairness audit.
• Trade-off: Local fidelity often comes at the cost of global consistency, and vice versa.