Post-hoc Explanation

A defining feature of post-hoc methods is their model-agnostic capability. They treat the AI model as a black-box, requiring only access to its inputs and outputs. This allows the same explanation technique (e.g., LIME or SHAP) to be applied to vastly different models, from deep neural networks to ensemble methods, without knowledge of their internal architecture. This universality is critical for enterprises with diverse, legacy ML systems.

Post-hoc explanations operate at two primary scopes:

• Local Explanations: Justify a single prediction for a specific input instance. For example, explaining why a loan was denied for Applicant X by highlighting their low credit score and high debt-to-income ratio as key factors.
• Global Explanations: Describe the model's overall behavior or logic across the entire dataset. This might involve identifying which features (e.g., annual_income, loan_term) are most important on average for all predictions. Most methods are designed for one scope, though some, like SHAP, can provide both.

Since they do not access the model's true reasoning, post-hoc explanations are approximations. A core evaluation metric is explanation fidelity, which measures how accurately the explanation reflects the model's actual decision boundary. Low-fidelity explanations are misleading. Methods improve fidelity by perturbing inputs (e.g., creating synthetic data points near the instance) and observing output changes to infer feature importance.

Effective explanations often answer "why this outcome, and not another?"

• Contrastive Explanations: Highlight the features that differentiate the actual prediction from a plausible alternative (e.g., "Your loan was approved rather than denied because your income exceeded the threshold").
• Counterfactual Explanations: Provide actionable, minimal changes to the input to achieve a desired outcome (e.g., "If your credit score were 20 points higher, your loan would be approved"). This is closely related to algorithmic recourse.

Enterprise knowledge graphs elevate post-hoc explanations from feature-level attributions to semantic, human-understandable justifications. Instead of highlighting a raw feature value, an explanation can reference grounded entities and relationships:

• Feature Grounding: Raw features (e.g., a product ID) are linked to graph entities (e.g., Product:XYZ) with rich attributes.
• Causal Context: The graph can provide causal or domain-logic relationships that validate or contextualize the explanation.
• Rule Extraction: Explanations can be formatted as logical rules derived from the graph's ontology, enhancing auditability.

The utility of an explanation is ultimately judged by humans. Evaluation extends beyond computational metrics like fidelity to include:

• Comprehensibility: Is the explanation format (visual, textual, rule-based) understandable to the stakeholder (e.g., an engineer vs. a compliance officer)?
• Actionability: Does it provide insights that lead to better decisions or model improvements?
• Trust Calibration: Does it correctly increase or decrease user trust in the model's output? Interactive explanations that allow users to query and explore the reasoning are a key advancement in this area.

SHAP is a game theory-based method that attributes a model's prediction to each input feature by calculating its Shapley value. This value represents the average marginal contribution of a feature across all possible combinations of other features.

• Model-Agnostic: Can explain any machine learning model, including complex graph neural networks.
• Theoretically Grounded: Based on cooperative game theory, providing a unique, consistent attribution.
• Local & Global Explanations: Generates explanations for single predictions (local) and can aggregate them to understand overall model behavior (global).
• Example: In a credit scoring model, SHAP can show that 'annual income' contributed +50 points to a score, while 'number of recent credit inquiries' contributed -20 points.

LIME explains individual predictions by approximating the complex model locally with a simple, interpretable surrogate model (e.g., a linear model or decision tree).

• Local Fidelity: The surrogate model is trained on perturbed samples around the instance to be explained, ensuring it mimics the black-box model's behavior in that local region.
• Interpretable Representations: Operates on a human-understandable representation of the input, which for graphs could be the presence or absence of specific nodes, edges, or subgraphs.
• Flexible: Can be applied to text, images, tabular data, and graph-structured data with appropriate perturbation strategies.
• Limitation: Explanations are only faithful to the black-box model in the immediate vicinity of the prediction.

A Counterfactual Explanation answers the question: "What minimal changes to the input would have led to a different (usually more favorable) prediction?"

• Actionable Insights: Provides a clear, actionable path for recourse. For example, "Your loan would have been approved if your income was $5,000 higher."
• Plausibility & Proximity: A good counterfactual should be a realistic, small perturbation of the original input.
• Knowledge Graph Integration: In a graph context, a counterfactual might suggest adding a specific relationship (edge) or changing a node attribute.
• Contrastive Nature: Inherently answers a 'why not?' question, which aligns with human reasoning patterns.

Saliency Maps highlight the specific parts of an input (pixels in an image, words in text, nodes/edges in a graph) that were most influential for a given prediction.

• Gradient-Based Methods: Techniques like Grad-CAM or Integrated Gradients use the model's gradients with respect to the input to determine importance.
• Visual Interpretability: Provides an intuitive, visual heatmap overlay on the original input.
• Graph-Specific Adaptations: For Graph Neural Networks (GNNs), methods like GNNExplainer generate a saliency mask over nodes and edges, identifying the critical subgraph for a prediction.
• Use Case: In a molecule property prediction task, a saliency map can highlight the atomic substructure the model used to classify a compound as toxic.

Rule-Based Explanation methods extract human-readable IF-THEN rules that describe the conditions leading to a model's prediction.

• Anchors: A model-agnostic algorithm that finds a sufficient condition (an 'anchor') for a prediction. The rule is: "IF [anchor condition holds], THEN the prediction will be [class Y] with high probability."
• Global Surrogate Rules: A decision tree or rule list is trained to approximate the global behavior of the black-box model, providing an overall, interpretable proxy.
• High Interpretability: Logical rules are easily understood, audited, and validated against domain knowledge or a knowledge graph's ontology.
• Deterministic: Rules provide clear, Boolean logic, which is valuable for compliance and debugging in regulated industries.

These methods explain a prediction by contrasting it with an alternative outcome or by identifying causal drivers.

• Contrastive Explanations: Answer "Why prediction P, rather than plausible alternative Q?" by pinpointing the discriminating features. This aligns with natural human inquiry.
• Causal Explanations: Go beyond correlation to identify cause-and-effect relationships. They often require or leverage a structural causal model or a causal knowledge graph to validate paths of influence.
• Example: A contrastive explanation for a denied application might be: "The application was denied (rather than approved) primarily due to the applicant's debt-to-income ratio exceeding 40%."
• Higher Fidelity: Causal explanations, when derived from valid causal models, provide more robust and stable interpretations that hold under intervention.

A Model-Agnostic Explanation method can generate interpretations for any machine learning model without requiring internal access to its architecture or parameters. These techniques treat the model as a black-box, querying it with different inputs to understand its behavior.

• Key Examples: LIME (Local Interpretable Model-agnostic Explanations) and SHAP (Shapley Additive exPlanations).
• Primary Use: Providing explanations for complex, proprietary, or ensemble models where internal weights are unavailable or meaningless to end-users.
• Contrast with Intrinsic Explainability: Unlike intrinsically interpretable models (e.g., decision trees), model-agnostic methods are applied after training.

This distinction defines the scope of an explanation. A Local Explanation justifies a single prediction for a specific data instance (e.g., "Why was this loan application denied?"). A Global Explanation describes the overall behavior or logic of the model across its entire operating domain (e.g., "What features are most important for all loan decisions?").

• Local Methods: LIME, SHAP (local), counterfactual explanations.
• Global Methods: Partial dependence plots, global feature importance, rule extraction.
• Post-hoc Context: Most post-hoc techniques start by generating local explanations, which can be aggregated to infer global behavior.

Explanation Fidelity is a quantitative metric measuring how accurately a post-hoc explanation approximates the decision-making process of the underlying black-box model. The Faithfulness Metric is a specific evaluation that perturbs features deemed important by the explanation and measures the correlation between that importance and the actual impact on the model's prediction.

• Core Principle: A high-fidelity explanation should be a reliable proxy for the model. If the explanation highlights a feature as critical, changing that feature should significantly alter the model's output.
• Evaluation Challenge: A major research area is developing robust metrics to audit the quality of explanations, as a misleading explanation can be worse than none.

Counterfactual Explanations are a type of post-hoc explanation that answers: "What minimal changes to the input would have led to a different (usually more favorable) outcome?" Instead of highlighting important features, they provide an actionable "what-if" scenario.

• Example: "Your loan was denied due to a low credit score. If your score had been 720 instead of 680, it would have been approved."
• Link to Algorithmic Recourse: Counterfactuals naturally lead to Algorithmic Recourse, which provides actionable recommendations for individuals to achieve a desired outcome.
• Use in Knowledge Graphs: For graph-based predictions, counterfactuals might suggest adding, removing, or changing specific relationships or node attributes.

A Surrogate Model is a simple, interpretable model trained to approximate the predictions of a complex, black-box model for the purpose of generating explanations. It is a core technique in many post-hoc methods.

• How it Works: The black-box model is queried on a dataset (often perturbed versions of an instance). The inputs and corresponding predictions are used to train a transparent surrogate (e.g., a linear model or decision tree).
• Limitation: The surrogate only explains the black-box model's behavior within the sampled region; its fidelity may drop outside this local neighborhood.
• Example: LIME creates a local surrogate linear model to explain a single prediction.

SHAP is a unified, game theory-based framework for explaining the output of any machine learning model. It attributes the prediction for a specific instance to each input feature, calculating its Shapley value—the average marginal contribution of the feature across all possible coalitions (combinations) of other features.

• Theoretical Foundation: Based on cooperative game theory, ensuring properties like local accuracy (the sum of attributions equals the model output) and consistency.
• Model-Agnostic: KernelSHAP is the model-agnostic version. Model-specific, faster implementations exist (e.g., TreeSHAP for tree ensembles).
• For Graphs: Adaptations like GNNShap extend the framework to attribute predictions of Graph Neural Networks to nodes, edges, or features.