Explanation Fidelity

Faithfulness (or Fidelity) is the primary metric, measuring the correlation between the importance scores assigned by an explanation and the actual impact on model output. A faithful explanation correctly identifies which features the model truly relies on.

• Evaluation Method: Perturb important features identified by the explanation and measure the resulting drop in model prediction confidence or accuracy. A larger drop indicates higher faithfulness.
• Example: If an explanation highlights a specific node in a knowledge graph as critical, masking that node should cause a significant prediction change. A Faithfulness Score quantifies this relationship.

This metric evaluates the completeness of an explanation by measuring the prediction change when removing all important features versus a minimal sufficient subset.

• Comprehensiveness: The drop in model performance when the top-K most important features (per the explanation) are removed. Tests if the explanation captures enough of the model's reasoning.
• Sufficiency: The model's performance when only the top-K features are retained. Tests if the identified features are the minimal set needed for the prediction. An optimal explanation balances both, identifying a compact yet complete set of causal factors.

Monotonicity assesses whether the importance ranking provided by an explanation is consistent with the model's sensitivity. As more features are removed according to their explanation-based importance ranking, the model's prediction should degrade monotonically.

• Violation Example: If removing the 2nd-most important feature causes a larger prediction change than removing the 1st-most important feature, the explanation's ranking lacks monotonicity and is less trustworthy.
• Use Case: Critical for graph-based explanations where the order of node/edge importance must reflect the GNN's actual dependency structure.

Robustness measures an explanation's sensitivity to minor, semantically meaningless perturbations in the input. A robust explanation should not change drastically for nearly identical inputs.

• Evaluation: Apply small noise or transformations to the input graph (e.g., adding an irrelevant node) and measure the variance in the resulting explanations using metrics like Jaccard similarity or Rank correlation.
• Importance: Low robustness indicates the explanation method is capturing noise or artifacts rather than the model's stable decision logic. This is a key concern for post-hoc explainers like LIME for Graphs.

This metric quantifies the parsimony of an explanation. The Minimum Sufficient Explanation principle states that the best explanation is the simplest one that adequately accounts for the prediction.

• Measurement: Often the count of features (nodes, edges, triples) highlighted. For a knowledge-graph-grounded explanation, a lower number of referenced ontological concepts or facts is generally preferable, assuming sufficiency is maintained.
• Human-Centric Value: Sparse explanations are easier for human stakeholders (e.g., auditors, domain experts) to validate and trust, directly supporting Algorithmic Explainability goals.

This advanced metric evaluates if an explanation accurately reflects the implementation of the model, not just its input-output behavior. It tests whether the explanation's attributed importance aligns with the model's internal computational graph.

• Methodology: For Graph Neural Networks, this can involve comparing explanation-attributed importance to gradients, activation patterns, or attention weights within the specific GNN layers.
• Distinction: Goes beyond faithfulness (which is behavioral) to assess architectural fidelity. A high score here means the explanation correctly reverse-engineers the model's actual computational steps, a gold standard for GNN Explainers.

Post-hoc explanation methods must approximate the behavior of highly complex, non-linear models (e.g., deep neural networks) using inherently simpler, interpretable surrogates (e.g., linear models, decision rules). This fundamental complexity gap means the explanation is, by definition, a lossy compression of the model's true function. High fidelity requires this approximation to hold not just for a single input but across the model's operational domain, which is computationally intensive to verify.

A core tension exists between faithfulness (accurately describing the model's internal process) and plausibility (providing an explanation that seems reasonable to a human). Many methods optimize for human-understandable, coherent stories, which can inadvertently sacrifice fidelity. For example, an explanation might highlight features a human expert expects, while the model actually relied on subtle, counter-intuitive correlations in the data. Distinguishing between the two requires rigorous, model-driven evaluation.

High-fidelity explanations should be robust. However, many explanation techniques are unstable: minor, imperceptible perturbations to the input can lead to radically different explanations, even if the model's prediction and confidence remain unchanged. This sensitivity undermines trust and indicates the explanation is not capturing a stable aspect of the model's reasoning. It also poses a security risk, as explanations can be adversarially manipulated without affecting the prediction.

Quantifying fidelity is a meta-challenge because the 'ground truth' of a black-box model's reasoning is, by definition, inaccessible. Common evaluation approaches have significant limitations:

• Faithfulness Metrics: Measure correlation between explanation importance scores and the impact of feature perturbation on the model output.
• Robustness Tests: Check for consistency under input noise or model retraining.
• Sanity Checks: Ensure explanations are sensitive to the model's parameters, not just the data. The lack of a single, definitive metric makes consistent benchmarking difficult.

In machine learning, the Rashomon Effect describes a scenario where many different models (or explanations) can achieve similar predictive performance on the same data. This presents a fundamental challenge: if multiple, internally inconsistent explanations can all align with the model's outputs, which one has high fidelity? The chosen explanation may simply be one of many equally predictive but contradictory stories, making it impossible to uniquely identify the model's true logic from its behavior alone.

Using Enterprise Knowledge Graphs as a source of ground truth presents both a solution and a new challenge. While graphs provide a structured, deterministic framework for validating explanations (e.g., ensuring cited entities and relationships exist), achieving fidelity requires perfect alignment between the model's latent representations and the graph's explicit semantics. Any mismatch—such as the model learning a proxy feature not represented in the ontology—creates a semantic gap that the explanation cannot bridge, reducing its factual accuracy.

A quantitative evaluation of an explanation's accuracy. It measures the correlation between the importance scores an explanation assigns to input features and the actual impact of perturbing (removing or altering) those features on the model's prediction.

• Core Principle: A faithful explanation should identify features that, when changed, cause a significant change in the model's output.
• Measurement: Often calculated using metrics like log-odds or prediction probability drop when important features are masked.
• Directly Related: Explanation Fidelity is often the overarching goal, while faithfulness is a specific, measurable way to assess it.

An explanation generated after a model makes a prediction, using a separate method to interpret the black-box model's output. This is the most common paradigm for explaining complex models like deep neural networks.

• Contrast with Intrinsic Explainability: Unlike interpretable-by-design models (e.g., decision trees), post-hoc methods apply to any model.
• Examples: SHAP, LIME, and GNNExplainer are all post-hoc explanation techniques.
• Fidelity Challenge: The core problem is ensuring these external explanations accurately reflect the internal, opaque decision process.

A simple, interpretable model trained to approximate the predictions of a complex, black-box model for the purpose of generating explanations.

• Function: Acts as a local or global proxy. For example, LIME trains a linear model to mimic the complex model's behavior around a single prediction.
• Fidelity Link: The explanation's fidelity is directly tied to how well the surrogate model approximates the original model's function in the relevant region.
• Trade-off: There is often a tension between the surrogate's simplicity (for interpretability) and its accuracy (for fidelity).

An explanation method that can be applied to any machine learning model without requiring internal access to its architecture, parameters, or gradients.

• Advantage: Provides flexibility to explain diverse model types (trees, neural networks, ensembles) with a single technique.
• Methods: SHAP and LIME are the most prominent model-agnostic approaches.
• Fidelity Consideration: Being agnostic often means the method must treat the model as a black-box function, which can make achieving high fidelity more challenging compared to model-specific methods.

A key distinction in the scope of an explanation.

• Local Explanation: Justifies a single prediction for a specific input instance (e.g., "Why was this loan application denied?").
• Global Explanation: Describes the overall behavior or logic of the model across many or all instances (e.g., "What features are most important for all loan decisions?").
• Fidelity Implications: It is generally easier to achieve high explanation fidelity for a local region around a single data point than for the entire, potentially highly complex, global model function.

Fundamental but often conflated concepts in XAI.

• Interpretability: Refers to the ability to understand a model's mechanics and decision logic inherently from its structure (e.g., a short decision tree or a linear regression with few coefficients).
• Explainability: Involves using external, post-hoc methods to provide understandable reasons for the behavior or outputs of a model that is not inherently interpretable.
• Relation to Fidelity: For interpretable models, fidelity is inherent. For explainable systems, fidelity is a critical metric for the external explanation's validity.