Saliency Maps (Graph)

Graph saliency maps attribute importance to graph elements—nodes, edges, and node/edge features—rather than pixels. The attribution must respect the relational inductive bias of the graph. For example, in a molecule graph, a saliency map might highlight a specific carbon ring (subgraph) and the bonds (edges) within it as critical for a toxicity prediction, not just individual atom features.

Explanation methods for graphs fall into two categories:

• Model-Agnostic: Techniques like LIME for Graphs or SHAP for Graph Models treat the Graph Neural Network (GNN) as a black box, perturbing the input graph and observing output changes.
• Model-Specific: Methods like GNNExplainer or PGExplainer are designed for GNN architectures. They leverage internal model states (e.g., gradients, attention weights) to generate explanations, often providing higher explanation fidelity.

Graph saliency can be computed at different scopes:

• Local Explanations: Justify a single prediction for one graph instance (e.g., "Why was this specific protein predicted to bind to this drug?"). The output is a salient subgraph or feature set unique to that instance.
• Global Explanations: Describe the model's general behavior across many graphs (e.g., "What molecular substructures does this model consistently associate with high solubility?"). These are often aggregates of local explanations or learned explanation networks.

Saliency is communicated through different representations:

• Salient Subgraph: A connected subset of nodes and edges identified as most important. This is the most intuitive format for human analysis.
• Feature Mask: A vector of importance scores for each node or edge feature dimension.
• Importance Scores: Numerical scores assigned to every graph element, often visualized with color gradients or thickness on the original graph. High scores indicate high influence on the prediction.

The quality of a graph saliency map is measured quantitatively. Key metrics include:

• Fidelity: Measures how accurately the explanation reflects the model's logic. Fidelity+ tests if removing salient features causes a large prediction drop. Fidelity- tests if keeping only salient features preserves the prediction.
• Sparsity: Encourages concise explanations by penalizing maps that highlight too much of the graph. A good explanation is both faithful (high fidelity) and compact (high sparsity).

In enterprise settings, graph saliency maps are most powerful when the underlying graph is a Knowledge Graph (KG) with a formal ontology. This allows explanations to be grounded in business entities and relationships. For instance, a saliency map on a fraud detection KG can highlight not just account nodes, but specific transaction types (ontology classes) and pathways to known fraud rings, providing auditable, semantic explanations for compliance (Right to Explanation).

Primary use case: Engineers use saliency maps to identify why a GNN made a specific prediction, which is essential for trust and debugging.

• Spotting spurious correlations: Reveals if a model is relying on irrelevant subgraph structures or node attributes (e.g., a recommendation model focusing on a user's join date instead of their purchase history).
• Validating model logic: Confirms the model is using semantically meaningful patterns aligned with domain knowledge (e.g., a fraud detection GNN correctly highlights known suspicious transaction pathways).
• Improving architecture: Informs decisions on model depth, aggregation functions, and feature engineering by showing what information propagates through the network.

Critical for deterministic RAG: In Retrieval-Augmented Generation systems grounded in knowledge graphs, saliency maps explain which facts from the graph contributed to an answer.

• Attribution for generated text: Highlights the specific entities and relationships retrieved from the KG that led to a model's final output, providing traceability and reducing hallucinations.
• Auditing answer provenance: Allows users to verify an LLM's response by inspecting the activated subgraph, building confidence in enterprise AI assistants.
• Improving retrieval: Identifies gaps or irrelevant facts retrieved, guiding the optimization of the retrieval step in the RAG pipeline.

Essential in computational chemistry: Saliency maps applied to molecular graphs (atoms as nodes, bonds as edges) explain why a model predicts a specific biochemical property.

• Identifying pharmacophores: Highlights molecular substructures (functional groups, rings) critical for a drug's binding affinity or toxicity, guiding chemists in lead optimization.
• Validating against domain science: Ensures the AI's reasoning aligns with established chemical principles, a requirement for regulatory submission in pharmaceuticals.
• Accelerating iterative design: Provides actionable feedback for the next round of molecular synthesis by pinpointing which parts of a compound to modify.

Key for investigative efficiency: In graphs where nodes are accounts and edges are transactions, saliency maps pinpoint the most suspicious pathways and entities.

• Focusing analyst attention: Instead of reviewing thousands of transactions, investigators are directed to the specific subgraph (e.g., a set of accounts and money flows) the model found most anomalous.
• Meeting regulatory explainability mandates: Provides auditable, case-specific evidence for why a transaction was flagged, supporting compliance with regulations like the EU's AI Act or financial oversight rules.
• Uncovering novel fraud patterns: Helps human experts discover new, complex fraud schemes by visualizing the non-linear patterns the model has detected.

Drives transparency and user trust: For graphs linking users, items, and their interactions, saliency maps explain why a particular item was recommended.

• Explaining recommendations: Shows the influential users in a social network (social influence) or the key item attributes and past purchases that led to a suggestion.
• Mitigating filter bubbles and bias: Reveals if recommendations are overly reliant on a narrow set of features or a homophilic cluster, allowing engineers to debias the system.
• A/B testing and optimization: Provides causal insights into which graph features drive user engagement, informing better model design.

Foundational for responsible AI: Saliency maps provide the technical evidence required for internal audits and regulatory compliance.

• Satisfying the 'Right to Explanation': Enables the generation of meaningful, instance-specific explanations for automated decisions that affect individuals (e.g., loan denials, content moderation).
• Demonstrating fairness: Auditors can check if saliency attributions are consistently derived from legally protected attributes (like gender or race nodes/edges), revealing potential discrimination.
• Creating an explanation audit trail: The saliency scores themselves become part of the model's telemetry, logged for reproducibility and regulatory review.

GNN Explainers are specialized algorithms designed to provide post-hoc explanations for predictions made by Graph Neural Networks. Unlike generic saliency methods, they are architected for graph structures.

• GNNExplainer: Identifies a compact subgraph and a small subset of node features that are most influential for a model's prediction on a given node or graph.
• PGExplainer: A parameterized approach that learns to generate explanations for multiple instances, offering improved efficiency over instance-specific methods.
• These tools are critical for debugging GNNs, validating that the model uses semantically meaningful substructures (e.g., a functional group in a molecule) rather than spurious correlations.

SHAP (Shapley Additive exPlanations) adapted for graphs attributes a model's prediction to its input nodes, edges, or features based on cooperative game theory. It computes the average marginal contribution of a graph element across all possible coalitions (subsets) of other elements.

• Provides a rigorous, theoretically grounded measure of importance with desirable properties like local accuracy and consistency.
• For graphs, the "players" in the game can be nodes, edges, or their features. Computing exact Shapley values is combinatorially explosive, so approximations like KernelSHAP or GraphSHAP are used.
• Outputs values that indicate if a node's presence increases (positive SHAP value) or decreases (negative SHAP value) the predicted score relative to a baseline.

A Counterfactual Explanation for a graph model describes the minimal changes needed to the input graph to alter the model's prediction to a desired outcome. It answers "What if?" questions.

• Example: For a GNN that predicts a molecule as toxic, a counterfactual might be: "If this hydroxyl group (-OH) were removed, the molecule would be predicted as non-toxic."
• The changes are typically edits like adding/removing edges, nodes, or modifying node features.
• This method is highly actionable for domain experts (e.g., chemists, fraud analysts) as it suggests concrete interventions.

In Graph Attention Networks (GATs), the attention weights learned during message passing can be used as a form of intrinsic explanation. These weights indicate how much a node aggregates information from each of its neighbors when computing its new representation.

• Analysis of attention weights can reveal which neighbor relationships the model deems important for a given task.
• However, attention is not always a faithful explanation; high attention to a neighbor does not guarantee that neighbor's features were decisive for the final prediction. It explains the information flow, not necessarily the prediction cause.
• Often used in conjunction with post-hoc saliency methods for a more complete picture.

A Rule-Based Explanation extracts human-readable, logical rules from a model's decision process or from the knowledge graph itself to justify a prediction. This is a hallmark of neuro-symbolic AI systems.

• Example: A loan denial explanation: IF (Employment_Status = 'Unemployed') AND (Credit_Score < 580) THEN (Application = 'Denied').
• For graphs, rules may be expressed in terms of graph patterns or paths (e.g., IF (Drug) -[treats]-> (Disease) AND (Drug) -[has_side_effect]-> (Severe_Side_Effect) THEN (Risk_Score = High)).
• Provides high interpretability and can be directly validated against business logic or regulatory policies.

Explanation Fidelity refers to how accurately a post-hoc explanation approximates the true decision-making process of the underlying black-box model. It is measured by Faithfulness Metrics.

• Faithfulness (or Fidelity): Measures the correlation between the importance scores assigned by the explanation and the actual impact on the model's output when those features are perturbed or removed.
• Sparsity: A good saliency map should be concise, highlighting only the most critical elements. Metrics assess if the explanation is overly dense.
• Stability: The explanation should be robust to insignificant perturbations in the input graph. Evaluating these metrics is essential for trusting and deploying saliency methods in production.