Explainable AI (XAI)

Feature attribution methods assign importance scores to individual input features (like words or tokens) to explain a model's prediction. These techniques answer the question: "Which parts of the input most influenced this output?"

• Saliency Maps & Gradient-Based Methods: Visualize importance by calculating the gradient of the output with respect to the input. Common in vision models, adapted for text via token gradients.
• Attention Visualization: For Transformer-based LLMs, the model's internal attention weights can be inspected to see which tokens it "attended to" when generating a response. While intuitive, attention is not a direct measure of causal importance.
• SHAP (SHapley Additive exPlanations): A game theory approach that computes the marginal contribution of each feature to the prediction, providing a consistent and locally accurate attribution.
• LIME (Local Interpretable Model-agnostic Explanations): Approximates the complex model locally with a simple, interpretable model (like linear regression) to explain individual predictions.

Counterfactual explanations answer "What would need to change in the input to get a different output?" They are human-intuitive, as they mirror how people often reason about cause and effect.

• Method: Generate minimal, realistic perturbations to the input that would flip the model's decision. For an LLM that denied a loan application, a counterfactual might show that increasing the applicant's income by $5,000 would have led to approval.
• Use in LLMs: Applied to understand model sensitivity. For example, changing a single keyword in a user query to see if it triggers a safety filter or alters the factual grounding of an answer.
• Advantage: Focuses on actionable insights rather than just highlighting important features, which is valuable for debugging and recourse.

A surrogate model is a simple, interpretable model (like a decision tree or linear model) trained to approximate the predictions of a complex "black box" model. The surrogate's structure provides global intuition about the black box's behavior.

• Global vs. Local: A global surrogate aims to mimic the complex model across its entire input space, while a local surrogate (like LIME) approximates it for a single instance.
• Process: 1. Sample inputs. 2. Get predictions from the black-box LLM. 3. Train an interpretable model on this (input, prediction) dataset.
• Interpretation: The rules or weights of the surrogate model (e.g., "IF query contains 'code' AND 'execute' THEN flag for security review") offer a high-level, human-readable summary of the LLM's decision logic.

The model generates a textual justification for its own output, making the explanation native and accessible. This is increasingly a native capability of advanced LLMs.

• Self-Explaining Models: Some models are trained or prompted to output a chain-of-thought or a final answer accompanied by a reasoning trace (e.g., "I think the answer is X because the document states Y and Z").
• Post-hoc NLE Generation: A separate model or module analyzes the primary model's input and output to generate a textual explanation. This decouples the task model from the explanation generator.
• Challenge: The explanation itself must be faithful (accurately reflecting the model's true reasoning) and not a plausible-sounding but fabricated justification—a form of explanation hallucination.

Instead of explaining predictions in terms of raw features (tokens), concept-based methods explain them using human-understandable concepts (e.g., 'formality', 'toxicity', 'technical jargon').

• Testing with Concept Activation Vectors (TCAV): Measures a model's sensitivity to user-defined concepts. For an LLM, you could test how sensitive a sentiment classification is to the concept of "sarcasm" or how a code-generation model responds to the concept of "security vulnerability."
• Process: 1. Define a concept (e.g., "medical terminology") and provide positive/negative example sets. 2. Learn a direction in the model's activation space corresponding to that concept. 3. Quantify how much the concept influenced a specific prediction.
• Benefit: Provides explanations aligned with human semantic understanding, bridging the gap between low-level features and high-level reasoning.

For Retrieval-Augmented Generation (RAG) systems, a core XAI method is to show the provenance of the generated answer—the specific source documents or data snippets used—and how they were grounded.

• Citation Highlighting: The system returns the generated answer alongside direct citations to the source text that supports each claim, often with highlighted spans.
• Confidence Scoring: Attributing confidence scores to different parts of the answer based on the quality and relevance of the retrieved evidence.
• Retrieval Debugging: Tools to visualize the retrieval step, showing the query, the retrieved chunks, and their similarity scores. This helps diagnose failures where the model either didn't retrieve the right information or ignored the correct evidence it did retrieve.

Interpretability refers to the degree to which a human can understand the cause of a model's decision. It is often considered a broader, more qualitative goal than explainability, focusing on the model's inherent structure and logic.

• Intrinsic vs. Post-hoc: Models can be designed to be intrinsically interpretable (e.g., linear models, decision trees) or require post-hoc methods to explain a pre-existing black-box model.
• Scope: Interpretability can be assessed at the global level (understanding the model's overall logic) or the local level (explaining a single prediction).
• Core Question: It answers "How does the model work?" rather than just "Why did it make this specific output?"

Feature attribution is a class of XAI techniques that assign an importance score to each input feature (e.g., a word, pixel, or data column) indicating its contribution to a model's specific output.

• Key Methods: Includes SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and Integrated Gradients.
• Use Case: In an LLM generating a summary, feature attribution can highlight which words in the source document most influenced the summary's content.
• Output: Often visualized as a saliency map or a list of weighted features.

A saliency map is a visual heatmap overlay that highlights the regions of an input (typically an image) that were most influential in a model's prediction. For text, similar techniques highlight important words or tokens.

• Visual Explanation: Provides an intuitive, pixel-level explanation for computer vision models.
• Generation Methods: Created using backpropagation-based techniques like Grad-CAM or perturbation-based methods.
• Application: In a medical imaging AI, a saliency map can show which areas of an X-ray led to a "pneumonia" classification, aiding radiologist verification.

A counterfactual explanation describes the minimal changes required to an input to alter the model's prediction to a desired outcome. It answers the question, "What would need to be different for a different result?"

• Actionable Insights: Useful for providing recourse, such as telling a loan applicant, "Your application would have been approved if your income were $5,000 higher."
• Proximity & Plausibility: A good counterfactual should be a small, realistic change to the original input.
• Contrastive: It explains a prediction by contrasting it with a nearby, alternative reality.

Model Cards and Datasheets for Datasets are documentation frameworks for transparent reporting of AI model and dataset characteristics. They are foundational to responsible AI and explainability.

• Model Card: A short document providing key facts about a trained model: intended use, performance across different demographics, ethical considerations, and known limitations.
• Datasheet: A similar document detailing the provenance, composition, collection process, and recommended uses of a dataset.
• Purpose: These documents provide static, high-level explainability, enabling developers and auditors to understand a system's context and constraints before deployment.

An Algorithmic Impact Assessment (AIA) is a systematic, often regulatory, process for evaluating the potential risks, benefits, biases, and societal effects of deploying an AI system. It is a governance-level companion to technical XAI.

• Proactive Audit: Conducted before a model is put into production to identify and mitigate harms.
• Holistic Scope: Evaluates factors like fairness, privacy, security, economic impact, and environmental effects.
• Framework: Mandated or encouraged by regulations like the EU AI Act, it forces organizations to document a model's explainability, robustness, and data governance.