Explainable AI (XAI)

Feature attribution methods assign an importance score to each input feature (e.g., a pixel in an image or a word in text) to explain a model's prediction. These scores indicate how much each feature contributed to the final output.

• Saliency Maps: Visual heatmaps highlighting pixels most influential for an image classifier's decision.
• SHAP (SHapley Additive exPlanations): A game-theoretic approach providing consistent and locally accurate feature importance values.
• Integrated Gradients: Attributes the prediction to input features by integrating the model's gradients along a path from a baseline input.

These techniques are foundational for debugging model biases, such as a medical imaging model incorrectly focusing on a hospital bed tag instead of a tumor.

Surrogate models are simple, interpretable approximations (like linear models or decision trees) trained to mimic the predictions of a complex 'black-box' model locally or globally.

• LIME (Local Interpretable Model-agnostic Explanations): Creates a linear model that approximates the complex model's behavior for a single prediction by perturbing the input and observing changes in output.
• Global Surrogates: A single interpretable model trained on the dataset and the black-box model's predictions to provide a holistic, approximate understanding of its logic.

This method is model-agnostic, making it applicable to any algorithm, from deep neural networks to ensemble methods.

Rule extraction techniques distill the knowledge encoded within a neural network into a set of human-readable if-then rules or a decision tree.

• Pedagogical Approach: Treats the neural network as an oracle; an interpretable model is trained on its input-output pairs.
• Decompositional Approach: Analyzes the internal structure (weights, activations) of the network to extract rules directly from its components.

This is vital for regulatory compliance and safety validation, as it provides explicit, deterministic logic that can be audited line-by-line, unlike the network's distributed representations.

A counterfactual explanation answers the question: "What minimal changes to the input would have resulted in a different model output?" It provides a tangible, actionable path to a desired outcome.

• Example: For a loan denial, a counterfactual might state: "Your application would have been approved if your annual income was $5,000 higher."
• Properties: Effective counterfactuals should be feasible (respect real-world constraints), proximate (require minimal change), and sparse (change few features).

This technique is crucial for understanding decision boundaries and for providing recourse in high-stakes domains like finance and hiring.

Concept Activation Vectors (CAVs) provide a way to interpret the internal neurons or layers of a neural network in terms of human-understandable concepts.

• Method: A linear classifier is trained to distinguish between activations generated by data containing a concept (e.g., 'stripes') and data without it. The vector orthogonal to the decision boundary is the CAV.
• Testing with CAVs (TCAV): Quantifies a model's sensitivity to a concept by measuring how much the concept direction influences predictions for a specific class.

This moves beyond feature attribution to answer why a model behaves a certain way, e.g., determining if a 'zebra' classifier relies on the concept of 'stripes'.

In transformer-based models (like GPT and BERT), attention mechanisms explicitly compute a weighted importance of different parts of the input sequence when producing an output. These attention weights are a form of built-in, structural explanation.

• Self-Attention: Determines the relevance of each token to every other token in the sequence.
• Cross-Attention: In encoder-decoder models, determines which parts of the input the decoder focuses on to generate each output token.

Visualizing attention maps can reveal if a model is focusing on semantically relevant tokens (e.g., a question word when generating an answer) or spurious correlations, directly informing model debugging and trust.

The broader field encompassing techniques to make black-box models understandable. While XAI is often model-agnostic, interpretability can be intrinsic to model design (e.g., decision trees). Core methods include:

• Feature Importance: Quantifying each input's contribution to a prediction.
• Saliency Maps: Visualizing which parts of an input (e.g., pixels in an image) most influenced the output.
• Local Interpretable Model-agnostic Explanations (LIME): Approximating a complex model locally with a simple, interpretable one.
• SHapley Additive exPlanations (SHAP): Using cooperative game theory to assign a consistent value to each feature's contribution.

A mathematical approach to proving that an AI system's behavior satisfies specified safety properties under all possible inputs within a defined domain. Unlike testing, which samples behavior, formal verification provides guarantees. In safety-critical simulations, it is used to verify that a trained policy will never enter a predefined unsafe state. Techniques include:

• Satisfiability Modulo Theories (SMT) Solvers: Checking logical formulas.
• Reachability Analysis: Computing all possible states a system can reach.
• Neural Network Verification: A growing subfield focused on proving properties of DNNs, despite their nonlinearity.

The process of measuring and communicating the confidence or reliability of a model's predictions. For safe sim-to-real transfer, distinguishing between aleatoric uncertainty (inherent noise in the data) and epistemic uncertainty (model ignorance due to lack of data) is critical. High epistemic uncertainty often signals an Out-of-Distribution (OOD) input. Methods include:

• Bayesian Neural Networks: Represent weights as probability distributions.
• Monte Carlo Dropout: Using dropout at inference to approximate Bayesian inference.
• Ensemble Methods: Using predictions from multiple models to estimate variance.

The foundational mathematical framework for Safe Reinforcement Learning. A CMDP extends a standard MDP by adding cost functions. The learning objective shifts from simply maximizing cumulative reward to maximizing reward while ensuring cumulative costs remain below a safety threshold. This formalizes the trade-off between performance and safety. It is the basis for algorithms that learn policies which are provably safe with respect to defined constraints, a prerequisite for deploying simulation-trained agents in the physical world.

A real-time safety layer that observes a deployed system's state and predictions to detect imminent constraint violations or unsafe behavior. For an XAI system, the monitor might check if an explanation's confidence is below a threshold or if the decision rationale violates a safety rule. Techniques include:

• Predictive Safety Filters: Using a learned or analytical model to predict future state and override unsafe actions.
• Control Barrier Functions (CBFs): Mathematical functions that guarantee safe set invariance by filtering control inputs.
• Anomaly Detection: Flagging states or explanations that deviate significantly from trained behavior.

A specific XAI technique that answers the question: "What minimal change to the input would have resulted in a different (often more desirable) output?" For safety analysis, this is invaluable. Example: "The autonomous vehicle classified this object as safe. To be classified as a hazard, the object would need to be 0.5 meters closer." This provides actionable insight for debugging and improving systems. It helps identify the precise boundary between safe and unsafe operational domains learned by a model.