Interpretability

Feature attribution methods assign an importance score to each input feature, indicating its contribution to a model's specific prediction. This is foundational for understanding which parts of the input the model relied on.

• Saliency Maps: Visual heatmaps for image models, highlighting pixels that most influenced the output.
• SHAP (SHapley Additive exPlanations): A game theory-based approach that provides a unified measure of feature importance, ensuring fair attribution.
• Integrated Gradients: Attributes the prediction to input features by accumulating gradients along a path from a baseline (e.g., a black image) to the actual input.
• Use Case: In a loan approval model, feature attribution can reveal that an applicant's credit score and debt-to-income ratio were the dominant factors in a rejection.

For transformer-based models (like most modern LLMs), attention mechanisms determine which parts of the input sequence the model "focuses on" when producing an output token. Visualizing these attention weights reveals the model's internal alignment.

• Attention Heads: A transformer has multiple attention heads, each potentially learning different linguistic relationships (e.g., subject-verb agreement, coreference).
• Attention Maps: Show connections between input and output tokens, often visualized as a matrix of scores.
• Interpretation: In machine translation, strong attention links between an English word and its French equivalent provide a transparent view of the translation alignment, a form of mechanistic interpretability.

A counterfactual explanation answers the question: "What minimal changes to the input would have led to a different model prediction?" It provides actionable, contrastive insight.

• Actionable Guidance: Instead of just explaining a 'no' decision, it suggests how to get a 'yes'. For example, "Your loan would have been approved if your annual income was $5,000 higher."
• Proximity & Sparsity: Good counterfactuals should be close to the original input (proximity) and involve changing few features (sparsity).
• Key for Recourse: This technique is central to providing algorithmic recourse, enabling individuals to understand how to achieve a desired outcome from an automated system.

Concept Activation Vectors (CAVs) are a method for testing whether a human-defined concept (e.g., 'stripes', 'medical jargon') is present in a model's internal representations and influencing its predictions.

• Process: A linear classifier is trained to distinguish between model activations for data points that contain the concept versus those that do not. The vector orthogonal to the decision boundary is the CAV.
• Testing with CAVs (TCAV): Quantifies a model's sensitivity to a concept for a given class. For example, it can measure how important the concept 'wheel' is for the model's prediction of 'car'.
• Bridging the Gap: This technique provides a direct link between high-level, human-understandable concepts and the model's low-level, abstract neuron activations.

Local Interpretable Model-agnostic Explanations (LIME) approximates a complex, black-box model locally around a specific prediction with a simple, interpretable model (like a linear regression).

• Local Fidelity: LIME creates a new dataset of perturbed samples around the instance, gets predictions from the black-box model, and trains a simple model that is faithful locally.
• Model-Agnostic: It works with any classifier or regressor (images, text, tabular data) by defining a meaningful way to perturb the input.
• Output: For an image classification, LIME might highlight a super-pixel region and state: "This image is classified as 'wolf' because of these highlighted patches," which correspond to fur texture and background snow.

Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable algorithms and circuits. It seeks to find sub-networks that implement specific, discrete functions.

• Circuits: Identifying groups of neurons (a circuit) that work together to compute a specific feature, such as detecting curves or executing an induction head in a transformer.
• Ablation Studies: Systematically disabling neurons or connections to test their causal effect on a model's behavior.
• Goal: The ultimate ambition is full interpretability, where the model's entire computation graph can be translated into human-comprehensible code. This is considered the 'gold standard' but is extremely challenging for large models.

Uncertainty Quantification (UQ) is the process of measuring and communicating the confidence or reliability of a model's predictions. It is intrinsically linked to interpretability because a model's self-reported uncertainty is a crucial piece of information for a human to correctly interpret its output.

• Types of Uncertainty: Aleatoric (noise inherent in the data) and Epistemic (uncertainty in the model's knowledge).
• Methods: Include Bayesian neural networks, Monte Carlo dropout, and ensemble techniques.
• Practical Use: In safety-critical applications (e.g., medical diagnosis), high uncertainty should flag a prediction for human review, making the system's limitations interpretable.

Formal verification for neural networks is the application of mathematical methods to prove or disprove that a model satisfies a specific, rigorous property. It provides the strongest possible form of interpretability for a guaranteed behavior within a defined input space.

• Objective: To prove statements like "For all inputs within bounds X, the model will never output class Y."
• Contrast with Heuristic Methods: Unlike feature attribution (e.g., SHAP), which gives approximate importance scores, formal verification offers deterministic proofs.
• Key Challenge: Computational complexity limits its application to small regions of input space or specific network architectures.

Out-of-Distribution (OOD) Detection is the task of identifying when input data is statistically different from the data a model was trained on. A model's inability to recognize OOD inputs is a major failure of interpretability, as it may produce high-confidence but nonsensical predictions.

• Why it's Related: A truly interpretable system should communicate when it is operating outside its domain of competence.
• Common Techniques: Leverage prediction confidence scores, Mahalanobis distance in feature space, or dedicated OOD detection heads.
• Critical for Robustness: Prevents silent failures in deployment, such as a vision model trained on cats/dogs misclassifying a car with high certainty.

Model Inversion is a privacy attack where an adversary exploits a model's outputs—often its confidence scores or logits—to reconstruct representative features of its training data. This attack highlights a tension between model interpretability and privacy: the very signals that make a model's reasoning interpretable (e.g., high confidence for a specific class) can be exploited to leak information.

• Attack Vector: Uses a model's API to iteratively query it and approximate training data, such as reconstructing a face from a facial recognition model.
• Defensive Link: Techniques that enhance privacy, like differential privacy, often work by adding noise, which can reduce the fidelity of interpretability methods.

Adversarial Robustness is a model's ability to maintain correct predictions when subjected to deliberately crafted, often imperceptible input perturbations (adversarial examples). Interpretability tools are vital for diagnosing and improving robustness.

• Diagnostic Role: Feature attribution maps (e.g., from Grad-CAM) can reveal if a model is relying on nonsensical or easily perturbed features in the input, making it vulnerable.
• Improvement Role: Understanding why a model failed on an adversarial example informs better defense design, such as adversarial training.
• Shared Goal: Both fields aim to build trustworthy models—one by making them understandable, the other by making them reliable under attack.