Concept Activation Vectors (CAV)

Unlike feature attribution methods that highlight individual pixels or input tokens, CAVs map a model's internal state to high-level concepts (e.g., 'stripes', 'doctor', 'spiculated mass'). This bridges the gap between a network's latent geometry and human reasoning, allowing domain experts to query the model using their own vocabulary. The core idea is to find a vector in the activation space that points toward a specific concept, enabling the measurement of that concept's influence on a prediction.

TCAV is the practical algorithm for applying CAVs. It quantifies a concept's importance by measuring the directional derivative of a class prediction with respect to the concept vector. Key steps include:

• Concept Dataset Curation: Collecting positive and negative examples of a concept (e.g., images with and without 'hair').
• Linear Classifier Training: Training a linear model to distinguish concept activations from random activations in a chosen layer.
• Sensitivity Calculation: Computing the TCAV score, which represents the fraction of a class's inputs that are positively influenced by the concept. This yields a quantitative, statistically testable measure of concept sensitivity.

CAVs can be trained on the activations of any hidden layer, enabling an analysis of how conceptual understanding evolves through the network's depth. Early layers might encode low-level concepts like 'texture' or 'color', while deeper layers encode abstract, class-specific concepts like 'wheel' or 'beak'. This layer-wise probing provides a powerful debugging tool for verifying if a model is learning the correct hierarchical representations at the appropriate levels of abstraction.

A key strength of the TCAV framework is its use of statistical testing to validate findings. Instead of a single sensitivity score, TCAV runs multiple training runs with random splits of concept examples. It then performs a two-sided t-test against a null hypothesis of random, meaningless vectors. This ensures that a reported concept sensitivity is a genuine, reproducible signal in the model's representation, not an artifact of the probing classifier or data selection, which is critical for regulatory-grade model auditing.

The CAV methodology is fundamentally model-agnostic and applies to any neural network with accessible activations. It is not restricted to convolutional networks or image data. The framework has been successfully extended to:

• Natural Language Processing: Identifying concepts like 'formality' or 'sentiment' in text classifiers.
• Tabular Data: Discovering high-level feature interactions in predictive models.
• Reinforcement Learning: Understanding the concepts an agent uses to make decisions. This flexibility makes CAVs a universal tool for interpretability across domains.

CAVs enable a direct, quantitative comparison of how different concepts compete or cooperate in a model's decision-making. For a 'zebra' prediction, one can measure the TCAV score for 'stripes' versus 'horse-like snout'. This relative importance analysis reveals the model's learned conceptual hierarchy and can uncover spurious correlations, such as a 'doctor' concept being overly sensitive to 'male' gender cues. This allows developers to identify and mitigate unwanted biases embedded in the model's conceptual structure.

A game-theoretic approach to explainability that assigns each input feature an importance value for a particular prediction. SHAP (SHapley Additive exPlanations) computes Shapley values from cooperative game theory, fairly distributing the prediction output among features.

• Unifies several existing explanation methods under a single theoretical framework
• Provides both local explanations (per-prediction) and global insights (aggregated across samples)
• Computationally intensive for high-dimensional biomarker data; approximations like Kernel SHAP and Tree SHAP mitigate this
• Widely used in clinical decision support to identify which biomarkers drove a specific patient's risk score

An axiomatic attribution method that satisfies desirable properties including Sensitivity and Implementation Invariance. It computes feature importance by accumulating gradients along a straight-line path from a baseline input (typically a zero-information reference) to the actual input.

• Addresses the saturation problem where standard gradients fail to highlight important features that have pushed a neuron past its activation threshold
• Requires careful selection of the baseline; in medical imaging, a black image or blurred scan is common
• Produces pixel-level or feature-level attributions that are more faithful than raw gradient saliency maps
• Foundational to understanding what a model has learned to detect in radiological scans

Gradient-weighted Class Activation Mapping produces coarse localization maps highlighting the image regions most influential for a convolutional neural network's prediction. It uses the gradients of a target class flowing into the final convolutional layer to weight activation maps.

• Generates visual explanations that can be overlaid directly on medical images for intuitive clinician review
• Does not require architectural changes or re-training, making it suitable for post-hoc analysis of deployed diagnostic models
• Variants like Grad-CAM++ improve localization for multiple instances of the same class within a single image
• Commonly used in pathology and radiology AI to verify that models focus on clinically relevant anatomical regions

Local Interpretable Model-agnostic Explanations explains individual predictions by approximating the complex model locally with a simpler, inherently interpretable model such as a linear regression or decision tree. It perturbs the input and observes how predictions change.

• Truly model-agnostic: works with any classifier, including proprietary or black-box diagnostic systems
• Produces explanations in terms of human-interpretable features, such as the presence or absence of specific biomarkers
• The definition of the local neighborhood and the interpretable feature representation critically impacts explanation quality
• Useful for generating counterfactual-like insights by showing which feature changes would alter the diagnosis

A method that answers the question: 'What is the smallest change to this patient's data that would change the diagnosis?' It identifies a minimal perturbation to input features that flips the model's prediction to a predefined alternative outcome.

• Directly actionable for clinicians: 'If this biomarker level were reduced by X, the risk classification would change'
• Must respect real-world feasibility constraints; a counterfactual suggesting an impossible physiological change is useless
• Algorithms like Growing Spheres and DiCE (Diverse Counterfactual Explanations) generate multiple diverse scenarios
• Increasingly required for FDA submissions to demonstrate that model behavior aligns with clinical reasoning

The process of estimating the confidence a model should have in its own predictions, decomposing uncertainty into aleatoric (inherent data noise) and epistemic (model ignorance) components. Critical for diagnostic safety.

• Bayesian neural networks and Monte Carlo Dropout provide principled uncertainty estimates at inference time
• Conformal prediction offers distribution-free, finite-sample coverage guarantees for prediction sets
• High epistemic uncertainty on a patient sample signals that the model is out-of-distribution and its prediction should not be trusted
• Coupled with CAV, uncertainty quantification can reveal whether a concept is reliably detected or ambiguous