TCAV (Testing with Concept Activation Vectors)

The core mathematical object of TCAV. A CAV is a direction in a neural network's activation space that represents a human-defined concept (e.g., 'stripes', 'wheels', 'doctor').

• Derivation: A linear classifier is trained to distinguish between example inputs representing the concept and random counterexamples. The vector orthogonal to the decision boundary becomes the CAV.
• Role: Acts as a probe to measure how aligned the network's internal representations are with the concept.
• Key Property: CAVs are post-hoc and do not require retraining the original model.

TCAV produces a quantitative metric called the TCAV score, which measures the sensitivity of a model's prediction for a specific class to a given concept.

• Calculation: The fraction of inputs for which the directional derivative of the class logit with respect to the CAV is positive.
• Interpretation: A score of 0.8 for 'stripes' on the 'zebra' class means that for 80% of zebra images, making the representation more 'striped' increases the probability of predicting 'zebra'.
• Statistical Rigor: Scores are validated using a two-sided t-test against random vectors to ensure the sensitivity is not accidental.

Unlike many attribution methods that only work on the input layer, TCAV can probe any hidden layer of a deep neural network.

• Mechanism: CAVs are generated from the activations of the target layer, allowing analysis of how concepts are represented at different levels of abstraction.
• Practical Use: An engineer can discover that low-level concepts like 'texture' dominate in early convolutional layers, while high-level concepts like 'gender' or 'profession' only emerge in fully connected layers.
• Benefit: Enables hierarchical debugging of the feature learning process.

TCAV enables direct comparison of multiple concepts to understand which ones are most influential for a model's decision-making.

• Example: For a 'doctor' classifier, TCAV can quantify whether the model relies more on the concept of a 'stethoscope' or on spurious correlations like 'male'.
• Fairness Auditing: This is a critical tool for detecting unwanted bias. If the TCAV score for a protected attribute (e.g., 'female') is high and statistically significant, it reveals a dependency the model should not have.
• Output: Generates a ranked list of concepts by their TCAV score for a target class.

The TCAV methodology is not limited to image data. It applies to any modality where you can define a concept through a set of examples.

• Image Domain: Concepts like 'curved lines' or 'red color' using example image patches.
• Text Domain: Concepts like 'formal tone' or 'sarcasm' using example sentences or embeddings.
• Reinforcement Learning: Concepts like 'enemy proximity' or 'low health' using example state representations to explain an agent's policy.
• Requirement: Only needs access to a set of positive and negative examples for the concept.

In the context of Explainable Reinforcement Learning (XRL), TCAV is used to test whether a trained policy has learned to recognize human-relevant game concepts.

• Process: A CAV is built for a concept like 'enemy in sight' using game frames. The TCAV score then quantifies how sensitive the policy's action selection (e.g., 'attack') is to that concept.
• Debugging: If a policy has a high TCAV score for 'ally proximity' when choosing 'attack', it may have learned a dangerous, unintended behavior.
• Validation: This bridges the gap between scalar reward signals and semantic understanding of an agent's strategy.

A heatmap that highlights the regions of an input state—such as pixels in a game frame or sensor readings—that most strongly influence a reinforcement learning agent's action selection. Unlike TCAV, which tests for high-level concepts, saliency maps provide fine-grained spatial attribution at the input level.

• Computed via gradient backpropagation to the input layer
• Reveals where the agent is 'looking' when making a decision
• Commonly used with Deep Q-Networks (DQN) and policy gradient methods

A causal interpretability method that systematically removes or occludes input features to measure the resulting change in policy output. While TCAV measures sensitivity to concepts, ablation directly tests feature necessity by observing performance degradation.

• Removes one feature or group of features at a time
• Measures drop in expected return or action probability
• Provides ground-truth causal evidence, not just correlation

An explanation format that answers 'Why action A instead of action B?' by highlighting the minimal state differences that caused the policy to diverge. This complements TCAV by providing counterfactual reasoning at the action level rather than concept-level sensitivity.

• Identifies minimal sufficient causes for decision changes
• Useful for debugging near-boundary states
• Often generated via perturbation analysis or generative models

The process of breaking down a scalar reward signal into constituent sub-rewards to explain which objectives are driving an agent's behavior. While TCAV explains what concepts influence a policy, reward decomposition explains why those concepts matter in terms of the agent's optimization goals.

• Decomposes reward into semantically meaningful components
• Reveals trade-offs between competing objectives
• Enables credit assignment to specific sub-goals

A model-agnostic attribution method that computes the path integral of gradients from a baseline state to the actual input. This satisfies the completeness axiom (attributions sum to the output difference), providing a principled alternative to TCAV's directional derivative approach for feature-level explanations.

• Uses a black-to-white baseline for image inputs
• Satisfies sensitivity and implementation invariance axioms
• Applicable to any differentiable policy network

A technique for distilling a neural network policy into a structurally interpretable decision tree that mimics the original policy's input-output mapping. This provides a globally interpretable surrogate that can be audited directly, complementing TCAV's concept-level sensitivity analysis with explicit logical rules.

• Produces if-then rules for action selection
• Trades fidelity for interpretability
• Enables formal verification of extracted policy logic