TCAV (Testing with Concept Activation Vectors)

The fundamental building block of TCAV. A CAV is a vector in the direction of a high-level concept (e.g., 'stripes', 'wheels', 'doctor') within a neural network's activation space.

• Derivation: CAVs are learned by training a linear classifier to distinguish between example inputs representing the concept and random counter-examples.
• Representation: The vector orthogonal to the classifier's decision boundary defines the concept's direction in the latent space.
• Purpose: CAVs provide a mathematical bridge between human-interpretable ideas and the network's internal representations.

TCAV measures how sensitive a model's prediction for a specific class is to a given concept. This is quantified by the TCAV score.

• Mechanism: The TCAV score is the fraction of inputs for which the directional derivative of the class logit with respect to the CAV is positive.
• Interpretation: A high TCAV score (e.g., 0.95 for 'stripes' on 'zebra' class) indicates the model heavily relies on that concept for classification.
• Granularity: This allows testing for the presence of spurious correlations, such as a 'doctor' concept being sensitive to 'male' gender.

To ensure that a TCAV score is not a random artifact, the method employs rigorous statistical significance testing.

• Process: The TCAV score is computed multiple times using random, non-concept 'negative' sets instead of the true concept examples.
• Validation: A two-tailed t-test is performed to determine if the score from the true concept is significantly different from the distribution of random scores.
• Output: Only concepts that pass this significance test are reported as valid explanations, ensuring robustness and reliability.

TCAV's power lies in its flexibility; concepts are defined by the user, not the model's architecture. This enables truly domain-specific interpretability.

• Input: A user simply provides a set of positive examples (e.g., 20-50 images of 'curved lines') and a set of negative examples.
• No Retraining: The underlying model remains a black box. CAVs are learned post-hoc on the model's activations.
• Application: This allows a medical regulator to test if a diagnostic model is sensitive to a clinically irrelevant concept like 'scanning artifact'.

A comparative metric that ranks the importance of multiple concepts for a single model prediction, answering 'which concept mattered most?'.

• Calculation: RCI is derived by comparing the magnitude of the directional derivative for each CAV.
• Utility: For a 'sports car' prediction, RCI might reveal that the 'aerodynamic shape' concept is far more influential than the 'red color' concept.
• Insight: This provides a nuanced, multi-faceted explanation of a model's reasoning process, moving beyond single-feature attribution.

TCAV is applicable to any differentiable model and can probe the internal logic at different levels of abstraction.

• Model-Agnostic: It works on CNNs, Transformers, and other architectures without requiring architectural changes.
• Layer Probing: CAVs can be trained on activations from any layer, from low-level texture layers to high-level semantic layers.
• Hierarchical Insight: This reveals how concept understanding evolves through the network, showing that early layers might detect 'edges' while later layers encode 'facial expression'.

TCAV allows clinicians to verify if a diagnostic model relies on clinically relevant concepts rather than spurious correlations.

• Concept Testing: Quantifies sensitivity to 'tissue density' or 'lesion margin irregularity' rather than scanner metadata.
• Regulatory Alignment: Provides evidence for FDA SaMD (Software as a Medical Device) submission by proving the model's logic aligns with radiological standards.
• Bias Detection: Reveals if a dermatology classifier is using 'bandage presence' or 'ruler marks' as a proxy for malignancy.

TCAV serves as a technical bridge to comply with NYC Local Law 144 and the EU AI Act by testing for protected attribute leakage.

• Concept Sensitivity Analysis: Measures the model's directional sensitivity toward concepts like 'gender presentation' or 'ethnic hair texture' without needing access to demographic labels.
• Disparate Impact Testing: Auditors can prove a hiring model rejects candidates based on 'unrelated hobby keywords' rather than 'relevant experience'.
• Causal Concept Probing: Distinguishes between legitimate business necessity and illegal discriminatory proxies.

Engineers use TCAV to debug perception stacks in end-to-end driving models to ensure safety-critical object recognition.

• Traffic Light Concepts: Validates that the model's braking logic is triggered by the concept of 'red light' rather than 'reflection on a wet road'.
• Pedestrian Sensitivity: Quantifies how much the concept of 'human silhouette' influences the steering vector compared to 'background foliage'.
• Edge Case Analysis: Tests if rare concepts like 'jaywalking child' or 'occluded stop sign' are properly represented in the model's latent space.

TCAV helps safety teams understand and steer the internal representations of text-to-image diffusion models to prevent harmful outputs.

• Concept Erasure: Identifies the activation vector for 'violence' or 'explicit nudity' to apply targeted fine-tuning or latent space editing.
• Style Decomposition: Separates the concept of 'artistic style' from 'copyrighted character likeness' to mitigate intellectual property infringement.
• Prompt Interrogation: Reveals if a model is ignoring negative prompts by testing if the concept vector for 'bypass' is still active.

Quantitative analysts apply TCAV to satisfy SR 11-7/OCC 2011-12 guidance on model risk management by opening the black box of deep learning trading models.

• Factor Attribution: Tests if a volatility prediction model is sensitive to the high-level concept of 'geopolitical risk' or merely 'short-term momentum'.
• Regime Shift Detection: Monitors if the model's concept activation vectors for 'liquidity crisis' drift over time, indicating model decay.
• Adversarial Robustness: Ensures that the concept of 'creditworthiness' is not easily flipped by adversarial perturbations in application data.

Researchers use TCAV to extract scientific insights from models trained on complex physical or biological data.

• Drug Discovery: Tests if a molecular property predictor is sensitive to the concept of 'hydrogen bond donor' or 'aromatic ring count'.
• Genomics: Validates if a splicing predictor relies on the biological concept of 'splice site motif' rather than sequencing artifacts.
• Climate Science: Probes a weather model to confirm it has learned the physical concept of 'convective available potential energy' for storm prediction.