Testing with CAVs (TCAV)

TCAV measures a model's sensitivity to a concept by computing the directional derivative of a class prediction score along the Concept Activation Vector (CAV). This scalar value represents how much the prediction changes as the input's activations are infinitesimally perturbed toward the concept. A high sensitivity score indicates the concept strongly influences the model's decision for that class.

To ensure a concept is not an artifact, TCAV employs rigorous statistical significance testing. It performs a two-sided t-test comparing the sensitivity scores obtained from the true CAV against a null distribution of scores from random direction vectors. A concept is only considered valid if it passes this test, typically at a threshold like p < 0.01, confirming the model's genuine reliance on the concept.

Unlike saliency maps that highlight raw pixels, TCAV operates on concept activation vectors in the model's latent space. A CAV is derived by training a linear classifier to separate a set of positive examples of a concept from random negative examples. This allows users to define concepts in their own vocabulary—like 'stripes,' 'wheels,' or 'texture'—and test if the model has learned them.

TCAV provides both global and local insights. The TCAV score (the fraction of inputs where the concept positively influences the class) offers a global view of concept importance across a dataset. For local explanations, sensitivity scores can be computed for individual predictions, revealing which concepts drove a specific decision. This dual granularity supports both model auditing and individual prediction debugging.

By extracting CAVs from different layers of a neural network, TCAV enables layer-wise concept analysis. This reveals the hierarchical abstraction process: lower layers may encode basic concepts like edges or colors, while deeper layers encode complex, composite concepts. This helps researchers understand how conceptual understanding emerges through the network's depth.

The Automatic Concept Extraction (ACE) algorithm extends TCAV by automatically discovering concepts. ACE clusters image patches that activate similar spatial patterns, generates visual concepts from these clusters, and then uses TCAV to test their significance. This removes the need for manual concept definition, enabling scalable, unsupervised auditing of a model's internal representations.