Concept Activation Vector (CAV)

The foundational assumption of a CAV is that a high-level concept can be represented as a linear direction in a layer's activation space. A CAV is derived by training a linear classifier to distinguish between a set of example inputs that represent the concept and a set of random counterexamples. The resulting vector, orthogonal to the decision boundary, points in the direction of increasing concept presence. This linearity is crucial for interpretability, as it allows for simple vector arithmetic and directional derivative calculations.

• Input: Concept examples vs. random examples.
• Process: Train a linear SVM or logistic regression model.
• Output: A vector (the CAV) normal to the classifier's decision boundary.

A raw CAV is just a vector; its explanatory power is validated through Testing with CAVs (TCAV) . TCAV quantifies a model's sensitivity to a concept by measuring the directional derivative of a class prediction score along the CAV's direction. To ensure this sensitivity is not a random artifact, TCAV performs a two-sided t-test comparing the sensitivity scores from the real CAV against scores from thousands of random vectors. A concept is only considered meaningful if it passes this statistical significance test, providing a rigorous, user-defined hypothesis test for model behavior.

While CAVs are often user-defined, they can also be automatically discovered. Automatic Concept Extraction (ACE) is an algorithm that segments input images into diverse patches, clusters them based on their activation patterns in a network, and then tests each cluster's significance using TCAV. This process surfaces the concepts a model has learned organically, which may include both expected features and surprising, spurious correlations. This capability transforms CAVs from a purely hypothesis-testing tool into a powerful model auditing and debugging instrument.

CAVs are not just for passive observation; they are tools for active intervention. Concept Intervention involves directly editing a model's activations by adding or subtracting a scaled CAV during inference to amplify or suppress a concept, enabling causal tests of its influence. For a more permanent solution, Concept Erasure projects activations onto a subspace orthogonal to a CAV, effectively removing all linear information about a sensitive concept (e.g., a protected attribute) from the network's representation, which is a critical technique for fairness and privacy.

CAVs provide a unified framework for both global model understanding and local prediction explanation. Globally, the TCAV score reveals the overall importance of a concept for an entire class. Locally, the Concept Sensitivity Map can be generated by computing the directional derivative of a single prediction with respect to a CAV, highlighting the specific input regions that most strongly encode the concept for that particular instance. This dual granularity makes CAVs versatile for both high-level model auditing and fine-grained debugging of individual decisions.

The core value of a CAV is its alignment with human-understandable concepts like 'stripes,' 'wheels,' or 'doctor's coat,' rather than uninterpretable pixel patterns. This is achieved by defining the concept through a curated set of exemplar data. The quality and coherence of this dataset are paramount; a poorly defined concept will yield a meaningless CAV. This direct mapping between a semantic idea and a mathematical vector allows engineers and domain experts to formulate precise, testable hypotheses about a model's internal reasoning, bridging the gap between high-level domain knowledge and low-level network activations.

The primary application of CAVs that produces a quantitative metric called the TCAV score. This score represents the fraction of a target class's inputs whose predictions are positively influenced by a concept.

• Computes the directional derivative of logit values toward the CAV
• Uses a two-sided t-test against random vectors to establish statistical significance
• Outputs a score between 0 and 1 indicating concept sensitivity
• Enables hypothesis testing like "how sensitive is 'zebra' prediction to the 'stripes' concept?"

An inherently interpretable architecture that forces the model to predict human-specified concepts as an intermediate step before making the final classification.

• First layer predicts concept scores (e.g., 'wing color', 'beak shape')
• Final layer uses only concept scores as input for the task prediction
• Allows direct intervention: correcting a mispredicted concept immediately fixes downstream errors
• Trade-off: requires fully annotated concept labels during training

An algorithm that discovers concepts automatically rather than requiring manual definition. ACE clusters image segments that produce similar activation patterns across a dataset.

• Segments images into multi-resolution patches using superpixel methods
• Clusters patches by their activation similarity in a target layer
• Filters out outliers and tests remaining clusters with TCAV for significance
• Returns a set of discovered concepts with human-interpretable visual examples

Applies Shapley value theory to concept-based explanations, quantifying each concept's marginal contribution to a prediction.

• Treats concepts as cooperative game players
• Computes the average change in prediction when a concept is added across all possible concept coalitions
• Satisfies axioms of efficiency, symmetry, dummy, and additivity
• Provides a theoretically grounded alternative to sensitivity-based concept importance

A technique for removing sensitive or unwanted concepts from a model's latent representations through orthogonal projection.

• Identifies the concept direction via a linear probe or CAV
• Projects activations onto the nullspace of the concept vector
• Removes information like protected attributes (race, gender) while preserving task-relevant features
• Used for fairness interventions and controlled feature removal experiments

A module that replaces batch normalization layers to align latent space axes with predefined concepts, producing a disentangled representation.

• Whitens activations to remove correlations
• Rotates the whitened space so each axis corresponds to a known concept
• The activation magnitude along an axis directly indicates concept presence
• Enables interpretability without sacrificing model performance