Concept Embedding

Measures how well the representations of a single concept are clustered together and separated from other concepts in the activation space. High purity indicates internal consistency.

• A pure concept embedding has low intra-class variance
• Evaluated using silhouette scores or nearest-neighbor consistency
• Essential for reliable concept-based explanations
• Example: The embedding for 'stripes' should tightly cluster all striped patterns while remaining distant from 'polka dots'

The degree to which a linear or non-linear classifier can distinguish between the activation patterns of two different concepts. High separability confirms that concepts are distinctly encoded in the network.

• A linear SVM is typically used as the probe classifier
• Measured via ROC-AUC or classification accuracy
• Poor separability suggests overlapping or entangled concept representations
• Example: 'Fur' and 'texture' may have low separability if the model conflates them

Evaluates how sufficient a set of discovered or defined concepts is for explaining a model's full behavior on a given task. A complete concept set can reconstruct or approximate the model's decision boundary.

• Quantified by the fidelity of a concept bottleneck model
• Low completeness means important concepts are missing from the explanation
• Guides iterative concept discovery efforts
• Example: Explaining 'dog breed classification' may require concepts for ear shape, snout length, and coat pattern

Quantifies how much a model's prediction changes when activations are perturbed along the concept vector direction. This is the foundation of TCAV (Testing with CAVs).

• Computed as the directional derivative of the class score
• High sensitivity indicates the concept strongly influences predictions
• Statistical significance is validated via two-sided t-tests against random vectors
• Example: A 'doctor' classifier should show high sensitivity to the 'stethoscope' concept

Measures the causal effect that manipulating a concept's activation value has on the model's final output. Unlike sensitivity, influence is estimated through direct intervention.

• Assessed by adding or subtracting the concept vector from activations
• Reveals whether a concept is merely correlated or truly causal
• Used in concept intervention experiments
• Example: Boosting the 'red color' concept should increase 'fire truck' predictions if the concept is causally relevant

Determines whether a specific neuron, channel, or direction in the activation space responds exclusively to a single concept and not to others. High selectivity enables disentangled representations.

• Measured by the activation distribution across diverse inputs
• A selective neuron fires only for its target concept
• Contrasts with polysemantic neurons that respond to multiple unrelated concepts
• Example: A 'curve detector' neuron in a vision model should not also activate for straight edges

Concept embeddings serve as the foundation for quantitative model auditing by mapping human-understandable ideas into a model's activation space. Using techniques like Testing with CAVs (TCAV) , auditors measure how sensitive a model's predictions are to specific concepts such as 'stripes' or 'wheel' without requiring labeled test data.

• Define a concept using a small set of exemplar images or text
• Train a linear classifier to separate concept activations from random counterexamples
• Compute the directional derivative of the target class toward the concept vector
• Apply statistical significance testing to validate findings

This approach allows compliance officers to verify that a loan approval model is not relying on prohibited concepts like race or gender, even when those attributes are not explicitly input features.

Not all concepts are known in advance. Automatic Concept Extraction (ACE) uses concept embeddings to discover emergent abstractions that a model has learned independently. The algorithm clusters input patches that produce similar activation patterns, then tests each cluster as a candidate concept using TCAV.

• Segments images into multi-resolution patches
• Groups patches by activation similarity in a chosen bottleneck layer
• Filters out clusters that do not pass statistical significance testing
• Returns a set of discovered concepts with importance scores

This is particularly valuable for scientific discovery, where a model trained on histopathology images might reveal novel cellular morphologies that correlate with disease progression—concepts that human pathologists had not previously formalized.

Concept embeddings enable precise concept erasure—the removal of sensitive or unwanted information from a model's latent representations without full retraining. By projecting activations onto a subspace orthogonal to a protected concept vector, engineers can neutralize encoded bias.

• Define a concept vector for the sensitive attribute (e.g., gender, age)
• Compute the orthogonal projection matrix relative to that vector
• Apply the projection to activations at inference time or during fine-tuning
• Verify removal using concept sensitivity testing

This technique is used in fairness-critical systems such as resume screening tools, where the goal is to eliminate demographic encoding while preserving task-relevant information about qualifications and experience. Unlike adversarial debiasing, concept erasure provides a mathematically guaranteed removal within the projected subspace.

Concept Bottleneck Models (CBMs) embed concept embeddings directly into the model architecture, creating an inherently interpretable prediction pipeline. The model first predicts a set of predefined concept scores from the input, then uses only those concept scores—not raw features—to make the final prediction.

• Define a concept bank with labeled training data for each concept
• Train the first stage to predict concept presence from inputs
• Train the second stage to predict the target from concept scores only
• Enable human intervention by allowing operators to correct mispredicted concepts

This architecture is deployed in medical diagnosis systems, where a radiologist can inspect and override intermediate concept predictions (e.g., 'mass present,' 'calcification visible') before the final malignancy classification is computed, creating a true human-in-the-loop interpretability framework.

Concept embeddings enable causal intervention experiments that go beyond correlation to establish whether a concept genuinely influences model behavior. By directly editing activations along a concept vector direction during inference, researchers can observe the causal effect on outputs.

• Identify a concept vector in activation space
• Add or subtract a scaled version of the vector to activations
• Measure the change in the model's prediction distribution
• Compare against control interventions using random vectors

This methodology is used in AI safety research to verify that a vision model's 'wolf' classification genuinely depends on the concept of 'snout and fur' rather than spurious correlations like 'snowy background.' It provides stronger evidence than purely correlational attribution methods.

Concept embeddings support vector arithmetic operations that combine and transform semantic abstractions in mathematically meaningful ways. By adding and subtracting concept vectors in activation space, new composite concepts emerge that the model can recognize without additional training.

• Encode individual concepts as vectors (e.g., 'smiling,' 'beard')
• Perform linear combinations: v_smiling + v_beard - v_clean_shaven
• Apply the resulting vector to guide image generation or classification
• Validate the composite concept using TCAV sensitivity testing

This capability is leveraged in generative model control, where concept vectors steer diffusion models to produce images with specific attribute combinations. It also enables zero-shot concept composition for retrieval systems that must find data matching complex, previously unseen conceptual queries.

A CAV is the foundational vector in a neural network's activation space that represents a human-understandable concept. It is derived by training a linear classifier to distinguish between a set of exemplar images for a concept and random counterexamples. The resulting vector normal to the decision boundary defines the direction of that concept, enabling quantitative testing of its influence on the model's predictions.

TCAV is the technique used to quantify a model's sensitivity to a user-defined concept. It works by measuring the directional derivative of a class prediction score towards the CAV. The core output, the TCAV score, represents the fraction of inputs for which the model's prediction for a class increases as the inputs are infinitesimally perturbed along the concept vector direction.

A CBM is an inherently interpretable architecture that forces the model to predict a set of predefined human-understandable concepts from the input first, and then uses only those concept scores to make the final prediction. This creates a strict information bottleneck, ensuring the reasoning process is transparent and can be directly intervened upon by editing the predicted concept values.

ConceptSHAP applies the game-theoretic Shapley values framework to concept-based explanations. It quantifies the importance of each individual concept for a specific model prediction by fairly distributing the credit among all concepts. This provides a theoretically sound, additive attribution score that reflects the marginal contribution of a concept when considered in different coalitions with other concepts.

Concept erasure is a technique for removing a specific, often sensitive, concept's information from a model's latent representation. This is achieved by projecting the activation vectors onto a subspace orthogonal to the concept vector. It is a critical tool for mitigating unwanted bias and ensuring that protected attributes do not influence downstream decision-making.

ACE is an algorithm for automated concept discovery that does not require pre-defined concept labels. It works by clustering image patches that activate similar spatial patterns in a network, treating each cluster as a candidate concept. The significance of these discovered concepts is then validated using TCAV, allowing researchers to audit a model for unexpected or spurious learned abstractions.