Concept-based Explanations

TCAV quantifies the influence of a user-defined concept (e.g., 'stripes', 'medical condition') on a model's predictions. It works by:

• Defining a concept using a set of example images or data points.
• Training a linear classifier to separate concept examples from random examples in the model's activation space.
• The resulting direction (the Concept Activation Vector, or CAV) is used to compute a directional derivative, measuring the model's sensitivity to that concept for a given class.
• The output is a concept importance score, such as 'The concept of stripes is 70% responsible for the model's prediction of zebra.' This method is powerful because it tests for the presence of abstract, human-meaningful ideas within the model's internal representations.

A Concept Bottleneck Model (CBM) is an inherently interpretable architecture designed around human-defined concepts. Its prediction pipeline is explicitly structured:

1. Input Layer: Raw data (e.g., an image).
2. Concept Layer: The model predicts the presence or absence of a set of pre-defined concepts (e.g., 'has wings', 'is red', 'made of metal').
3. Prediction Layer: The final class prediction is made solely based on these predicted concept scores.

This design provides a self-explaining mechanism: you can audit which concepts were activated and see exactly how they led to the final prediction. For example, a model predicting 'airplane' would first detect concepts like 'has wings', 'has a fuselage', and 'has a tail', providing a transparent reasoning chain.

ConceptSHAP extends the SHAP (SHapley Additive exPlanations) framework from individual feature attribution to concept attribution. Instead of assigning importance to raw pixels or tokens, it attributes the prediction to high-level concepts.

• Concepts as 'Players': Each human-defined concept is treated as a 'player' in the cooperative game theory framework of SHAP.
• Computing Concept Importance: The method evaluates the model's output with and without the 'presence' of each concept (often approximated via concept classifiers or embeddings).
• The result is a Shapley value for each concept, indicating its average marginal contribution to the prediction across all possible combinations of concepts. This provides a rigorous, game-theoretically sound decomposition of a prediction into concept contributions, answering 'How important was the concept of rust to the model's prediction of corrosion?'

Automatic Concept Extraction (ACE) is an unsupervised method that discovers the concepts a model has learned internally, without requiring human pre-definition.

• Segment and Cluster: For a set of images, it segments them into patches and collects the model's internal activations for each patch.
• Cluster Activations: These activation vectors are clustered. Each cluster represents a recurring pattern the model detects.
• Concept Prototype: The image patches whose activations are closest to the cluster center are retrieved as visual prototypes for the discovered concept.
• For example, applying ACE to a medical imaging model might automatically discover concepts like 'textured tissue', 'circular structure', or 'linear boundary' that the model uses for diagnosis. This is critical for exploratory model auditing to uncover unknown learned biases or features.

This method generates counterfactual explanations in the space of concepts rather than raw features. Instead of saying 'change pixel X to get a different prediction,' it answers: 'What high-level concept would need to change?'

• Process: Given an input and its prediction, the system identifies the most influential concepts. It then generates a new, similar input where the state of one or more key concepts is altered.
• Example: For a loan denial prediction, a feature-based counterfactual might suggest changing 'age=45' to 'age=35'. A concept-based counterfactual would suggest 'increase the concept of income stability while keeping age the same.'
• This approach produces more actionable and semantically meaningful explanations for end-users, as it operates on the same abstract level as human reasoning.

Prototypical Part Network (ProtoPNet) is a deep learning architecture that incorporates concept-based reasoning directly into its classification mechanism.

• Learning Prototypes: The network learns a set of prototypical parts during training (e.g., a specific pattern of a bird's wing, a type of wheel). These are stored in a prototype layer.
• Similarity Comparison: For a new input, the network finds parts of the image that are similar to its learned prototypes.
• Explainable-by-Design: The final classification is based on a weighted combination of these prototype similarities. The explanation is visual and conceptual: 'This is a Great Horned Owl because it contains these prototypical parts (shown) that look like the learned prototypes for that class.'
• Variants like ProtoTree and ProtoPFormer extend this idea to create globally consistent, tree-based or transformer-based interpretable models.

TCAV is a direct precursor and methodological foundation for concept-based explanations. It quantifies the influence of user-defined, high-level concepts (e.g., 'stripes', 'medical condition') on a model's predictions using directional derivatives in the model's activation space. Unlike post-hoc feature attribution, TCAV requires pre-defining concepts and measuring their sensitivity for entire classes of inputs.

• Key Mechanism: Learns a Concept Activation Vector (CAV)—a direction in a neural network's latent space corresponding to a human-interpretable concept.
• Output: Provides a concept sensitivity score indicating how important that concept was for a specific prediction (e.g., 'the "striped" concept was 85% responsible for classifying this image as a zebra').

A faithfulness score is a core quantitative metric for validating any explanation, including concept-based ones. It measures how accurately the explanation reflects the true reasoning process of the underlying model. For concept-based explanations, this involves testing if the model's output changes predictably when the identified concepts are manipulated.

• Evaluation Method: Systematically perturb the input to alter or remove the identified concept and measure the corresponding change in the model's prediction. A faithful explanation will show a strong correlation between concept removal and prediction change.
• Contrast with Plausibility: Faithfulness is an intrinsic, model-centric metric, whereas plausibility is a human-centric judgment of whether the explanation makes sense.