Concept Importance

Concept importance can be calculated at two distinct scopes:

• Global Importance: Ranks concepts based on their average influence across an entire dataset or class, identifying the model's overall learned biases and priorities.
• Local Importance: Scores concepts for a single specific prediction, explaining why an individual input was classified a certain way.
• This distinction is critical for differentiating between systemic model behavior and instance-specific reasoning.

In the TCAV framework, concept importance is mathematically grounded in the directional derivative. The sensitivity of a logit to a concept is calculated as: S_{C,k,l}(x) = ∇h_{l,k}(f_l(x)) · v_C where h_{l,k} is the logit for class k, f_l(x) is the activation at layer l, and v_C is the Concept Activation Vector. A high absolute value indicates the concept strongly sways the prediction.

Raw sensitivity scores are not inherently meaningful. The TCAV score (Conceptual Sensitivity) validates importance by comparing concept sensitivities against random baselines:

• A two-sided t-test determines if the concept's sensitivity distribution is significantly different from random vector sensitivities.
• Only concepts passing this statistical significance test are considered genuinely important.
• This prevents spurious correlations in the activation space from being misinterpreted as meaningful concepts.

ConceptSHAP applies Shapley values from cooperative game theory to concept importance. This method:

• Treats each concept as a player in a coalition.
• Fairly distributes the prediction credit among concepts by evaluating all possible combinations.
• Satisfies key axioms: Efficiency (scores sum to the prediction difference from a baseline), Symmetry, and Linearity.
• Provides a theoretically robust alternative to gradient-based importance scores.

A set of concepts is only useful if it fully explains the model's behavior. Key evaluation metrics include:

• Concept Completeness Score: Measures how much of the model's predictive power can be recovered using only the identified concepts as features.
• Sufficiency: Tests if the concept scores alone are enough to accurately mimic the original model's output.
• Low completeness indicates that important latent concepts remain undiscovered, driving further concept discovery efforts.

Correlation does not imply causation in activation spaces. Concept intervention is the gold standard for validating importance:

• Directly manipulate the activation vector by adding or subtracting the concept vector v_C.
• Observe the causal change in the model's output logit.
• A large, consistent shift confirms that the concept is not just correlated but causally influences the decision.
• This technique is essential for rigorous model auditing and safety verification.

Concept importance reveals whether protected attributes like race or gender implicitly influence credit decisions, even when those features are excluded from training data. By measuring sensitivity to a gender concept vector derived from unrelated text corpora, auditors can detect proxy discrimination.

• Quantifies the directional derivative of approval probability toward sensitive concepts
• Uses statistical significance testing with random vectors to rule out spurious correlations
• Generates compliance reports showing concept-level influence scores for regulatory review

Radiologists validate AI-assisted diagnoses by inspecting which clinical concepts drove a prediction. A pneumonia classifier might show high importance for consolidation and pleural effusion concepts, confirming alignment with medical knowledge.

• Maps model reasoning to concepts from a concept bank of radiological findings
• Flags cases where importance scores contradict established diagnostic criteria
• Enables concept intervention to test counterfactual scenarios during review

When a perception model misclassifies a scene, concept importance pinpoints which visual abstractions contributed to the error. Engineers trace failures to specific concepts like lane markings, occlusion boundaries, or pedestrian pose rather than raw pixels.

• Uses Concept Relevance Propagation (CRP) to decompose decisions through latent space
• Localizes concept sensitivity to specific network layers for targeted retraining
• Accelerates root-cause analysis from days to hours in safety-critical systems

Social platforms justify content removal decisions by surfacing which policy concepts triggered a violation flag. A hate speech classifier might cite high importance for dehumanizing language and targeted slurs, providing explainable enforcement.

• Aligns moderation actions with specific clauses in community guidelines
• Supports automated rationale generation for user-facing explanations
• Enables appeals based on concept-level disagreement rather than opaque model outputs

Pharmaceutical researchers rank molecular concepts like hydrogen bond donors or aromatic ring count by their importance to a toxicity prediction. This guides medicinal chemists toward structural modifications that preserve efficacy while reducing adverse effects.

• Applies ConceptSHAP for game-theoretic attribution of molecular properties
• Integrates with explainable graph neural networks operating on molecular graphs
• Prioritizes synthetic targets based on concept-level structure-activity relationships

Fraud analysts use concept importance to understand why a transaction was flagged. Instead of raw features like transaction amount or time of day, they see influence from higher-level concepts such as velocity anomaly or geographic inconsistency.

• Bridges the gap between low-level features and investigator intuition
• Supports concept completeness scoring to measure explanation fidelity
• Enables threshold tuning based on acceptable concept-level false positive rates

A technique that quantifies concept sensitivity by computing the directional derivative of a class prediction toward a CAV. TCAV produces a score indicating what fraction of a class's predictions are positively influenced by the concept. Statistical significance testing using two-sided t-tests against random vectors ensures the concept is not an artifact.

The process of assigning relevance scores to high-level concepts for a specific prediction. Unlike feature attribution which operates on pixels or tokens, concept attribution explains decisions in terms of semantic abstractions. Methods include:

• ConceptSHAP: Game-theoretic Shapley values for concepts
• Concept Relevance Propagation (CRP): Decomposing decisions through latent concept flows
• Directional derivative-based scoring from TCAV

The automated identification of meaningful directions in activation space without pre-defining concepts. Key approaches:

• Automatic Concept Extraction (ACE): Clusters image patches by activation patterns and validates with TCAV
• Matrix factorization on activation datasets to find principal concept directions
• Clustering-based methods that group co-activating neurons into concept candidates

An inherently interpretable architecture that first predicts predefined human-understandable concepts from inputs, then uses only those concept scores for final predictions. This forces the model to reason through explicit concepts, making concept importance directly observable. Variants include hybrid CBMs that balance concept-based and unrestricted reasoning paths.

Techniques for causally testing concept influence by manipulating activations:

• Concept Intervention: Directly modifying activations along a concept vector during inference to observe output changes
• Concept Erasure: Projecting activations onto a subspace orthogonal to a concept vector to remove sensitive information
• Concept Subspace Projection: Decomposing activations into concept-aligned and orthogonal components for analysis