Concept Separability

The core operational definition of concept separability is the accuracy of a linear classifier (like an SVM or logistic regression) trained to distinguish between activations from examples containing the concept and random counterexamples. A high Area Under the Curve (AUC) score indicates that the concept's activations are linearly separable from noise, validating the existence of a clear Concept Activation Vector (CAV). This property is the prerequisite for techniques like TCAV to function reliably.

Concept purity measures the internal consistency of a concept's representation. It assesses how tightly the activations for a single concept are clustered together in the activation space relative to their distance from other concepts.

• High Purity: All exemplars of 'stripes' map to a tight, distinct cluster.
• Low Purity: 'Stripes' exemplars are scattered and intermingled with 'fur' or 'texture' activations. High purity is essential for concept erasure and concept intervention to be precise without causing collateral damage to unrelated representations.

Separability is a geometric property of the activation space, whereas concept importance is a behavioral property of the model's decision function. A concept can be highly separable (forming a clear, distinct cluster) but have zero importance for a specific prediction task. Conversely, a concept with low separability (a diffuse, entangled representation) cannot be reliably attributed importance, as the model itself has not formed a coherent internal definition of it.

While linear separability is the standard for defining a CAV, many complex concepts are encoded on non-linear manifolds. In these cases, a linear probe will fail, but a kernel SVM or a small neural network probe can achieve high classification accuracy. This indicates the concept is encoded but in a warped or entangled subspace. Techniques like Concept Whitening aim to transform these non-linear manifolds into linearly separable axes to improve interpretability.

The measured separability of a concept is critically dependent on the choice of negative examples (counterexamples). Using random, unrelated images as counterexamples often yields trivially high separability. A rigorous test requires adversarial counterexamples—images that are visually or semantically similar to the concept but lack its defining characteristic (e.g., 'zebra without stripes' vs. 'stripes'). This tests for a precise, rather than a superficial, encoding.

Concept separability is not uniform across a deep network. Lower layers typically encode low-level features (edges, colors) with low semantic separability. Separability for abstract concepts (e.g., 'fairness', 'danger') generally increases in deeper, post-attention layers of a transformer or the final convolutional layers of a CNN. Tracking the separability score across layers reveals the computational phase at which a coherent concept representation emerges.