Activation Space

In a sufficiently deep activation space, human-interpretable concepts are often encoded along linear directions. A single vector can represent a concept like 'stripes' or 'smiling', and the dot product of an activation with this vector indicates the concept's presence. This property is the foundation for Concept Activation Vectors (CAVs) and enables simple linear probes to reliably extract semantic knowledge from complex, non-linear networks.

Networks face a trade-off between disentanglement and superposition. In a disentangled representation, each dimension corresponds to a single, independent generative factor. However, to represent more concepts than available dimensions, models exploit superposition by compressing multiple concepts into overlapping, nearly orthogonal directions. This compression is a key challenge for mechanistic interpretability, as concepts are not neatly separated.

High-dimensional activation vectors for real-world data do not fill the space uniformly. Instead, they lie on or near a much lower-dimensional manifold. The geometry of this manifold—its curvature, distances, and paths—encodes the semantic relationships between inputs. Interpolating between two points on the manifold generates a semantically meaningful morph, while points off the manifold correspond to nonsensical or adversarial inputs.

Individual neurons in an activation space often respond to multiple, seemingly unrelated input patterns—a phenomenon called polysemanticity. A single neuron might fire for both 'cat faces' and 'car fronts'. This implies that the meaningful unit of computation is not the single neuron but a direction in the activation space. Concept-based methods therefore analyze distributed representations rather than individual node activations.

Any activation vector can be decomposed into components that are parallel and orthogonal to a concept vector. The parallel component measures concept sensitivity, while the orthogonal component represents residual information. This mathematical operation enables concept erasure—removing sensitive attributes like protected class information by projecting activations onto the nullspace of the concept vector, effectively blinding the model to that concept.

Remarkably, activation spaces from different model architectures trained on similar tasks often converge to encode concepts in isomorphic ways. A 'dog' concept vector derived from a ResNet can be mapped to the corresponding vector in a Vision Transformer via a simple linear transformation. This universality suggests that models learn fundamental structural properties of the data distribution, not just architecture-specific artifacts.

The mathematical operation of decomposing an activation vector into components parallel and orthogonal to a concept vector. This enables precise manipulation of representations:

• Project onto concept: Isolate the concept's contribution
• Project away from concept: Perform concept erasure by removing sensitive attributes
• Decompose activations: Understand how multiple concepts combine linearly in the space

Performing linear operations on concept vectors to create semantically rich composite directions. The activation space exhibits compositional structure where:

• Addition combines concepts: king - man + woman ≈ queen
• Subtraction isolates distinguishing features
• Interpolation generates continuous concept variations This property reveals that neural networks encode meaning in a surprisingly structured, vector-based format.

The act of directly modifying activations during inference to causally test a concept's influence. By steering activations toward or away from a CAV direction, engineers can observe how outputs change. This provides causal evidence—not just correlation—that a concept drives model behavior, enabling precise debugging and controlled generation in large language models.