Concept Subspace Projection

The fundamental operation of splitting an activation vector v into two orthogonal components: v_parallel (the projection onto the concept vector c) and v_orthogonal (the residual). This is computed as:

• v_parallel = (v · c / ||c||²) * c
• v_orthogonal = v - v_parallel This decomposition allows analysts to isolate the exact contribution of a concept to a network's internal state.

A defining property of subspace projection is that the parallel and orthogonal components are guaranteed to be orthogonal, meaning their dot product is zero: v_parallel · v_orthogonal = 0. This ensures a clean, lossless separation of information. The concept-aligned component captures all variance in the direction of c, while the residual contains everything else, enabling precise concept erasure without affecting unrelated information.

The projection operation can be expressed as a linear transformation using a projection matrix P:

• P = c cᵀ / (cᵀ c) for a single concept vector
• v_parallel = P v This matrix formulation is computationally efficient and generalizes naturally to projecting onto multi-dimensional concept subspaces spanned by multiple concept vectors, enabling analysis of composite semantic directions.

The scalar coefficient (v · c) / ||c||² represents the activation strength of the concept in the input. This score quantifies how strongly the concept is encoded at that layer. When tracked across a dataset, these scores form a distribution that can be analyzed for:

• Concept prevalence in specific classes
• Concept drift between training and production data
• Spurious correlation detection when concepts appear unexpectedly

Subspace projection enables concept intervention by modifying the activation vector and re-injecting it into the forward pass:

• Concept Removal: Set v_new = v_orthogonal to erase a concept
• Concept Amplification: Set v_new = v + α * v_parallel to boost a concept
• Concept Replacement: Project onto a different concept vector entirely This provides a causal test of whether a concept drives downstream predictions.

The decomposition enables precise measurement of geometric relationships:

• Concept Alignment: The cosine similarity cos(θ) = (v · c) / (||v|| ||c||) measures how aligned an activation is with a concept
• Projection Residual Norm: ||v_orthogonal|| quantifies how much information is unexplained by the concept
• Subspace Overlap: The principal angles between two concept subspaces measure their semantic similarity