Concept Vector Arithmetic

The foundational principle that concepts encoded as vectors can be meaningfully combined using standard linear algebra operations. Because concept vectors capture directional semantics in activation space, vector addition produces a new direction representing the composite concept. For example, adding a vector for 'stripes' to a vector for 'horse' yields a direction associated with 'zebra'. This property emerges from the linear representation hypothesis, which posits that high-level features are encoded as linear directions in a model's latent space. The resulting composite vector can be used to steer generation or probe model understanding without retraining.

Subtracting one concept vector from another isolates differentiating semantic attributes, enabling analogical reasoning in activation space. The classic example follows the form: king - man + woman ≈ queen. This operation removes the 'male' semantic component from 'king' and adds the 'female' component, arriving at the vector for 'queen'. This demonstrates that concept vectors encode disentangled attributes that can be independently manipulated. In practice, subtraction is used for concept erasure—projecting activations orthogonal to a sensitive attribute vector to remove unwanted bias from representations.

Multiplying a concept vector by a scalar coefficient and adding it to an activation controls the intensity of that concept's influence on model behavior. A positive coefficient amplifies the concept's presence; a negative coefficient suppresses it. This technique, known as activation steering, allows precise, continuous control over generated outputs. For instance, scaling a 'formality' vector can shift language model outputs from casual to professional register without altering the underlying prompt. The linearity of this operation makes it computationally efficient and compatible with real-time inference pipelines.

Any activation vector can be decomposed into components that are parallel and orthogonal to a concept direction using vector projection. The parallel component represents the degree to which the concept is present; the orthogonal component captures all other information. This decomposition enables:

• Concept removal: discarding the parallel component to erase a concept
• Concept measurement: computing the scalar projection length as a quantitative sensitivity score
• Disentanglement analysis: verifying that manipulating one concept does not corrupt orthogonal semantic dimensions This mathematical property underpins techniques like Concept Erasure and TCAV sensitivity calculations.

Linear interpolation between two concept vectors creates a continuous spectrum of intermediate semantic states. By computing v_interp = (1-α)v_concept_A + αv_concept_B for α in [0,1], models can generate outputs that smoothly transition between concepts. This enables:

• Morphological blending: generating hybrid visual concepts like a 'tiger-lion' blend
• Style transfer: interpolating between artistic style vectors
• Attribute dials: creating user-facing sliders that adjust specific semantic dimensions in real time The smoothness of the interpolation validates that concepts form a continuous semantic manifold rather than discrete clusters.

The validity of concept vector arithmetic is empirically verified through counterfactual intervention. After performing an operation—such as adding a 'smiling' vector to a neutral face encoding—the resulting activation is decoded or passed through the remainder of the network. The output is inspected to confirm that the intended semantic change occurred without unintended side effects. Statistical tests compare the effect against random direction vectors to ensure the result is not an artifact. This validation loop is essential for building trustworthy concept-based explanations and ensuring that discovered concept directions are causally meaningful rather than correlational.