Activation Engineering

The core mechanism involves computing a difference vector between contrasting activation distributions (e.g., 'love' vs. 'hate') and adding this vector, scaled by a coefficient, to the model's residual stream at specific layers during the forward pass. This linearly shifts the model's internal representations toward the target behavior without modifying weights.

• The steering vector is derived from contrastive pairs of prompts or completions
• Addition occurs at one or more target layers, typically mid-to-late transformer blocks
• A multiplier coefficient controls intervention strength, balancing efficacy against output degradation

Steering vectors are constructed by collecting residual stream activations from two sets of forward passes: one representing the desired behavior and one representing its opposite or a neutral baseline. The mean difference between these activation distributions forms the steering direction.

• Requires a curated dataset of paired prompts that isolate the target concept
• Activations are typically extracted at the last token position of the prompt
• The resulting vector captures the linear direction in representation space corresponding to the concept

Not all layers respond equally to steering. Effective activation engineering requires identifying which transformer layers encode the target concept most causally. Early layers process syntax, middle layers encode semantics, and late layers prepare output distributions.

• Causal tracing or patching experiments identify the optimal injection site
• Steering at multiple layers simultaneously often yields more robust control
• Injecting at the wrong layer can cause off-target effects or output degradation

Unlike prompt engineering, which relies on natural language instructions, activation engineering operates below the text interface by directly manipulating the model's representational geometry. This enables control over behaviors that are difficult to articulate in prompts.

• Can induce stylistic shifts, emotional tone, or truthfulness biases
• Effective for jailbreak defense by steering away from harmful response directions
• Enables concept erasure by subtracting rather than adding activation vectors

Activation engineering is theoretically grounded in the Linear Representation Hypothesis, which posits that high-level concepts are encoded as linear directions in a model's activation space. Steering vectors exploit this property by moving activations along these interpretable axes.

• Validated by the success of linear probes in recovering concepts from activations
• Implies that complex behaviors can be decomposed into independent feature directions
• Connects to superposition research, where features are compressed into lower-dimensional spaces

The efficacy of a steering vector is measured by its ability to shift model outputs on held-out test prompts while maintaining general capabilities. Metrics include target behavior adherence, output fluency, and minimal degradation on unrelated tasks.

• A/B testing compares steered vs. unsteered completions on diverse prompts
• Capability benchmarks (MMLU, HellaSwag) verify general performance is preserved
• Specificity tests confirm the intervention does not inadvertently alter unrelated behaviors