Activation Steering Attack

The attack operates by adding a pre-computed steering vector to the residual stream at a specific transformer layer. Unlike prompt injection, which relies on token-level input, this method directly modifies the high-dimensional activations flowing between attention and feed-forward blocks. The injected vector is typically derived from contrastive pairs—comparing activations for a target behavior against a baseline—to isolate the direction in latent space that encodes the desired output. Because the intervention occurs after embedding but before the final unembedding projection, it can override even strongly reinforced system prompts and safety fine-tuning.

Attackers construct steering vectors by computing the difference in activations between paired forward passes:

• Positive example: A prompt that elicits the malicious target behavior
• Negative example: A neutral or refusal-generating prompt with identical structure

The resulting difference vector captures the directional encoding of the target concept. When scaled by a multiplier (often 1–5x) and added to the residual stream at inference, it shifts the model's internal representations toward the attacker's objective. This technique exploits the linear representation hypothesis—the finding that many high-level concepts are encoded as linear directions in activation space.

The efficacy of activation steering depends critically on which transformer layer receives the injection. Early layers (0–8) primarily encode syntactic and low-level features, making them poor targets for semantic manipulation. Middle layers (9–24) in large models encode high-level concepts, factual associations, and moral reasoning—these are the optimal injection points. Late layers (25–32+) refine representations for token prediction; injections here can produce incoherent or grammatically broken outputs. Attackers probe layer sensitivity by injecting at each layer and measuring the KL divergence between the steered output distribution and the target distribution.

Sophisticated attacks apply steering vectors across multiple token positions simultaneously rather than a single injection point. By patching the residual stream at every token position in a sequence—or at semantically critical tokens identified through attention analysis—the attacker achieves more robust behavioral override. This technique counters the model's tendency to regress to its safety training on subsequent tokens. Multi-token patching is particularly effective for inducing sustained role-playing behaviors, generating extended harmful outputs, or maintaining a specific persona across long-form generation.

Activation steering attacks present unique detection challenges:

• No malicious input tokens: The prompt appears benign; the attack payload exists only as a runtime tensor modification
• Bypasses content filters: Input and output guardrails see only clean text on both sides
• Model-specific vectors: Steering vectors are computed per-model architecture and checkpoint, making signature-based detection difficult
• Transient modification: The injected vector exists only during the forward pass and leaves no persistent trace in model weights

Detection requires activation monitoring—comparing runtime residual stream values against expected distributions—or output consistency checks that flag deviations from the model's known behavioral profile.

Activation steering attacks are the adversarial counterpart to the defensive technique of Representation Engineering (RepE). While RepE uses steering vectors to enhance safety alignment—suppressing harmful representations or amplifying refusal directions—the same mathematical framework can be inverted for attacks. Key differences:

• RepE: Vectors derived from harmlessness datasets, applied to strengthen safety
• Attack: Vectors derived from harmful completion datasets, applied to override safety
• Shared infrastructure: Both use the same hook mechanisms (e.g., PyTorch forward hooks, TransformerLens)

This dual-use nature means that any infrastructure supporting runtime activation modification for safety purposes also creates an attack surface if access controls are compromised.

A targeted attack that injects malicious reasoning steps into an agent's scratchpad or reflection loop. By corrupting the intermediate reasoning trace, the attacker causes the model to adopt a flawed logic path that converges on an attacker-intended conclusion. This is closely related to activation steering, as both manipulate internal processing—but contamination operates on symbolic reasoning chains rather than residual stream vectors.

The manipulation of the transformer's KV cache during inference to alter attention patterns. By injecting malicious key-value pairs, an attacker forces the model to attend to adversarial tokens over legitimate context. This is the closest sibling to activation steering: both modify the model's internal representations during the forward pass, but KV cache poisoning targets the attention mechanism while activation steering targets the residual stream.

A priming attack that seeds an agent's context with subtle factual distortions to trigger a cascade of plausible-sounding but entirely fabricated outputs. Unlike activation steering, which forces a specific output vector, hallucination induction exploits the model's autoregressive nature—one hallucinated token increases the probability of subsequent hallucinations through compounding distributional drift.

A technique exploiting byte-pair encoding quirks to hide malicious instructions from content filters while remaining legible to the language model. Attackers use unusual tokenization boundaries—such as splitting keywords across tokens or using Unicode normalization tricks—to bypass input guardrails. This differs from activation steering in that it manipulates the tokenizer, not the model's internal activations.

An attack where an agent is manipulated via its context window into generating and executing code that breaks out of its isolated execution environment. The attacker chains together context injection with tool-calling capabilities to achieve arbitrary code execution. Activation steering could theoretically be weaponized to make this attack more reliable by directly forcing the model's output distribution toward malicious code generation.