Activation Patching

A standard activation patching experiment requires three forward passes to isolate a specific computation:

1. Clean run: Process the original input and cache all activations at every layer.
2. Corrupted run: Process a perturbed input (e.g., with noise or a different context) where the target behavior is absent.
3. Patched run: Re-run the corrupted input, but at a specific layer and token position, replace the activation with the cached clean version.

If the model's behavior on the corrupted input is restored to match the clean run, the patched component is causally implicated in that behavior.

Researchers can apply activation patching at different levels of granularity to progressively narrow down the responsible component:

• Layer-level patching: Replace the entire residual stream output of a transformer layer to identify which depth is critical.
• Component-level patching: Target specific attention heads or MLP blocks within a layer.
• Neuron-level patching: Patch individual neurons within an MLP to find the exact knowledge neurons responsible for a fact.
• Position-level patching: Restrict patching to specific token positions in the sequence to isolate where information is processed.

While both are causal methods, activation patching and ablation serve different purposes and have distinct failure modes:

• Ablation (zero or mean ablation): Removes a component entirely by setting its activation to zero or its dataset mean. This tests necessity but can create out-of-distribution activations that the downstream layers never encountered during training.
• Activation patching: Replaces with a real, in-distribution activation from another run. This preserves the statistical structure of the activation space, avoiding the pathological behaviors that zero ablation can trigger.
• Best practice: Use patching from a counterfactual input rather than zeroing out for cleaner causal evidence.

Identifies where a transformer stores and retrieves factual knowledge. By patching activations from a subject prompt into a corrupted prompt, researchers can trace the causal pathway of memory recall.

• Mechanism: Corrupt the subject token (e.g., 'The Eiffel Tower is in...') and restore clean activations layer by layer.
• Finding: Early-middle MLP layers in models like GPT-2 are causally responsible for recalling factual associations.
• Impact: Enables precise model editing and knowledge updates without full retraining.

Demonstrates how transformers route information between token positions. Activation patching reveals the specific attention heads that move object information to the correct grammatical position.

• Setup: Patch activations from a sentence with a clear indirect object into one where it's ambiguous.
• Result: Isolates name mover heads in later layers that copy the indirect object to the output position.
• Significance: Proves transformers implement discrete, interpretable algorithms for syntactic tasks.

Maps the complete computational subgraph responsible for a specific behavior. By iteratively patching nodes and edges, researchers reverse-engineer circuits—the minimal set of components that explain a model's output.

• Example: The 'greater-than' circuit in a small transformer computes numerical comparisons using a specific sequence of attention heads and MLP neurons.
• Method: Path patching measures the direct causal effect of one component on another, isolating interaction pathways.
• Outcome: Provides a blueprint for verifying model safety and alignment at the mechanistic level.

Locates the internal origin of harmful or biased outputs. Activation patching can isolate the specific neurons or attention heads that drive stereotypical associations.

• Process: Use a counterfactual prompt pair (e.g., gendered profession sentences) and patch activations to see which components flip the output bias.
• Application: Enables precise concept erasure by zero-ablating or editing only the identified toxic sub-circuit.
• Advantage: Avoids degrading overall model performance by surgically removing only the problematic computation.

Reveals how multilingual models share or separate linguistic knowledge. Patching activations between different languages tests whether a component is language-agnostic or language-specific.

• Experiment: Patch the activation of a factual subject in English into a run processing the same fact in French.
• Interpretation: If the output recovers, the patched component encodes a language-independent concept.
• Use Case: Informs architecture design for better cross-lingual generalization and knowledge sharing.

Confirms the existence and function of induction heads—attention heads that perform in-context learning by attending to previous occurrences of the current token.

• Pattern: An induction head attends to token B in sequence [A, B, ..., A] to predict that B follows A again.
• Patching Proof: Replacing the key-value cache of a candidate head with one from a sequence where the pattern is absent destroys the copying behavior.
• Relevance: Induction heads are the primary mechanism behind few-shot learning in transformers.