Activation Patching

Activation patching is a form of causal mediation analysis applied to neural networks. It tests whether a specific component (a node, attention head, or MLP layer) is a necessary mediator in a causal chain. By corrupting the input to create a baseline (e.g., a model failing to answer) and patching in a clean activation from a forward pass where the model succeeds, you can measure the indirect effect of that specific activation on the final output. This isolates the component's unique contribution.

The direction of the patch defines the experiment:

• Denoising Patch: A clean activation from a successful run is patched into a corrupted, failing run. If performance is restored, the patched location is a critical bottleneck for the behavior.
• Corrupting Patch: A corrupted activation (e.g., from a run with Gaussian noise added to input) is patched into a clean run. If performance degrades, the location is confirmed to be causally necessary. This bidirectional approach provides robust evidence for a component's functional role.

The primary goal is to identify circuits—subgraphs of the computational graph that implement a specific algorithm. For example, in a transformer performing indirect object identification (IOI), patching has revealed distinct attention heads responsible for:

• Moving the subject token's position information
• Copying the indirect object's value
• Suppressing duplicate names By systematically patching activations at every layer and position, researchers build a precise map of the mechanistic pathway for a single, well-defined behavior.

To ensure results are not an artifact of a single input, activation patching relies on resampling. The clean and corrupted activations are averaged over a distribution of prompts that share the target behavior. This distinguishes components that are genuinely part of a general algorithm from those that only activate for specific tokens. A component is considered part of a circuit only if patching it has a consistent, statistically significant effect across the entire distribution.

Standard patching modifies a node's output, affecting all downstream paths. Path patching is a refined technique that isolates a specific sender-receiver edge. It replaces the activation only along the direct path from component A to component B, leaving A's output to other components unchanged. This is crucial for understanding composition, revealing how information flows through specific attention head interactions without disrupting the rest of the residual stream.

While related, patching is more surgical than ablation. Ablation (zeroing or mean-ablating a neuron) destroys information, potentially causing cascading failures that obscure the component's true function. Activation patching replaces information with a counterfactual alternative, preserving the overall activation distribution. This allows researchers to ask, 'What specific information does this component contribute?' rather than just 'Does the model break without it?'

The foundational methodology that activation patching operationalizes. Causal tracing systematically identifies which hidden state representations at which layers are causally responsible for a model's factual predictions.

• Locates the specific MLP layers where facts are encoded
• Distinguishes between subject representation and relation extraction
• Uses corrupted activations from a counterfactual input as the intervention
• Reveals the causal graph of information flow through the network

The alternative activation used to replace the original during a patching experiment. The choice of baseline is critical to the validity of the causal claim.

• Zero ablation: Replacing activations with zeros (crude, can destroy model function)
• Mean ablation: Using the average activation across a dataset
• Gaussian noise: Adding random perturbations to measure sensitivity
• Counterfactual input: Running the model on a minimally different input to isolate a specific feature
• Resampling: Drawing activations from other tokens in the same sequence

An extension of activation patching that isolates not just a single component, but the entire computational path between two specific components. This reveals how information flows through the network's computational graph.

• Patches the direct connection from component A to component B
• Leaves all other paths into B intact
• Used to map attention head composition in transformers
• Demonstrates how heads in earlier layers write to subspaces read by later heads
• Critical for understanding induction heads and in-context learning circuits

Two complementary experimental paradigms that test necessity and sufficiency of a component for a behavior.

Knockout Patching:

• Replaces the target activation with a corrupted version
• Tests: Is this component necessary for the behavior?
• If performance drops, the component is causally implicated

Restoration Patching:

• Starts with a fully corrupted model
• Restores only the target activation to its clean state
• Tests: Is this component sufficient to recover the behavior?
• If performance recovers, the component carries the critical signal

A gradient-based approximation of activation patching that estimates the indirect effect of an activation on the output without requiring a full forward pass for every intervention.

• Computes the first-order Taylor expansion of the patching effect
• Uses the gradient of the loss with respect to the activation
• Scales linearly with model size instead of requiring separate runs
• Enables rapid screening of thousands of components
• Trade-off: approximation error for large-magnitude interventions
• Used as a fast filter before precise causal tracing with full patching

The end goal of iterative activation patching experiments: identifying the minimal subgraph of model components that are jointly necessary and sufficient for a specific behavior.

• Combines knockout and restoration patching across components
• Identifies sparse circuits that explain complex behaviors
• Reveals that models often use surprisingly narrow, interpretable algorithms
• Key findings include induction heads for copying patterns
• Demonstrates that large models can be decomposed into composable functional units
• Directly informs AI safety by locating and potentially editing undesirable circuits