Activation Patching

Activation patching is a form of causal mediation analysis adapted for neural networks. It measures the indirect effect of a specific node or layer by comparing model outputs under three conditions:

• Clean run: Normal input, baseline performance.
• Corrupted run: Noised or altered input, degraded performance.
• Patched run: Corrupted input, but a specific activation is replaced with the clean version. If patching restores performance, that activation is a necessary mediator for the behavior.

The technique can be applied at various levels of the model's computational graph to isolate functionality:

• Residual Stream Patching: Replaces the entire hidden state at a specific layer and token position. Tests if a layer's output is sufficient.
• Attention Head Patching: Replaces the output of a single attention head. Isolates the function of specific QK and OV circuits.
• MLP Neuron Patching: Replaces the activation of a specific neuron. Used to identify knowledge neurons storing factual associations.
• Path Patching: A refined variant that patches activations along a specific path between two components, isolating direct interactions.

Two primary experimental paradigms exist:

• Denoising Patching: The corrupted run has Gaussian noise added to embeddings. Patching restores clean activations to find which components recover the signal. This tests sufficiency.
• Knockout Patching: The clean run is patched with corrupted or mean-ablated activations. If performance drops, the component is necessary for the task. This is often used to validate circuits found via denoising. The choice between them depends on whether you are testing for sufficiency or necessity.

A critical complication in activation patching is self-repair. When a component is ablated (knocked out), later layers can dynamically adjust their computations to compensate for the missing input.

• This means a knockout patch might show no effect, even if the component is normally involved.
• Self-repair reveals redundant pathways in the network.
• Researchers must account for this by patching late enough in the forward pass or by analyzing how downstream activations shift in response to the intervention.

Activation patching is the experimental backbone of Causal Scrubbing, a formal hypothesis-testing framework. Once a hypothesized circuit is identified, causal scrubbing systematically resamples activations:

• It replaces activations of nodes outside the hypothesized circuit with values from other inputs.
• If the circuit hypothesis is correct, the model's output should remain unchanged because only the identified nodes matter.
• If performance degrades, the hypothesis is incomplete or incorrect, requiring further patching experiments to refine.

Activation patching is central to Causal Tracing, which locates where facts are stored in transformer MLP layers. The process:

• Corrupt the subject token embeddings with noise so the model cannot predict the correct attribute.
• Patch clean hidden states back in, one layer at a time, starting from early to late layers.
• The layer where patching most restores the correct output is identified as the site of factual recall. This technique led to the discovery of knowledge neurons and the development of ROME for model editing.

The process of identifying and validating the minimal subgraph of a neural network's computational graph that is necessary and sufficient to perform a specific behavior. Activation patching is the primary experimental tool for circuit discovery.

• Nodes: Attention heads, MLP neurons, or residual stream directions
• Edges: Interactions between components via attention or residual connections
• Faithfulness: The circuit must reproduce the full model's behavior on the target task
• Completeness: The circuit must account for all components that causally influence the output

The primary information highway in a transformer model where each layer reads from and writes to a shared accumulating state. Activation patching often targets specific positions in this stream to isolate functional circuits.

• Accumulation: Each layer adds its output to the residual stream via a residual connection
• Token Positions: The stream maintains separate representations for each token in the sequence
• Linear Structure: Information is preserved additively, enabling direct probing with techniques like the logit lens
• Patching a residual stream position can reveal what information is carried forward to later layers

A phenomenon observed during ablation studies where the network dynamically compensates for the removal of a component by adjusting the activations of other, redundant components in later layers. This complicates the interpretation of activation patching results.

• Redundancy: Multiple heads or neurons may encode overlapping features
• Compensation: Later layers amplify signals from remaining components to recover lost information
• Implication: A null effect from patching does not guarantee the component is unimportant
• Detection: Comparing patching effects at different layers can reveal repair dynamics