Activation Patching

Activation patching is a form of causal mediation analysis applied to neural networks. It tests whether a specific model component (e.g., an attention head or MLP neuron) is a mediator for a particular behavior. By replacing the component's activation with a value from a counterfactual forward pass—where the input is corrupted or altered—researchers can measure the indirect effect on the model's output. If patching restores or breaks a specific behavior, the component is causally implicated in that computation.

The technique relies on establishing two distinct forward passes:

• Clean Run: A standard forward pass with the original input, capturing the model's normal behavior.
• Corrupted Run: A forward pass with a perturbed input (e.g., noise added, tokens shuffled) that breaks the targeted behavior. Patching involves taking an activation from the corrupted run and transplanting it into the clean run at a specific layer and token position. The resulting drop in performance on a specific metric quantifies the component's importance.

Activation patching enables the precise spatial localization of a behavior within a model's compute graph. By systematically patching activations at different layers, token positions, and component types (attention output vs. MLP output), researchers can map the exact nodes and edges responsible for a task. This granularity is essential for moving beyond correlational methods like probing and establishing a true causal graph of the model's internal algorithms.

A critical methodological refinement is resample ablation, where the patched-in activation is drawn from a distribution of corrupted runs rather than a single fixed example. This prevents the results from being confounded by idiosyncratic noise in one specific corrupted activation. By averaging over multiple corrupted samples, the technique provides a statistically robust measurement of the average treatment effect of removing a component's clean information, ensuring the findings are not artifacts of a single data point.

It is crucial to distinguish activation patching from direct weight modification:

• Activation Patching: Intervenes on the transient activations during a forward pass. The underlying model weights remain unchanged. This is a runtime intervention.
• Weight Patching/Editing: Surgically modifies the model's parameters permanently. This is a static intervention. Activation patching is preferred for causal tracing because it does not risk permanently damaging the model and allows for clean, reversible experiments to test functional hypotheses.

Path patching is a direct extension of activation patching that isolates the causal effect of a specific computational path between two components (e.g., from an attention head at layer 1 to an MLP layer at layer 5). While standard activation patching replaces a node's entire output, path patching uses a clever double-run setup: it takes the sending node's activation from the clean run and the receiving node's input from the corrupted run, then patches only the specific residual stream direction connecting them. This isolates direct effects from indirect, multi-hop effects.

Two fundamental patching strategies that differ in what they replace the original activation with:

• Zero ablation: Sets the activation to zero, completely removing the component's contribution. Tests necessity but can push the model into out-of-distribution states.
• Mean ablation: Replaces the activation with the average value computed across many forward passes on a reference distribution. Preserves typical activation statistics while removing instance-specific information.
• Resampling ablation: Replaces with an activation from a different, corrupted input — the gold standard for activation patching experiments.