Causal Mediation Analysis

The primary experimental tool for causal mediation in neural networks is activation patching. This technique performs a surgical intervention by replacing a specific hidden state (the mediator) with a cached value from a counterfactual forward pass, while keeping all other network components frozen.

• Clean Run: The model processes the original prompt.
• Corrupted Run: The model processes a perturbed input where key information is obfuscated.
• Patch Run: The corrupted run is executed, but the activation at a specific layer and token position is overwritten with the value from the clean run.

By measuring the degree to which the patch restores the original output, researchers assign a causal score to that specific component.

Causal Scrubbing is a formal hypothesis-testing framework that operationalizes causal mediation at scale. A researcher proposes a specific computational circuit as the explanation for a behavior. The framework then systematically replaces activations that are off-circuit with corrupted values while preserving on-circuit activations. If the proposed circuit is faithful, the model's performance should be fully recovered. Crucially, causal scrubbing tests the sufficiency of the circuit by resampling activations across diverse input distributions, ensuring the explanation generalizes and doesn't rely on spurious correlations.

A core causal intervention technique that replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass. This isolates the functional contribution of a circuit by observing how the output changes when a single component's state is swapped.

• Purpose: To localize where a behavior is computed.
• Mechanism: Corrupts a clean run with a patched activation from a corrupted run, or vice versa.
• Key Metric: The restoration of correct output probability after patching a clean activation into a corrupted run.

A specific methodology for locating where factual knowledge is stored in a transformer model. It runs the model with corrupted input embeddings and then systematically restores clean hidden states one layer at a time to measure the causal effect on the output.

• Goal: Identify the specific MLP layers and attention heads that mediate factual recall.
• Process: Corrupt the subject tokens, then iteratively patch clean states from the uncorrupted run.
• Finding: Revealed that MLP modules in middle layers have the strongest causal role in retrieving factual associations.

A formal hypothesis-testing framework that systematically replaces activations to verify if a proposed circuit explains a model's behavior. It checks if the circuit's components are faithful under resampling.

• Function: Tests if a hypothesized circuit is necessary and sufficient for a behavior.
• Method: Replaces activations from a distribution that should be equivalent if the hypothesis is correct.
• Outcome: If the model's performance is unchanged after scrubbing, the hypothesis is validated; a drop in performance indicates the hypothesis is incomplete.

The statistical measure quantifying how much a change in an upstream variable influences the final output through a specific mediator, while holding the direct path constant. In neural networks, this isolates the contribution of a single neuron or layer.

• Calculation: Total Effect - Direct Effect.
• Application: Used to determine if an attention head's influence on the logits is mediated by a downstream MLP layer.
• Significance: Distinguishes between a component's direct writing to the residual stream and its influence on subsequent components.

A specific neuron within an MLP layer identified through causal mediation that is primarily responsible for expressing a particular piece of factual knowledge. These neurons act as key-value memory slots.

• Identification: Found by deactivating neurons and measuring the drop in probability for a specific fact.
• Property: Often exhibit a thresholding behavior, activating strongly only when the context is sufficiently related.
• Role: Mediates the causal pathway from the subject token to the correct attribute prediction.

A phenomenon observed during ablation studies where the network dynamically compensates for the removal of a component by adjusting the activations of other, redundant components in later layers. This complicates causal mediation analysis.

• Impact: The measured indirect effect of a component can be masked by downstream compensation.
• Detection: Requires comparing the effect of an ablation immediately versus after allowing the signal to propagate through several layers.
• Implication: A small drop in output after ablation does not necessarily mean the component is unimportant; it may indicate robust redundancy.