Causal Mediation Analysis

The ACME is the core metric that quantifies the indirect effect of a treatment on an outcome through a specific mediator variable. In mechanistic interpretability, this translates to measuring how much a model's final prediction changes when you intervene on an intermediate activation, while holding the direct input path constant. The effect is calculated as the difference between the outcome when the mediator is set to its value under the treatment condition versus the control condition. This decomposition allows researchers to isolate the causal contribution of a single neuron, attention head, or layer from the total effect.

The total causal effect of an input on a model's output is mathematically decomposed into two distinct pathways:

• Natural Direct Effect (NDE): The effect that flows directly from the input to the output, bypassing the mediator of interest entirely.
• Natural Indirect Effect (NIE): The effect that is transmitted exclusively through the mediator. This decomposition is critical for circuit discovery, as it allows researchers to determine whether a specific model component is a necessary bottleneck for a particular behavior or merely a redundant parallel pathway.

Unlike purely correlational probing methods, causal mediation analysis relies on do-calculus and counterfactual reasoning. The analysis asks: What would the model's output have been if we had set this specific neuron's activation to a different value? This requires physically intervening in the model's forward pass—clamping, zeroing, or replacing an activation—while leaving all other computations unchanged. This interchange intervention framework is what distinguishes causal mediation from mere correlation, providing evidence that a representation is not just predictive of an outcome but functionally responsible for it.

A critical design choice is selecting the mediator variable from the high-dimensional activation space. Common targets include:

• Residual stream states at specific token positions and layers.
• Attention head outputs before the concatenation and projection step.
• MLP neuron activations within a feed-forward block. Researchers often apply dimensionality reduction or train a sparse autoencoder to decompose polysemantic activations into monosemantic features before using them as mediators, ensuring the intervention targets a single, interpretable concept rather than a superposition of features.

A prominent application of causal mediation analysis is causal tracing, which identifies the specific hidden states responsible for recalling factual associations in language models. The methodology involves three states:

• Clean run: The model processes a factual prompt normally.
• Corrupted run: The subject entity is obfuscated, corrupting the model's ability to recall the fact.
• Restoration run: Clean activations from a single layer and token position are patched back into the corrupted run. By measuring the probability of the correct answer after each restoration, researchers localize the knowledge neurons and MLP layers that mediate factual recall.

Valid causal mediation analysis requires satisfying strong identification assumptions:

• Sequential Ignorability: There are no unmeasured confounders between the treatment and mediator, or between the mediator and outcome.
• Consistency: The observed outcome equals the potential outcome under the observed treatment and mediator values. In neural network analysis, these assumptions are often violated due to superposition and distributed representations. Researchers conduct sensitivity analyses by varying the intervention magnitude and location to test the robustness of their causal claims against model-specific noise.