Causal Tracing

Causal tracing relies on three distinct forward passes to isolate factual storage:

• Clean Run: The model processes the original, uncorrupted prompt to establish a baseline output distribution
• Corrupted Run: The subject entity in the input embeddings is obfuscated (e.g., adding Gaussian noise), causing the model to fail at recalling the target fact
• Corrupted-with-Restoration Run: The corrupted run is repeated, but a specific hidden state from the clean run is patched back in at a single layer and token position

The causal effect is measured as the increase in probability of the correct answer between the corrupted and restoration runs.

By iterating the restoration process across every layer and token position, causal tracing produces a causal effect map that reveals precisely where factual recall occurs:

• Early sites (layers 0-5): Restoring here has minimal effect, as the model is still processing syntax and entity recognition
• Middle sites (layers 5-15): The peak causal effect typically appears in the MLP layers of middle transformer blocks, identifying the specific neurons that encode the fact
• Late sites (layers 15+): Effect diminishes as the model transitions to generating the output token This gradient of influence demonstrates that factual knowledge is not uniformly distributed but concentrated in specific knowledge neurons within MLP modules.

Causal tracing reveals a critical architectural insight: the restoration effect is strongest when applied to the last subject token in the prompt. For example, in the prompt 'The Eiffel Tower is located in', restoring the hidden state at the token 'Tower' yields the highest causal effect on predicting 'Paris'.

This occurs because:

• The model pools information about the subject entity into the representation of its final token
• Attention heads in earlier layers copy subject-attribute bindings to this token position
• The MLP at the peak causal layer then reads from this enriched representation to retrieve the stored fact This finding directly informed the design of Rank-One Model Editing (ROME), which surgically modifies the MLP weights at the identified layer.

Causal tracing differs fundamentally from gradient-based attribution techniques:

• Gradient-based methods (e.g., integrated gradients, saliency maps) measure input sensitivity—how much changing an input feature affects the output—but do not reveal internal storage mechanisms
• Causal tracing performs structural interventions on internal activations, directly testing counterfactual scenarios: 'Would the model still know this fact if this specific state were corrupted?'
• The method draws from the potential outcomes framework in causal inference, treating each hidden state as a mediator variable and measuring the indirect effect through that mediator This interventionist approach provides stronger evidence for functional localization than purely observational methods.

Causal tracing has several known constraints that researchers must account for:

• Subject-relation specificity: The method works best for subject-relation-object triples (e.g., 'Paris is the capital of France'). Performance degrades on facts requiring multi-hop reasoning or temporal context
• Self-repair effects: When a knowledge neuron is ablated, the model may dynamically recruit redundant pathways in later layers, causing causal tracing to underestimate the distributed nature of some facts
• Prompt sensitivity: The measured causal effect varies with prompt phrasing; a fact stored in multiple locations may show different peak layers depending on how the subject is introduced
• Computational cost: Tracing a single fact requires O(n) forward passes where n is the number of layers, making large-scale analysis expensive These limitations motivate complementary techniques like causal scrubbing and dictionary learning for more complete mechanistic pictures.

Causal tracing provides the foundational localization step for surgical model editing techniques:

• ROME (Rank-One Model Editing) uses causal tracing to identify the specific MLP layer where a fact is stored, then treats that layer as a linear associative memory and performs a rank-one update to insert a new fact
• MEMIT (Mass-Editing Memory in a Transformer) extends this to edit thousands of facts simultaneously by spreading updates across multiple layers identified through causal tracing
• The causal trace map serves as a surgical map—editing at the wrong layer can corrupt unrelated knowledge, while editing at the peak causal layer maximizes specificity This pipeline from localization to intervention demonstrates how mechanistic interpretability directly enables practical model control.

A foundational causal intervention technique that replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass. By observing how the output changes, researchers can isolate which components are necessary for a behavior. Causal tracing is a specific, systematic form of activation patching that restores clean states from a corrupted run to measure the indirect causal effect of each layer on factual recall.

A specific neuron within an MLP layer identified through causal tracing that is primarily responsible for expressing a particular piece of factual knowledge. Causal tracing often reveals that factual recall is localized to a small set of neurons in the early-to-middle MLP layers of a transformer. These neurons act as key-value memory slots, firing when the model needs to retrieve a stored fact like 'The Eiffel Tower is in Paris'.

A precise model editing technique that treats an MLP layer as a linear associative memory. ROME uses causal tracing to first locate the specific layer where a fact is stored, then inserts a new fact by performing a rank-one update to the weight matrix of that layer. This surgically alters the stored knowledge without affecting unrelated capabilities, demonstrating the practical application of causal tracing for targeted model modification.

A statistical framework adapted from the social sciences for neural networks. It quantifies the contribution of a specific intermediate variable—such as a neuron or attention head—to a model's output by measuring the natural indirect effect through that mediator. Causal tracing operationalizes this by corrupting the input and then restoring clean states at a single mediator to isolate its causal role in the total effect.

The process of identifying and validating the minimal subgraph of a neural network's computational graph that is necessary and sufficient to perform a specific behavior. Causal tracing is often the first step in circuit discovery, highlighting which layers and components are causally implicated. Subsequent analysis then maps the precise QK and OV circuits within attention heads and the downstream MLP neurons that form the complete functional pathway.

A direct probing method that applies the unembedding matrix to intermediate residual stream states to interpret the model's next-token predictions before the final layer norm is applied. While causal tracing measures causal necessity through intervention, the logit lens measures representational content by decoding hidden states. Using both techniques together provides a complete picture: what information is present at each layer and whether that information is actually used.