Causal Tracing

Causal tracing moves beyond correlation by performing controlled interventions on a neural network's internal state. The core technique, activation patching, involves:

• Running a clean forward pass with an unmodified input to establish a baseline.
• Running a corrupted forward pass where a specific fact or token is altered.
• Surgically patching activations from the clean pass into the corrupted pass, one component at a time (e.g., a specific attention head's output at layer 5).
• Measuring the causal effect by observing how much performance is restored. This isolates the component's necessary contribution to the computation.

The technique distinguishes components that are causally necessary from those that are merely correlated with an output. A neuron that activates for a concept is not proven causal; causal tracing demonstrates that ablating or altering its activation changes the specific output. This is critical for building falsifiable hypotheses about model mechanisms. It answers: "If I change this internal state, holding all else constant, does the model's behavior change predictably?"

Tracing operates at the level of individual model components, not just layers. It can pinpoint:

• Specific attention heads and their roles (e.g., a head that moves information from a subject token to a predicate).
• Individual neurons or channels within feed-forward networks (identified as knowledge neurons).
• MLP layers that perform non-linear transformations on retrieved information.
• The critical path—the minimal sequence of components required to produce a behavior. This granularity is foundational for targeted model editing.

The method is inherently counterfactual. It asks: "What would the model's output have been if this component's activation were different?" This requires a well-defined corruption that breaks the behavior of interest (e.g., changing 'Paris' to 'London' in the prompt 'The capital of France is _'). The difference between the corrupted output and the output after patching quantifies the causal strength of the intervened-upon component.

Causal tracing directly informs and validates model editing methods like ROME and MEMIT. By identifying the exact layers and neurons responsible for storing a specific fact, edits can be surgically applied to those locations. For example, tracing might reveal that a fact about a person is causally mediated by neurons in MLP layers 5-7; a rank-one weight update (ROME) is then applied precisely there. Tracing is used post-edit to verify the edit altered the intended causal pathway.

A key application of tracing is testing the locality hypothesis—the idea that knowledge is stored in localized parameters. After an edit, causal tracing can determine if the new behavior uses the same computational pathway as the original fact (suggesting localized rewriting) or a different, patched-in pathway (suggesting external override). It also evaluates edit specificity by tracing unrelated inputs to ensure their causal pathways remain unchanged, detecting unwanted side effects.

The locality hypothesis is a central conjecture in model editing which posits that specific pieces of knowledge or behaviors in a neural network are encoded in localized, relatively sparse subsets of its parameters (e.g., specific neurons or attention heads).

• Implication for Editing: If true, it enables targeted edits that change a model's behavior on a narrow set of inputs without degrading its general capabilities (edit specificity).
• Evidence: Findings like knowledge neurons and the success of localized editing methods (ROME, MEMIT) provide empirical support.
• Contrast: Opposed by a 'distributed representation' view where knowledge is densely entangled across many parameters, making isolated edits impossible without side effects.

Model surgery is a metaphor for the practice of making precise, low-level interventions on a neural network's architecture or parameters to alter specific behaviors—analogous to surgical procedures in medicine.

• Encompasses Techniques: Includes parameter patching, neuron editing, and methods like ROME/MEMIT that directly modify weights.
• Guiding Principle: Minimally invasive changes that achieve a specific objective while preserving the overall 'health' (performance) of the model.
• Tooling: Relies on mechanistic interpretability (causal tracing) as a 'diagnostic' to identify the correct 'surgical site' before making an edit.

Intervention analysis is the broader category of experimental methods in causal inference for machine learning that involves actively manipulating a system's internal state to measure causal effects. Causal tracing is a specific form of intervention analysis designed for transformer models.

• General Principle: To establish causality (A causes B), one must intervene on A and observe changes in B, moving beyond passive observation and correlation.
• Methods in ML: Includes activation patching, ablation studies (zeroing out components), and path patching.
• Outcome: Produces causal graphs or attribution scores that quantify the importance of different model components for a given computation.