Causal Scrubbing

Causal scrubbing operates by formalizing a mechanistic hypothesis about which computational subgraph implements a behavior. The hypothesis specifies exactly which nodes (attention heads, MLP neurons) and edges (connections between them) are necessary and sufficient. This hypothesis is then tested by resampling activations from a reference distribution—typically activations from other inputs where the hypothesized circuit should behave identically—and measuring whether the model's output remains invariant. If the circuit is correctly specified, the scrubbed model's behavior should match the original.

Unlike ablation methods that zero out or add noise to activations, causal scrubbing replaces activations with real activations drawn from other forward passes. This preserves the natural statistics of the activation distribution, avoiding out-of-distribution artifacts that can trigger spurious compensation mechanisms. The key insight: if a component is truly irrelevant to a behavior, replacing its activations with those from a different input should not change the output. The resampling is performed at specific token positions and layers according to the hypothesis tree.

The core quantitative output of causal scrubbing is a faithfulness score that measures how well the hypothesized circuit explains the model's behavior. This is computed by comparing the model's output distribution under the scrubbed activations against the original output:

• Perfect faithfulness: The scrubbed model produces identical outputs to the original
• Low faithfulness: The hypothesis misses important components or includes irrelevant ones
• The metric can be decomposed to identify which specific nodes in the hypothesis are most responsible for faithfulness failures

Hypotheses in causal scrubbing are represented as tree structures where each node corresponds to a computational component and its activation context. The tree encodes:

• Which inputs should map to the same activation under the hypothesis
• Equivalence classes of token positions that should share representations
• Hierarchical dependencies between components This tree structure enables systematic testing of complex, multi-component circuits by specifying precisely how activations should be resampled across different inputs while preserving the hypothesized computational structure.

While both are causal intervention techniques, causal scrubbing differs fundamentally from activation patching:

• Activation patching: Replaces a single activation with one from a specific counterfactual input to measure local causal effects
• Causal scrubbing: Systematically resamples many activations simultaneously according to a global hypothesis about the entire circuit
• Scrubbing tests joint faithfulness of all hypothesized components together, while patching measures individual causal contributions
• Scrubbing uses distributional resampling rather than single-point interventions, making it more robust to polysemantic neurons

Causal scrubbing enables a scientific loop for mechanistic interpretability:

1. Propose an initial circuit hypothesis based on observational techniques like attention pattern analysis
2. Scrub the model by resampling activations according to the hypothesis
3. Measure faithfulness—identify where the scrubbed model diverges from the original
4. Refine the hypothesis by adding missing components or removing unnecessary ones
5. Repeat until the circuit achieves high faithfulness, indicating a complete mechanistic explanation This iterative process transforms interpretability from speculation into falsifiable science.

A foundational causal intervention technique that replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass. This isolates the functional role of a component by observing how the output changes when specific information is substituted. Causal scrubbing generalizes this by systematically resampling activations across an entire hypothesized circuit rather than patching a single location.

The process of identifying and validating the minimal subgraph of a neural network's computational graph that is necessary and sufficient to perform a specific behavior. Causal scrubbing serves as the primary validation step in circuit analysis: once a candidate circuit is proposed, scrubbing tests whether the circuit's components are faithful by checking if behavior is preserved when all other activations are resampled from a distribution where the circuit should fail.

A statistical framework adapted from epidemiology that quantifies the contribution of a specific intermediate variable to a model's output. It measures the indirect effect through a mediator by comparing outcomes when the mediator is set to values from different treatment conditions. Causal scrubbing extends this logic to entire circuits by testing whether the hypothesized information flow path mediates the full effect on the model's behavior.

A phenomenon observed during ablation studies where the network dynamically compensates for the removal of a component by adjusting the activations of other, redundant components in later layers. This complicates simple ablation-based analysis because removing a single node may show no effect even if it participates in the circuit. Causal scrubbing addresses this by resampling activations from a reference distribution rather than zeroing them out, making it harder for the network to repair the intervention.

An unsupervised technique that decomposes a model's dense, polysemantic internal activations into a sparse set of interpretable, monosemantic features using a learned overcomplete basis. SAEs produce the feature-level decomposition that causal scrubbing can then test: instead of scrubbing at the neuron level, researchers can formulate hypotheses about which sparse features constitute a circuit and scrub those specific feature activations.

The primary information highway in a transformer model where each layer reads from and writes to a shared accumulating state. Causal scrubbing often focuses on the residual stream as the central medium of information flow, testing whether hypothesized circuits correctly model how information is written to, read from, and transformed within this accumulating state across layers and token positions.