Ablation

Ablation transforms a passive observation into a controlled experiment. The core logic is counterfactual: if component X is necessary for behavior Y, removing X must degrade Y. This directly tests the causal mediation pathway, distinguishing components that are merely correlated with an output from those that are functionally required to produce it.

The replacement value critically impacts the interpretation of the result:

• Zero Ablation: Sets activations to 0. This can push the network into an out-of-distribution state, causing spurious failures unrelated to the component's function.
• Mean Ablation: Replaces activations with the statistical mean from a distribution of inputs. This preserves the baseline statistics of the network and provides a cleaner causal signal.

A more rigorous variant where a component's activation is replaced by its activation from a different, randomly sampled input. This destroys the specific information content of the original activation while preserving its marginal distribution. It is the gold standard for testing if a component encodes a specific, context-dependent feature.

Ablation granularity varies by architecture:

• Neuron Ablation: Zeroing out individual neurons in an MLP layer to find knowledge neurons responsible for specific facts.
• Head Ablation: Removing entire attention heads to identify specialized functions like syntactic parsing or anaphora resolution.
• Layer Ablation: Deleting entire layers to probe the depth at which a capability emerges, often used to validate the superposition hypothesis.

The result of an ablation is a performance delta (Δ). Researchers must distinguish a critical drop from noise. Common metrics include:

• Task Accuracy Drop: A direct measure of degradation on a specific benchmark.
• KL Divergence Shift: Measures how much the model's output distribution changes, even if the top-1 prediction remains the same.
• Causal Tracing Score: The indirect effect (IE) of restoring a clean activation while corrupting others, localizing the exact hidden state responsible for a fact.

While often used interchangeably, a subtle distinction exists. Ablation typically implies a clean, targeted removal of a specific architectural component in a digital model. A lesion study is a broader term inherited from neuroscience, referring to the general process of damaging a system to observe deficits. In deep learning, ablation is the precise, engineering-focused implementation of a lesion study.

A statistical framework for quantifying how much a model's output depends on a specific intermediate representation. It measures the average treatment effect of intervening on a neuron or attention head across a distribution of inputs. Unlike simple ablation, this decomposes total effect into direct and indirect pathways, providing a rigorous causal graph for understanding information flow through the network.

A technique for training a simple linear classifier on top of a frozen model's internal representations to diagnose what information is encoded at a specific layer. If a probe can accurately predict a linguistic property like part-of-speech or a factual attribute from a layer's activations, that information is linearly accessible. This contrasts with ablation by focusing on representational content rather than causal necessity.

A method for identifying the specific hidden states within a transformer that are causally responsible for recalling a fact. It works by corrupting the input with noise and then systematically restoring clean activations from a clean forward pass at individual layers and token positions. The restoration point that recovers the original output reveals where the factual association is stored.

An unsupervised neural network trained to reconstruct activations from a model while enforcing sparsity. This decomposes polysemantic neurons—which fire for multiple unrelated concepts—into a set of monosemantic features that each represent a single interpretable concept. Ablating these disentangled features provides much cleaner causal evidence than ablating entire neurons.

A causal method that replaces the activation of a model component processing a base input with the activation from a source input. If the model's output changes exactly as predicted by a high-level causal hypothesis, it confirms that the component computes a specific abstract variable. This provides stronger evidence than ablation by testing for exact equivalence rather than just necessity.