Activation Patching

Activation patching is the core experimental method for causal circuit discovery. By systematically replacing activations from a 'clean' forward pass with those from a 'corrupted' pass (where an input is altered to change the output), researchers can identify which specific components—such as individual attention heads or MLP neurons—are necessary for a given computation.

• Process: Measure a baseline output, corrupt a specific input feature, then patch in activations from the clean run one component at a time to see which restoration fixes the output.
• Outcome: This reveals causal pathways, like identifying the 'induction head' circuit responsible for in-context learning in transformers.

Engineers use activation patching to debug model failures such as hallucinations or biased outputs. By comparing activations from a failing example to those from a similar, correctly handled example, they can pinpoint the computational step where the error originates.

• Application: For a factual hallucination, patch activations from a factually correct run into the hallucinating run at successive layers. The layer where the output corrects itself indicates where the erroneous representation was formed.
• Value: This moves debugging from guesswork to a causal, localized diagnosis, enabling precise fixes via techniques like model editing.

Activation patching enables direct behavioral steering by injecting desired activation patterns. This is a precursor to more permanent model editing techniques.

• Method: Identify a component whose activation pattern correlates with a desired behavior (e.g., truthful answering). Record this pattern from a 'source' run, then patch it into a 'target' run during inference to steer the output.
• Use Case: This can be used for contrastive activation patching, where patching the difference between 'truthful' and 'deceptive' activations can amplify a model's honesty without retraining.

The technique provides causal validation for hypotheses generated by other interpretability methods, such as logit lens or attention visualization, which are often only correlational.

• Workflow: A researcher might hypothesize that a specific neuron cluster represents a concept (e.g., 'Python code'). They can test this by patching in high-activations for that cluster from a code-related prompt into a non-code prompt. If the output becomes code-like, it causally confirms the hypothesis.
• Importance: This turns soft 'interpretations' into falsifiable, mechanistic theories about model internals.

In AI safety, activation patching is used for adversarial circuit analysis to find vulnerabilities. Researchers corrupt inputs with jailbreak prompts or adversarial suffixes and patch to find the components that propagate unsafe outputs.

• Process: Patch activations from a safe, refused response into a run that produced a harmful response. The component where patching blocks the harmful output is a critical control point.
• Outcome: This identifies safety-relevant circuits that can be monitored, reinforced, or ablated to improve model robustness, informing the design of safety fine-tuning and monitoring systems.

Activation patching directly informs and evaluates model editing algorithms like ROME and MEMIT. By identifying which layers and neurons are causally important for a fact, edits can be targeted precisely.

• Pre-Editing: Patching determines the edit locus—the minimal set of layers where an intervention will be effective.
• Post-Editing: Patching is used for side-effect evaluation, testing whether an edit changed only the intended behavior by patching old activations back in to see if original performance on other tasks is restored.
• Link: This creates a rigorous R&D loop from interpretability (causal tracing) to intervention (model editing).

Causal tracing is a foundational mechanistic interpretability technique used to identify the specific internal components (e.g., neurons, attention heads, MLP layers) within a neural network that are causally responsible for a particular model output. It works by:

• Injecting noise into intermediate activations during a forward pass.
• Observing the resulting degradation in the model's performance on a specific task.
• The components where noise injection causes the most significant performance drop are identified as part of the causal circuit for that task. Activation patching is often used to validate the findings of causal tracing by surgically restoring clean activations to confirm their effect.

Model editing is a family of techniques for making precise, targeted updates to a neural network's knowledge or behavior without performing full retraining. The goal is to correct errors, update facts, or adjust behaviors with surgical precision. Key paradigms include:

• Direct Parameter Editing (e.g., ROME, MEMIT): Modifies a small subset of the model's weights.
• Hypernetwork Editors (e.g., MEND): Uses a secondary network to predict weight deltas.
• External Memory Patching: Stores edits in a separate, non-parametric memory. Activation patching is a critical analysis tool used to understand model internals before designing an edit and to evaluate the specificity and side effects of an edit afterward.

Knowledge neurons are specific, identifiable neurons or units within a neural network—particularly in the feed-forward (MLP) layers of transformer models—that activate strongly in response to, and are causally important for, specific pieces of factual knowledge. For example, a cluster of neurons may fire consistently when the model processes facts about 'the Eiffel Tower.'

• They are discovered using gradient-based attribution and activation analysis.
• Their existence supports the locality hypothesis, suggesting knowledge is stored in localized parameters. Activation patching experiments are used to ablate or stimulate these neurons to prove their causal role, directly demonstrating that patching in their activations from a 'Paris' context can change a model's answer on a related factual query.

The locality hypothesis is a central conjecture in mechanistic interpretability and model editing. It posits that a neural network's knowledge and behaviors are locally stored in specific, relatively narrow circuits, parameters, or activation patterns. This implies:

• Edits can be targeted to a small subset of the model without affecting unrelated capabilities.
• It is possible to isolate the function of specific components. Activation patching provides direct experimental evidence for or against this hypothesis. By successfully patching a local activation and changing a model's output for a specific input without affecting other inputs, researchers demonstrate locality. Conversely, if patching causes widespread side effects, it challenges strict locality.

Mechanistic interpretability is the field of research aimed at reverse-engineering neural networks into human-understandable algorithms and circuits. The goal is not just to know that a model works, but to develop a causal, component-level understanding of how it works.

• Core Techniques: Include activation patching, causal tracing, dimensionality reduction (e.g., PCA), and automated circuit discovery.
• Distinction: Contrasts with functional interpretability (e.g., feature attribution maps like SHAP), which describes correlations between inputs and outputs but not internal mechanisms. Activation patching is a primary experimental tool in this field, allowing researchers to perform ablation studies at the level of individual vectors and matrices to test hypotheses about circuit function.

These are the two key evaluation metrics for any model editing or intervention technique, directly tested using activation patching.

• Edit Specificity: Measures whether the changed behavior is confined to the intended set of inputs. High specificity means the edit only affects the target case (e.g., 'The capital of France is Paris') and not unrelated queries.
• Side Effects: Unintended, negative impacts on the model's performance on tasks or knowledge unrelated to the edit. For example, an edit to correct a fact about France should not degrade the model's ability to answer questions about chemistry. Activation patching is used in evaluation by comparing activations between the original and edited models across a broad suite of inputs. Significant divergence in activations for non-edited inputs indicates potential side effects, challenging the locality of the edit.