Reverse Engineering

A causal intervention technique for isolating the function of a specific model component.

• Mechanism: Replaces a model's internal activation at a specific layer and token position with a value from a corrupted or alternative forward pass
• Purpose: Measures how much a specific activation contributes to a behavior by observing output changes
• Variants: Includes mean ablation (replacing with average value) and zero ablation (setting to zero)
• Application: Used to identify which attention heads or MLP neurons are causally responsible for factual recall, syntax processing, or in-context learning
• Limitation: Can miss distributed effects where multiple components redundantly encode the same feature

A refined causal method that isolates the direct effect of a specific computational path between two model components.

• Mechanism: Patches activations along a single sender-to-receiver path while freezing all other computational routes
• Advantage over Activation Patching: Eliminates confounding indirect effects that flow through other components
• Key Insight: A sender component may influence a receiver through multiple paths; path patching isolates just one
• Use Case: Determining whether an attention head directly writes to the residual stream in a way that a downstream head reads, versus influencing it through intermediate layers
• Origin: Developed to address the limitations of simple activation patching in highly interconnected transformer architectures

A systematic evaluation methodology that tests a hypothesized circuit by replacing all activations outside the circuit with corrupted values.

• Hypothesis Testing: Formally defines a circuit as the minimal set of components necessary and sufficient for a behavior
• Procedure: Corrupts all activations not part of the hypothesized circuit, then verifies if model performance is preserved
• Rigor: If performance degrades, the hypothesis is incomplete; if it holds, the circuit is validated
• Corruption Strategies: Includes shuffling activations across different inputs, adding noise, or replacing with mean values
• Significance: Provides a falsifiable framework for mechanistic interpretability, moving beyond qualitative inspection to quantitative validation

Techniques for decoding intermediate predictions by applying the unembedding matrix directly to residual stream activations at each layer.

• Logit Lens: Applies the final unembedding matrix to intermediate residual states to reveal what the model would predict if it stopped at that layer
• Tuned Lens: An improvement that learns a separate affine transformation for each layer's residual stream, producing more accurate decoded predictions
• Revelation: Shows how predictions evolve layer by layer, often revealing that the model settles on an answer early and uses later layers for refinement
• Application: Used to study how transformers build predictions incrementally and to identify which layers are responsible for specific output components
• Key Finding: Many factual associations are resolved in middle MLP layers, with later attention heads primarily performing syntactic cleanup

An unsupervised architecture that decomposes dense, polysemantic activations into a sparse set of interpretable, monosemantic features.

• Training Objective: Reconstructs a layer's activations while enforcing sparsity in the latent representation via an L1 penalty
• Dictionary Learning: Learns an overcomplete basis of feature directions where each direction corresponds to a single human-interpretable concept
• Addressing Polysemanticity: Resolves the problem where individual neurons respond to multiple unrelated concepts by disentangling them into separate features
• Scale: Modern sparse autoencoders can discover tens of thousands of distinct features in a single transformer layer
• Validation: Features are validated by showing that activating them causally influences model behavior in predictable ways

Algorithmic methods that automatically identify the minimal subgraph of a neural network responsible for a specific behavior without manual human inspection.

• Approaches: Includes attribution patching, which efficiently estimates the importance of each edge in the computational graph using gradient-based approximations
• Scaling: Enables analysis of circuits across entire models rather than isolated behaviors, moving beyond manual reverse engineering
• Output: Produces a sparse subgraph of attention heads and MLP neurons with their connection weights
• Validation: Discovered circuits are tested with causal scrubbing to verify they are both necessary and sufficient
• Current Frontier: Scaling automated discovery to find circuits for complex behaviors like chain-of-thought reasoning and multi-step problem solving

Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific algorithm. For example, an induction head circuit enables in-context copying by attending to previous tokens that followed similar current tokens. Circuit analysis involves isolating these minimal computational pathways and verifying their function through causal scrubbing.

A causal intervention technique that replaces a model's internal activation at a specific layer and position with a value from a corrupted or alternative forward pass. This isolates the function of individual components by observing how the output changes. Path patching extends this by isolating the direct effect of a specific computational path between two components while freezing all other paths.

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. This addresses the superposition hypothesis, where networks represent more features than dimensions by compressing them. Dictionary learning uses these learned basis vectors to identify distinct, human-interpretable concepts encoded in activations.

Techniques for decoding intermediate model predictions. The logit lens applies the unembedding matrix directly to residual stream activations at each layer, revealing the model's evolving next-token predictions. The tuned lens improves on this by learning an affine transformation for each layer's residual stream, producing more accurate decoded outputs and enabling precise tracking of how predictions form across layers.

A systematic evaluation methodology that tests a hypothesized circuit by replacing all activations outside the circuit with corrupted values. If the model's performance is preserved, the circuit is validated as sufficient for the behavior. This formalizes the hypothesis testing cycle central to reverse engineering neural networks and distinguishes genuine understanding from superficial correlation.

The conjecture that high-level concepts are encoded as linear directions in the representation space of activation vectors. This enables activation engineering, where steering vectors are added to activations during a forward pass to control behavior. Function vectors derived from in-context examples can trigger specific task behaviors without the original prompt, supporting the hypothesis that features are linearly accessible.