Mechanistic Interpretability

Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific, human-understandable algorithm. A circuit is the minimal set of model components that are necessary and sufficient for a particular behavior.

• Example: The "induction circuit" in transformers copies patterns from earlier in a sequence
• Key property: Circuits are discovered through causal interventions, not just correlation
• Goal: Decompose a black-box model into a collection of verified, composable algorithms

A hypothesized phenomenon where a neural network represents more independent features than it has dimensions in a given layer, compressing sparse features into a lower-dimensional space. This explains why individual neurons often appear polysemantic.

• Mechanism: Features are stored as almost-orthogonal vectors in a compressed space
• Consequence: A single neuron can participate in representing dozens of unrelated concepts
• Resolution: Sparse autoencoders are used to decompress these superimposed features into interpretable, monosemantic directions

A causal intervention technique that replaces a model's internal activation at a specific layer and position with a value from a corrupted or alternative forward pass to isolate its function. This is the primary experimental tool for circuit discovery.

• Process: Run the model on a clean input, then patch in activations from a corrupted run
• Purpose: Determine if a specific component is causally necessary for a behavior
• Variants: Includes path patching, which isolates specific computational paths between components by freezing all other routes

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. This is the leading approach for resolving superposition.

• Architecture: Encoder maps activations to a higher-dimensional sparse latent space; decoder reconstructs the original activation
• Sparsity constraint: An L1 penalty on the latent activations forces most features to be zero
• Result: Each active latent dimension corresponds to a single human-interpretable concept, enabling dictionary learning for neural networks

A systematic evaluation methodology that tests a hypothesized circuit by replacing all activations outside the circuit with corrupted values and verifying the model's performance is preserved. This is the gold standard for circuit validation.

• Hypothesis testing: If the circuit is correct, scrubbing everything else should not degrade performance
• Process: Identify a candidate circuit, corrupt all other activations, measure output fidelity
• Rigor: Unlike ablation, scrubbing preserves the circuit's internal structure while destroying alternative pathways

Techniques that decode intermediate residual stream activations into next-token predictions by applying the unembedding matrix at each layer. This reveals how the model's predictions evolve through the forward pass.

• Logit Lens: Directly applies the final unembedding matrix to each layer's residual stream
• Tuned Lens: Learns an affine transformation for each layer to produce more accurate decoded predictions
• Insight: Reveals the iterative refinement process—early layers propose candidates, later layers disambiguate and confirm

Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific, human-understandable algorithm. Circuit analysis moves beyond single-neuron explanations to understand how features are composed through weights. A classic example is the induction head circuit in transformers, which enables in-context learning by attending to previous occurrences of a token sequence.

A hypothesized phenomenon where a neural network represents more independent features than it has dimensions in a given layer. This compresses sparse, high-dimensional features into a lower-dimensional space by exploiting near-orthogonality in high dimensions. Superposition explains why individual neurons often appear polysemantic—responding to multiple unrelated inputs—and is a central challenge for achieving monosemanticity through techniques like sparse autoencoders.

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. By enforcing a sparsity penalty on the latent representation, SAEs identify a set of dictionary features that form an overcomplete basis. This is the primary tool for dictionary learning in large language models, aiming to extract the true feature directions from the linear representation hypothesis.

A causal intervention technique that replaces a model's internal activation at a specific layer and position with a value from a corrupted or alternative forward pass to isolate its function. By comparing model outputs with and without the patch, researchers can localize where a specific computation occurs. Path patching extends this by isolating the direct effect of a computational path between two components, while causal scrubbing validates hypothesized circuits by corrupting all activations outside the proposed circuit.

Techniques for decoding intermediate model predictions. The logit lens applies the unembedding matrix directly to residual stream activations at each layer, revealing the model's evolving next-token predictions. The tuned lens improves this by learning an affine transformation for each layer to produce more accurate decoded outputs. These methods provide a window into the iterative refinement process occurring in the residual stream.

A phenomenon where a model abruptly transitions from memorizing a training dataset to perfectly generalizing a clean, general solution after a prolonged period of overfitting. First observed in small algorithmic tasks, grokking demonstrates that validation accuracy can suddenly jump from chance to perfect long after training loss has saturated. It suggests that models can learn generalizing circuits that compete with memorization pathways, with the generalizing solution eventually winning out through weight decay.