Knowledge Neurons

Knowledge neurons encode subject-relation-object triples within their weights. A single neuron or a sparse group of neurons in an MLP layer activates strongly when the model processes a specific fact (e.g., 'The capital of France is Paris'). These neurons act as a key-value memory, where the key is the subject-relation pair and the value is the object. This storage mechanism is distinct from the contextual processing performed by attention heads, which route information rather than store it.

The defining test for a knowledge neuron is causal intervention. By setting a candidate neuron's activation to zero (zero ablation) or its mean value (mean ablation) during a forward pass, researchers can observe a specific degradation in the model's ability to express the associated fact. If suppressing a neuron causes the model to fail on 'The Eiffel Tower is in ___' but not on unrelated facts, that neuron is causally implicated in storing that specific piece of knowledge.

Factual knowledge is predominantly localized in the intermediate and late MLP layers of transformer architectures, not in attention heads. The first MLP layer of a transformer block functions as a key-value associative memory. Research using causal tracing has shown that subject tokens are enriched by attention heads in early layers, but the actual factual retrieval occurs when these enriched representations pass through specific MLP layers in the middle-to-late stages of the network.

Knowledge is stored in a sparse and discrete manner. For any given fact, only a small fraction of all neurons in an MLP layer are active. This sparsity enables the model to store a vast number of facts without catastrophic interference. The phenomenon is linked to the superposition hypothesis, where models represent more features than they have dimensions, but factual knowledge often resolves into monosemantic neurons that correspond to a single, interpretable concept.

Because knowledge neurons are localized, they can be surgically modified to update facts without retraining. Techniques like ROME (Rank-One Model Editing) identify the specific MLP weight matrix responsible for a fact and apply a rank-one update to alter the stored association. This allows changing a model's knowledge from 'LeBron James plays for the Lakers' to 'LeBron James plays for the Cavaliers' with minimal disruption to other stored facts.

Knowledge neurons are identified by analyzing activation patterns across diverse prompts. A neuron is classified as a knowledge neuron if its activation is consistently high when the model processes the subject of a known fact, regardless of the surrounding context. The knowledge attribution score is computed by measuring the mutual information between a neuron's activation and the model's correct prediction of the factual object across many paraphrased prompts.

Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific, human-understandable algorithm. Circuit analysis moves beyond single-neuron explanations to understand the collaborative computations that produce complex behaviors.

• Induction heads are a classic example, enabling in-context copying
• Identified through causal interventions like activation patching
• Validated using causal scrubbing to ensure completeness

A causal intervention technique that replaces a model's internal activation at a specific layer and position with a value from a corrupted or alternative forward pass to isolate its function. This allows researchers to pinpoint which components are causally responsible for a behavior.

• Clean vs. corrupted run: Compare outputs to measure effect
• Path patching extends this to isolate specific computational paths
• Essential for building and validating mechanistic circuits

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. This addresses the challenge of superposition, where neurons represent multiple concepts simultaneously.

• Learns an overcomplete basis of feature directions
• Enables dictionary learning for neural networks
• Cross-coders extend this to compare features across models

Techniques that decode intermediate residual stream activations into next-token predictions by applying the unembedding matrix directly. The logit lens reveals how predictions evolve layer by layer, while the tuned lens learns affine transformations for more accurate early-layer decoding.

• Provides a window into the model's evolving beliefs
• Helps identify where specific computations occur
• Reveals the progressive refinement of predictions

A phenomenon where a model abruptly transitions from memorizing a training dataset to perfectly generalizing a clean, general solution after a prolonged period of overfitting. This phase change reveals insights into how neural networks learn underlying algorithms.

• Discovered in small algorithmic tasks like modular arithmetic
• Characterized by a sudden drop in validation loss
• Suggests models can discover parsimonious circuits given enough time

A causal tracing methodology to identify the specific MLP layers storing a fact and then surgically modifying those weights to update the stored association. This demonstrates that factual knowledge is localized and editable.

• Uses causal tracing to find knowledge neurons
• Enables model editing without full retraining
• Validates the mechanistic view of factual storage