Knowledge Neuron

A knowledge neuron encodes a specific factual triplet (subject, relation, object) within its weights. Causal tracing reveals that restoring the activation of a single neuron in an early MLP layer can recover a corrupted fact. For example, a neuron in a specific layer might be primarily responsible for the fact "The Eiffel Tower is located in Paris." This localization is the basis for rank-one model editing (ROME) techniques.

Knowledge neurons function as a linear associative memory. The input weights of the MLP layer act as a key, detecting the subject and relation in the residual stream. The neuron's activation then writes a value—the object of the fact—into the residual stream via its output weights. This mechanism is formalized by viewing the MLP layer as a two-layer linear network where facts are stored as rank-one updates to the weight matrix.

Knowledge neurons exhibit high selectivity, activating strongly for a narrow set of related prompts. A neuron storing "Steve Jobs founded Apple" will fire for "Who created Apple?" but remain dormant for unrelated queries. This sparsity is crucial for model editing, as it allows surgical modification of a single fact without causing catastrophic interference with other stored knowledge. Sparse autoencoders can further decompose these neurons into monosemantic features.

Knowledge neurons are predominantly found in the early to middle MLP layers of transformer models. Early layers extract low-level entity attributes, while middle layers encode high-level relational facts. Causal tracing experiments show that factual recall has a distinct causal pathway: early attention heads copy subject tokens, and subsequent MLP layers retrieve associated attributes. This layered architecture mirrors a multi-step retrieval process.

The GELU activation function in MLP layers plays a critical role in knowledge neuron behavior. It acts as a soft threshold, suppressing low-magnitude activations while amplifying those that exceed the threshold. This non-linearity allows the network to selectively activate knowledge neurons only when the input key sufficiently matches the stored pattern, preventing spurious retrieval of unrelated facts and maintaining the sparsity of the knowledge representation.

The knowledge neuron hypothesis is validated through activation patching and causal mediation analysis. By corrupting the subject token's embedding and then restoring the clean activation of a candidate neuron, researchers measure the indirect effect on the model's output probability for the correct object. A high indirect effect confirms the neuron is a necessary and sufficient mediator of the factual association, not merely correlated with it.

The primary experimental method used to locate knowledge neurons. This technique corrupts the input embeddings with Gaussian noise and then systematically restores clean hidden states at specific layers and token positions. By measuring the probability of the correct output being restored, researchers can identify which MLP layers have the strongest causal effect on factual recall. The method revealed that knowledge is concentrated in a small number of middle-layer MLP neurons rather than being distributed evenly throughout the network.

The computational substrate where knowledge neurons reside. Unlike attention heads that move information between token positions, MLP layers process each token independently through two linear projections separated by a non-linear activation function. The first projection expands the representation into a high-dimensional space where individual neurons can activate for specific patterns. These layers function as the model's long-term factual memory, storing associations learned during pre-training.

A neuron that responds to multiple unrelated concepts, complicating the interpretation of individual units. While a knowledge neuron ideally encodes a single fact, many neurons exhibit polysemanticity—firing for seemingly disconnected inputs. This phenomenon is explained by the Superposition Hypothesis, which posits that models represent more features than they have dimensions by encoding them in nearly-orthogonal directions. Sparse autoencoders are a primary tool for disentangling these superimposed features.

An unsupervised technique for decomposing dense, polysemantic activations into monosemantic features. An SAE learns an overcomplete basis of vectors that can reconstruct the original activation as a sparse linear combination. This sparsity constraint forces the learned features to be more interpretable than raw neurons. When applied to MLP layers, SAEs can reveal the distinct factual concepts encoded within what initially appeared to be a single knowledge neuron.

A causal intervention technique used to validate knowledge neuron function. The method replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass. By patching a candidate knowledge neuron's activation from a clean run into a corrupted run, researchers can measure how much of the factual recall ability is restored. This provides direct causal evidence that the neuron is necessary for expressing that specific knowledge.