Knowledge Neuron

A diagnostic technique that trains a simple linear classifier on top of a frozen model's internal representations to test what information is encoded at a specific layer. By keeping the probe simple, researchers ensure that any extracted knowledge was already linearly separable in the model's activations, not introduced by the probe itself.

• Used to verify if factual knowledge is localized in specific layers
• A high probe accuracy indicates the representation contains the target concept
• Contrasts with non-linear probing, which can memorize the task independently

A method for identifying the specific hidden states within a transformer that are causally responsible for recalling a fact. The technique systematically restores clean activations from a corrupted forward pass, one layer at a time, to measure where the model's ability to output the correct fact is recovered.

• Pinpoints the exact MLP layers where factual associations are stored
• Demonstrates a causal, not just correlational, link between a state and an output
• Foundational to the original knowledge neuron identification research

An unsupervised neural network trained to reconstruct a model's internal activations while enforcing a sparsity constraint on its hidden layer. This forces the autoencoder to decompose polysemantic neurons—which fire for multiple unrelated concepts—into a set of independent, monosemantic features that each correspond to a single interpretable concept.

• Addresses the superposition hypothesis by finding an overcomplete feature basis
• A single knowledge neuron may be decomposed into dozens of distinct factual features
• Enables more granular control and understanding than raw neuron analysis

A causal intervention method that replaces a model's internal activation at a specific location—such as a particular layer's MLP output—with a cached activation from a different input. By measuring the resulting change in the model's output, researchers can localize where a specific computation occurs.

• Directly tests the functional role of a suspected knowledge neuron
• If patching a neuron's activation changes the predicted fact, it is causally implicated
• Used to build circuit diagrams of how information flows through a model

A decoding technique that applies the unembedding matrix directly to intermediate residual stream states, bypassing the final layer norm and remaining transformer blocks. This allows researchers to inspect the model's iterative token predictions at every layer, revealing how a fact is gradually constructed before the final output.

• Exposes the step-by-step refinement of a prediction
• A knowledge neuron's effect can be seen as a sudden spike in the correct token's logit
• Useful for identifying the exact layer where a fact becomes fully represented

A statistical framework for quantifying how much a model's output depends on a specific intermediate representation. It measures the indirect effect of an input on the output through a mediator variable—here, a neuron or activation—while controlling for other pathways.

• Provides a formal average treatment effect for a neuron's influence
• Distinguishes between a neuron that is merely correlated with an output and one that mediates it
• Used to validate that a knowledge neuron is a necessary step in factual recall