Superposition Hypothesis

Models leverage the high-dimensional geometry of activation space to store more concepts than available neurons. Features are not assigned to dedicated, single neurons but are instead encoded as linear combinations of basis vectors. This allows a layer with n dimensions to represent m > n sparse features by exploiting the fact that random vectors in high-dimensional spaces are almost orthogonal. The compression is lossy, introducing interference between non-orthogonal features.

Superposition is computationally tractable only when features are sparsely activated. If all features fired simultaneously, the interference would destroy the signal. The model relies on the statistical property that only a tiny fraction of all possible features are active for any given input. This sparsity prior is a fundamental assumption; without it, the model is forced to allocate features to orthogonal directions, a state known as privileged basis alignment.

When two non-orthogonal features are active simultaneously, they create residual interference in the activation vector. The model must learn to tolerate or cancel this noise. This interaction is not a bug but a feature of the computation, allowing the model to perform non-linear computations implicitly through vector addition. The degree of interference is a direct trade-off between representational capacity and computational fidelity.

In superposition, individual neurons become polysemantic, activating for multiple unrelated concepts. A single neuron might fire for both 'car' and 'cat' because the model uses the same dimension to represent different features in different contexts. The goal of mechanistic interpretability is to decompose these polysemantic neurons into their constituent monosemantic features—irreducible, independently meaningful concepts—using techniques like sparse autoencoders.

The phenomenon is a direct function of the bottleneck dimension. In very low-dimensional spaces, features are forced into superposition aggressively. As dimensionality increases, the space becomes more forgiving, allowing for cleaner separation. This explains why larger models exhibit more monosemantic neurons; they have more room to allocate orthogonal directions, reducing the pressure to compress features into superposition.

The primary tool for validating superposition is the sparse autoencoder (SAE) . An SAE is trained to reconstruct a layer's activations through a high-dimensional hidden layer with an L1 sparsity penalty. The learned dictionary features of the SAE often correspond to interpretable, monosemantic concepts, proving that the original activations were a compressed mixture of these features. The ability to successfully decompose activations is the strongest evidence for the hypothesis.

An unsupervised technique used to decompose a model's dense, polysemantic activations into a sparse set of monosemantic features. By training a network to reconstruct activations through a bottleneck with an L1 sparsity penalty, researchers can identify the independent, interpretable directions that were previously superimposed, effectively disentangling the compressed representations predicted by the hypothesis.

A causal intervention method for testing the functional role of specific directions in activation space. To verify if a feature is truly encoded in superposition, researchers replace a clean activation with a corrupted one at a specific location. The resulting change in model output isolates the causal effect of the superimposed feature, distinguishing it from a mere correlational observation.

A direct technique for interpreting residual stream activations by applying the model's unembedding matrix at intermediate layers. This reveals the model's current 'best guess' for the next token at each stage of computation. It provides a window into how superimposed features are incrementally computed and refined across the depth of the network before the final output.

The property of a neuron or feature direction responding to a single, human-interpretable concept. The Superposition Hypothesis posits that models sacrifice monosemanticity for representational power, leading to polysemantic neurons that fire for multiple unrelated inputs. A core goal of mechanistic interpretability is to reverse this compression and recover the underlying monosemantic features.

A technique to synthesize inputs that maximally activate a specific neuron or feature direction, providing a visual understanding of what it detects. When applied to superimposed neurons, this often reveals a confusing mix of patterns. By optimizing for activation, researchers can generate the archetypal stimulus for a feature, helping to validate whether a disentangled direction is truly monosemantic.

A single neuron that responds to multiple, seemingly unrelated input patterns. This is the primary evidence for the Superposition Hypothesis, demonstrating that a model represents more features than it has dimensions by assigning multiple meanings to one neuron. For example, a single neuron in a vision model might activate for both cat faces and car fronts, two concepts that are mathematically orthogonal in the data.