Superposition Hypothesis

Models exploit high-dimensional geometry to store more features than neurons. Instead of a 1:1 mapping, features are encoded as nearly orthogonal vectors in activation space.

• Johnson-Lindenstrauss Lemma: Random projections preserve distances, allowing many vectors to exist almost orthogonally.
• Interference: Slight non-orthogonality causes features to interact, creating the primary challenge for interpretability.
• Capacity: A model with n dimensions can theoretically represent an exponential number of sparse features.

A direct consequence of superposition is the polysemantic neuron. A single neuron fires for multiple, seemingly unrelated input patterns.

• Shared Activation Space: The neuron participates in multiple feature representations simultaneously.
• Interpretability Barrier: Directly labeling a neuron becomes impossible; its activation map is a tangled mixture of concepts.
• Resolution: Techniques like sparse autoencoders are required to disentangle these superimposed features into a monosemantic basis.

Feature vectors are not perfectly orthogonal. The residual 'noise' from one feature's activation can interfere with the computation of another.

• Dot Product Noise: The interference is proportional to the dot product between the feature vectors.
• Error Correction: Models learn to tolerate or correct this noise, but it limits the total number of features that can be safely packed.
• Phase Change: As the feature-to-dimension ratio increases, the model undergoes a sharp transition from clean representation to a noisy, superimposed state.

Superposition is computationally viable only because features are sparse—only a tiny fraction are active at any given time.

• Statistical Leverage: If 99% of features are zero, the active set can be uniquely identified despite compression.
• Compressed Sensing Link: The model acts as a compressed sensing system, recovering a sparse signal from a lower-dimensional measurement.
• Capacity Scaling: The maximum number of storable features scales inversely with the density of feature activation.

Research uses toy models with synthetic data and known ground-truth features to prove superposition occurs.

• Controlled Setting: A small ReLU network is trained to reconstruct a high-dimensional sparse input from a low-dimensional bottleneck.
• Emergent Behavior: The model consistently learns to represent more features than bottleneck dimensions, confirming the hypothesis.
• Weight Analysis: The learned embedding matrix shows distinct, non-orthogonal vectors for each ground-truth feature, directly visualizing the superposition.

Superposition fundamentally reframes the goal of interpretability from 'understanding neurons' to 'disentangling features'.

• Dictionary Learning: The primary tool becomes finding an overcomplete basis of interpretable features.
• Causal Intervention: Patching a single concept requires identifying and manipulating its specific direction in the superimposed space, not just a neuron.
• Universality: The hypothesis suggests superposition is a universal property of any sufficiently compressed, high-performing neural network.