Superposition Hypothesis

Features are represented by vectors that are nearly but not perfectly orthogonal to each other. This allows a d-dimensional space to represent n > d features by exploiting the exponential volume of high-dimensional spaces. The dot product between feature vectors is non-zero but small, creating a slight interference that the model tolerates to gain representational capacity. This is a direct consequence of the Johnson-Lindenstrauss lemma, which guarantees that many almost-orthogonal vectors can exist in relatively few dimensions.

When superposition occurs, individual neurons become polysemantic—they activate for multiple unrelated input features. A single neuron might fire for both 'academic citations' and 'rabbits', making direct neuron-level interpretation impossible. This is the primary obstacle that mechanistic interpretability seeks to overcome, as it prevents assigning a single human-understandable label to any given neuron.

Superposition is only computationally viable when features are sparsely activated. The model leverages the fact that at any given time, only a small fraction of all possible features are active. This sparsity allows the model to use the same dimensions to represent different features at different times, with the interference noise from overlapping representations remaining manageable when few features fire simultaneously.

The model faces a fundamental tension:

• More features per dimension increases representational power but adds interference noise
• Less compression preserves signal fidelity but wastes model capacity
• The optimal balance depends on feature sparsity and importance This trade-off explains why models allocate more dimensions to frequent, high-importance features and compress rare features into superposition.

Sparse autoencoders (SAEs) are the primary tool for disentangling superimposed features. By learning an overcomplete basis of monosemantic feature vectors, SAEs decompose dense activations into a sparse combination of interpretable directions. This transforms the polysemantic mess back into distinct, labelable features. The learned dictionary typically contains many more features than the original activation dimensionality.

The superposition hypothesis was validated using toy models trained on synthetic data with known ground-truth features. Researchers demonstrated that when the number of features exceeds available dimensions, models spontaneously learn to represent them in superposition rather than ignoring some features entirely. The learned representations exhibit predictable geometric structures like pentagonal or hexagonal tilings in 2D projections.

A single neuron that activates in response to multiple unrelated input features, making it impossible to assign a single human-interpretable label. This is the primary empirical observation that motivated the Superposition Hypothesis—neurons are not the fundamental unit of representation. A neuron might fire for both French text and curly brackets in code, with no obvious shared semantic meaning. The model exploits the fact that these features rarely co-occur in training data, allowing it to reuse the same dimension for distinct concepts without catastrophic interference.

An unsupervised technique that decomposes a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features using a learned overcomplete basis. Key properties:

• Overcomplete basis: More dictionary elements than input dimensions, enabling the model to represent superposed features explicitly
• Sparsity constraint: An L1 penalty on hidden activations forces the representation to use only a small subset of features for any given input
• Reconstruction loss: The decoder must faithfully reproduce the original activation from the sparse code SAEs are the primary tool for empirically validating the Superposition Hypothesis by extracting the ground-truth features a model has compressed.

A decomposition method that learns an overcomplete basis of vectors to represent activations as a sparse linear combination of interpretable features. The mathematical objective is to find a dictionary D and sparse codes c such that x ≈ Dc where ||c||₀ is small. In the context of superposition, dictionary learning formalizes the inverse problem: given the compressed, entangled activations, recover the independent features the model is actually tracking. Implementations range from classical algorithms like K-SVD to modern neural approaches using sparse autoencoders with tied or untied weights.

The primary information highway in a transformer where each layer reads from and writes to a shared accumulating state. The Superposition Hypothesis applies directly to this high-dimensional space—the residual stream's dimensionality creates the capacity for superposition. Key dynamics:

• Each attention and MLP layer adds its output to the stream via residual connections
• Features from different layers can coexist in near-orthogonal subspaces
• The stream's width determines how many features can be superposed before interference becomes destructive
• Logit lens techniques probe this stream to reveal how superposed features evolve across layers

A phenomenon where a single interpretable feature in a smaller model splits into multiple distinct features in a larger model with more dimensions. This provides direct evidence for the Superposition Hypothesis: as capacity increases, the model allocates dedicated dimensions to features that were previously compressed together. For example, a feature representing positive sentiment in a small model might split into separate features for joy, gratitude, and satisfaction in a larger model. This splitting is observed by training SAEs of increasing width and tracking how feature dictionaries evolve.

The fundamental trade-off at the heart of superposition: representing more features than dimensions introduces interference between non-orthogonal feature directions. The model manages this through:

• Feature sparsity: If features are rarely active simultaneously, interference is minimized even with non-orthogonal encoding
• Importance weighting: Critical features get more orthogonal, higher-magnitude representations; rare features are compressed more aggressively
• Dimensionality expansion: Wider layers can support more superposed features before interference degrades performance
• Phase transitions: Theory predicts a sharp transition where models switch from dedicated to superposed representation as the feature-to-dimension ratio exceeds a critical threshold