Monosemanticity

A monosemantic neuron activates exclusively for a single, natural concept regardless of context. For example, a neuron might fire for 'curly brackets' in code, photographs of curly brackets, and the text '{ }', but never for other punctuation or unrelated patterns. This stands in direct opposition to polysemanticity, where a single neuron responds to multiple unrelated concepts like 'cat faces' and 'car hoods'. True monosemanticity implies the model has dedicated a distinct computational unit to a specific feature.

Monosemanticity is grounded in the Linear Representation Hypothesis, which conjectures that high-level concepts are encoded as linear directions in a model's activation space. Under this framework, a monosemantic neuron is simply a basis vector aligned with a single interpretable feature. The model's internal state can be decomposed into a sparse linear combination of these features. This hypothesis is what makes techniques like dictionary learning and sparse autoencoders viable for extracting interpretable features from dense, polysemantic activations.

Since most neurons in large models are polysemantic, monosemantic features are typically discovered using sparse autoencoders. These auxiliary networks are trained to reconstruct a layer's dense activations using a sparse overcomplete basis of learned feature directions. The sparsity constraint forces each basis vector to represent a single, interpretable concept. Key properties of the resulting features include:

• Sparsity: Only a small fraction of features activate for any given input
• Disentanglement: Features are statistically independent
• Interpretability: Each feature can be labeled by a human inspecting its top-activating dataset examples

A critical characteristic of a truly monosemantic feature is that it is causally implicated in the model's output, not merely correlated. This is validated through activation patching experiments: if you artificially clamp the neuron's value to zero or a high value, the model's output should change in a predictable way related to the concept. For instance, clamping a 'golden retriever' neuron should suppress the model's tendency to output dog-related text. Without causal influence, the feature may be an epiphenomenon of the training process rather than a functional unit.

Research suggests that monosemantic features exhibit a degree of universality—different models trained on similar data independently learn similar features. A 'Arabic text' feature or a 'DNA sequence' feature can be found in the activation spaces of distinct language models. This is validated using cross-coders, which are sparse autoencoders trained simultaneously on the activations of two different models to identify shared feature directions. Universality implies these features correspond to genuine statistical structures in the training data rather than arbitrary artifacts of a specific initialization.

Monosemantic features are not atomic—they can exhibit feature splitting, where what appears to be a single concept at one scale decomposes into finer-grained sub-features upon closer inspection. For example, a 'mathematics' feature might split into 'linear algebra', 'calculus', and 'topology' features when a larger sparse autoencoder is trained. Conversely, features can be compositional: a 'neural network' feature may be constructed from more primitive features like 'weights', 'layers', and 'backpropagation'. This hierarchical structure mirrors how human concepts relate to one another.

The observed phenomenon where a single neuron or feature direction responds to multiple, seemingly unrelated input concepts. For example, a neuron in a vision model might activate for both cat faces and car fronts. This is the default state of most neurons in dense models and the primary problem that monosemanticity seeks to solve. Polysemanticity arises because neural networks compress more features than they have dimensions, forcing superposition.

A hypothesized phenomenon where a neural network represents more independent features than it has dimensions in a given layer. The model compresses sparse features into a lower-dimensional space by exploiting the fact that features are sparse—most are zero at any given time. This compression directly causes polysemanticity and makes individual neurons difficult to interpret. Superposition is the mechanism; polysemanticity is the symptom.

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. The autoencoder learns an overcomplete basis of feature directions, where each direction corresponds to a single concept. Key properties include:

• Sparsity constraint: Only a small number of features are active for any input
• Reconstruction fidelity: The original activation must be recoverable from the sparse representation
• Feature interpretability: Each learned direction maps to one human-understandable concept

A method for decomposing a model's activations into a sparse linear combination of learned basis vectors, each representing a distinct, interpretable feature. The goal is to find an overcomplete dictionary where each basis element is monosemantic. This is closely related to sparse autoencoders and is a core technique for extracting monosemantic features from polysemantic layers. The learned dictionary provides a human-readable vocabulary of the model's internal representations.

The conjecture that high-level concepts are encoded as linear directions in the representation space of a neural network's activation vectors. If true, this means that a concept like 'honesty' or 'French text' corresponds to a specific vector direction. Monosemanticity is a special case of this hypothesis: a neuron is monosemantic if its activation direction aligns perfectly with a single concept's linear representation. This hypothesis underpins much of mechanistic interpretability research.

The objective of learning a representation where individual latent dimensions correspond to independent, interpretable generative factors of the data. A perfectly disentangled representation is inherently monosemantic—each dimension controls exactly one factor of variation. Techniques include:

• β-VAE: Adds a hyperparameter to balance reconstruction and latent channel capacity
• FactorVAE: Encourages statistical independence between latent dimensions
• TCVAE: Penalizes total correlation in the latent space