Dictionary Learning

The learned dictionary contains more feature vectors than the input dimension, violating traditional basis constraints. This overcompleteness allows the representation of superposed features that were previously compressed into a lower-dimensional space. A typical SAE might use a dictionary size 4x to 16x larger than the model's residual stream dimension, enabling the disentanglement of thousands of distinct concepts from a single activation vector.

The core inductive bias forces the model to activate only a tiny fraction of dictionary elements for any given input. This is enforced through:

• L1 regularization on the latent activations during training
• Top-K activation, where only the K largest feature activations are kept and the rest are zeroed out
• Typical sparsity levels target fewer than 100 active features out of tens of thousands, ensuring each feature captures a distinct, interpretable concept

The primary goal is to produce features where each dictionary element responds to exactly one human-understandable concept. This contrasts with the polysemantic neurons found in raw transformer MLP layers, where a single neuron might fire for academic citations, URLs, and DNA sequences simultaneously. Monosemantic features enable direct interpretability, allowing researchers to label features as 'Arabic text', 'HTTP status codes', or 'sycophantic praise'.

The autoencoder is trained to minimize the difference between the original activation and the reconstructed output. The loss function balances two competing objectives:

• MSE reconstruction loss: Ensures the sparse representation captures all meaningful information
• Sparsity penalty: Prevents the model from simply learning the identity function
• Dead features—dictionary elements that never activate—are periodically resampled to maintain dictionary utilization

As the dictionary size increases, coarse features naturally split into finer-grained sub-features. A single 'mathematics' feature in a small dictionary might decompose into separate features for 'calculus', 'linear algebra', and 'number theory' in a larger one. This phenomenon, observed in Anthropic's scaling studies, suggests that larger dictionaries reveal increasingly granular semantic distinctions without requiring manual feature engineering or supervised labels.

Dictionary features are not merely correlational—they support causal manipulation of model behavior. By clamping a specific feature's activation value and reconstructing the modified residual stream, researchers can steer the model's output. For example, amplifying the 'Golden Gate Bridge' feature causes the model to mention the bridge in unrelated contexts, while suppressing the 'refusal' feature can bypass safety training, demonstrating the features' direct role in computation.

The primary implementation vehicle for dictionary learning in neural networks. An SAE is an unsupervised architecture that compresses a model's dense, polysemantic internal activations into a higher-dimensional latent space using an overcomplete basis, then reconstructs the original activation. The key is an L1 sparsity penalty on the latent representation, which forces the model to represent each input as a linear combination of only a few active features. This decomposes superposition into a set of monosemantic features—individual latent dimensions that each correspond to a single, human-interpretable concept. The encoder maps activations to sparse codes; the decoder's weight matrix columns form the learned dictionary of feature vectors.

The theoretical foundation motivating dictionary learning. This hypothesis posits that neural networks represent more independent features than they have dimensions in a given activation space. They achieve this by encoding features in almost-orthogonal directions, exploiting the high-dimensional geometry where exponentially many near-orthogonal vectors can exist. This compression creates polysemantic neurons that fire for multiple unrelated concepts, making direct interpretation impossible. Dictionary learning directly attacks this problem by projecting activations into a space with higher dimensionality than the original, providing enough room for each feature to occupy its own axis and resolve the superposition into distinct, monosemantic components.

A single neuron that activates in response to multiple unrelated input features or concepts, making it impossible to assign a single human-interpretable label. For example, a neuron might fire for both academic citations and Hebrew text, or for both positive sentiment and exclamation marks. This phenomenon is a direct consequence of superposition—the model compresses many features into a shared representational space. Dictionary learning via SAEs resolves polysemanticity by disentangling these mixed representations into a sparse set of monosemantic features, where each latent dimension corresponds to exactly one concept, enabling researchers to build a clean, interpretable feature atlas of the model's internal knowledge.

An architectural refinement of the standard SAE that addresses a critical failure mode: shrinkage. In standard SAEs, the L1 penalty causes the model to systematically underestimate feature magnitudes because it can reduce the penalty by making activations smaller. The gated architecture separates the selection of which features are active from the estimation of their magnitudes using a gating mechanism with a Heaviside step function. The gate determines which features are active (binary decision), while a separate magnitude network estimates their strengths without shrinkage pressure. This yields more accurate feature reconstructions and cleaner separation between active and inactive features.

A technique for extending dictionary learning across the depth dimension of a transformer. While standard SAEs decompose activations at a single layer, features often evolve across layers—a concept detected at layer 5 may be refined at layer 8. A transcoder learns to translate the sparse feature representation of one layer into the feature representation of a subsequent layer, effectively modeling the computational pathway between dictionaries. This enables researchers to track how interpretable features transform through the network, building a layered map of computation. It bridges the gap between static feature atlases and dynamic circuit analysis.

The downstream application that dictionary learning enables. Once activations are decomposed into interpretable features, researchers identify the minimal subgraph of the computational graph responsible for a specific behavior. This involves tracing how features connect across attention heads and MLP layers—for example, mapping how a QK circuit in an attention head selects which tokens to attend to, and how the corresponding OV circuit copies information from source to destination. Dictionary learning provides the vocabulary; circuit analysis writes the grammar, revealing the algorithms the model actually learned rather than what we assumed it learned.