Gated Sparse Autoencoder

The foundational architecture that Gated SAEs extend. An SAE learns an overcomplete basis of feature vectors to decompose dense, polysemantic activations into a sparse set of monosemantic features. It uses an encoder to project activations to a higher-dimensional latent space, applies a sparsity penalty (typically L1), and a decoder to reconstruct the original input. The key limitation of standard SAEs is the shrinkage problem: the L1 penalty biases feature magnitudes toward zero, conflating the decision of whether a feature is active with how strongly it fires.

The theoretical motivation driving SAE research. This hypothesis posits that neural networks represent more independent features than they have dimensions by encoding them in almost-orthogonal directions within activation space. Key implications:

• A single neuron can respond to multiple unrelated concepts (polysemanticity)
• Features are compressed into a lower-dimensional manifold
• Sparse autoencoders attempt to disentangle this superposition by projecting into a higher-dimensional space where features can be separated
• Gating mechanisms improve this separation by decoupling detection from magnitude estimation

The broader mathematical framework that SAEs and Gated SAEs implement. Dictionary learning decomposes a data matrix X into a dictionary D and sparse codes C such that X ≈ DC. In mechanistic interpretability:

• The dictionary D is the learned set of feature vectors (decoder weights)
• The sparse codes C are the feature activations for a given input
• The overcomplete nature (more dictionary elements than input dimensions) enables monosemanticity
• Gated SAEs improve this decomposition by introducing a gating function that separates feature selection from magnitude estimation, reducing the interference between the two objectives

A single neuron that activates in response to multiple unrelated input features, making it impossible to assign a single human-interpretable label. This is the core problem that Gated SAEs solve. Examples from language models:

• A neuron firing for both academic citations and geographical locations
• A neuron responding to both positive sentiment and religious terminology
• A neuron activating for mathematical operators and musical notation

Gated SAEs decompose these polysemantic neurons into separate, independently gated feature directions, each representing a single interpretable concept.

The primary limitation of standard SAEs that Gated SAEs directly address. When an L1 sparsity penalty is applied to feature activations, it systematically shrinks all feature magnitudes toward zero, even for features that should be strongly active. This creates a fundamental tension:

• Strong sparsity → better feature separation but severe magnitude underestimation
• Weak sparsity → accurate magnitudes but poor feature separation and dead features

The gating mechanism resolves this by using a separate gating network to determine which features are active (with a sparsity penalty) while allowing the magnitude estimation pathway to operate without shrinkage bias.

The downstream application that Gated SAE features enable. Once activations are decomposed into interpretable features, researchers can trace computational circuits—the minimal subgraphs of a model necessary for specific behaviors. Process:

• Decompose activations at each layer using Gated SAEs
• Identify which features are causally relevant through activation patching
• Trace connections between features across layers
• Validate that the identified circuit is both necessary and sufficient for the behavior

Gated SAEs improve circuit fidelity by providing more accurate feature magnitudes, reducing false negatives in circuit detection.