Feature Visualization

Rather than selecting an image from a dataset, feature visualization generates an input from scratch by iteratively updating pixels to maximize a target activation. Starting from random noise, the process uses gradient ascent on the input space, treating the chosen neuron's activation as the objective function. This reveals the ideal, platonic form of what the neuron detects, unconstrained by natural image statistics.

Unconstrained optimization often produces high-frequency noise patterns that are unrecognizable but maximally activating. To produce human-interpretable visualizations, a suite of regularization priors is essential:

• Total Variation (TV): Penalizes pixel-to-pixel variation to smooth the image
• L2 Decay: Prevents pixel values from exploding to extreme intensities
• Transformation Robustness: Applies small jitters, rotations, or scaling during optimization to force the visualization to be stable under geometric transforms
• Frequency Penalization: Suppresses adversarial high-frequency patterns

Feature visualization can target different levels of abstraction within a network:

• Individual Neurons: Visualizes what a single neuron in an MLP or convolutional filter detects
• Entire Channels: Optimizes for the mean activation across a full feature map channel, common in CNNs
• Logit Units: Synthesizes the prototypical input for a specific output class before the softmax
• Arbitrary Directions: Visualizes any linear combination of activations, enabling exploration of semantic feature vectors discovered through techniques like PCA or dictionary learning

A single neuron can detect multiple facets of a concept. To capture this polysemanticity, feature visualization employs diverse random initializations and optionally penalizes similarity between generated images. This produces a gallery of distinct patterns that collectively characterize the neuron's full receptive repertoire, revealing whether it is monosemantic (one concept) or polysemantic (multiple unrelated concepts).

Beyond single neurons, feature visualization can be applied to entire circuits—connected subgraphs of attention heads and MLP layers. By optimizing an input to simultaneously maximize the activations of all components in a hypothesized circuit, researchers can synthesize the composite concept the circuit computes. This bridges the gap between mechanistic interpretability and visual understanding, validating circuit hypotheses with concrete visual evidence.

Feature visualization has known failure modes that require careful interpretation:

• Adversarial Noise: Without sufficient regularization, the result is a texture of high-frequency noise invisible to humans but perfectly activating the neuron
• Mode Collapse: The optimization converges to a single narrow pattern, missing the neuron's full representational range
• Interpretability Gap: A human-recognizable visualization does not guarantee the neuron's actual function matches human intuition—it may be a coincidental correlate
• Layer Depth Effects: Early layers produce simple textures and edges; deeper layers synthesize complex, compositional scenes

The discipline of reverse-engineering a neural network's learned algorithms and internal computations from its weights and activations into human-understandable components. Feature visualization is a foundational tool in this field, used to generate hypotheses about what individual neurons or feature directions represent. The ultimate goal is to decompose a model into circuits—sparse, interpretable subgraphs that implement specific, human-understandable algorithms.

The observed phenomenon where a single neuron or feature direction in a neural network responds to multiple, seemingly unrelated input concepts. For example, a neuron might activate for both cat faces and car fronts. This complicates feature visualization because an optimized input may show a superposition of unrelated patterns. The existence of polysemanticity motivates the use of sparse autoencoders to disentangle these mixed representations into monosemantic features.

Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific, human-understandable algorithm. Feature visualization is used to annotate the nodes of a circuit by revealing what each component detects. For instance, in a vision model, a circuit for detecting curves might feed into a circuit for detecting loops, which then feeds into a face detector.

A hypothesized phenomenon where a neural network represents more independent features than it has dimensions in a given layer, compressing sparse features into a lower-dimensional space. This is a direct challenge to feature visualization because a single activation vector may encode dozens of concepts in a compressed, entangled format. Dictionary learning and sparse autoencoders are the primary methods for decompressing these representations into visualizable, monosemantic features.

The practice of directly modifying a model's internal activations during a forward pass by adding steering vectors to control its behavior without prompt engineering. Feature visualization helps identify the correct activation directions to modify. For example, visualizing the feature direction for 'sycophancy' allows researchers to then subtract that vector from the residual stream, making the model more honest in its responses.

An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features. Once trained, each latent dimension of the autoencoder corresponds to a single concept. Applying feature visualization to these latent dimensions produces far cleaner, more interpretable images than visualizing raw neurons, directly addressing the superposition problem.