Feature Visualization

Feature visualization is fundamentally an optimization process rather than a feedforward pass. Starting from random noise, the input is iteratively updated via gradient ascent to maximize the activation of a target neuron, channel, or layer. This transforms the network from a passive classifier into a generative model, producing a prototypical exemplar of what the unit detects. The optimization objective is typically the mean activation of the target unit, with additional regularization terms to produce natural-looking results.

Without constraints, optimization produces high-frequency noise patterns that maximally activate neurons but are uninterpretable to humans. Key regularization techniques include:

• Total variation regularization: Penalizes adjacent pixel differences to encourage spatial smoothness
• L2 decay: Prevents individual pixel values from exploding
• Frequency penalization: Suppresses high-frequency noise in the Fourier domain
• Transformation robustness: Applies small random jitters, rotations, and scaling during optimization to force the visualization to be invariant to these transforms
• Learned priors: Uses a separately trained generator network to constrain the output to the manifold of natural images

Feature visualization enables circuit analysis by revealing how features combine across layers. By visualizing neurons at successive depths, researchers can trace the feature hierarchy: early layers detect edges, textures, and colors; middle layers combine these into patterns like curves, circles, and repeating motifs; deep layers assemble complex, semantically meaningful structures such as faces, wheels, or text. This technique has been instrumental in identifying polysemantic neurons that respond to multiple unrelated concepts and superposition where more features are represented than there are dimensions.

A single optimization run reveals only one facet of what a neuron detects. To capture the full range of stimuli that excite a unit, practitioners employ diverse initializations—starting from different random seeds or real images from the training set. This produces a visual vocabulary of patterns that collectively characterize the neuron's receptive field. Clustering these diverse visualizations can reveal whether a neuron is monosemantic (responding to one coherent concept) or polysemantic (responding to multiple distinct patterns).

Feature visualization operates at multiple levels of granularity:

• Neuron-level: Maximizes a single scalar activation, revealing the precise pattern that excites one unit
• Channel-level: Maximizes an entire convolutional channel, showing the spatial pattern that the filter convolves with
• Layer-level: Maximizes the L2 norm of an entire layer's activation vector, producing an image that represents the canonical input for that processing stage
• Logit-level: Maximizes a specific output class before the softmax, generating the network's idealized version of that category—often revealing learned biases and dataset artifacts

Feature visualization serves as a critical tool for AI safety and alignment research. By revealing what internal representations correspond to, researchers can:

• Detect spurious correlations where models latch onto background features rather than the intended concept
• Identify backdoor triggers by visualizing neurons that activate for adversarial patterns
• Audit models for unintended feature learning such as sensitive attributes in supposedly anonymized representations
• Validate that concept bottleneck models have learned the intended intermediate concepts before trusting downstream decisions

An attribution method that assigns importance to input features by accumulating gradients along a straight-line path from a baseline to the actual input.

• Satisfies the completeness axiom: attributions sum to the output difference
• Baseline choice is critical—a zero-signal or noise baseline is common in RF applications
• Effective for identifying which time-frequency regions trigger a specific emitter identification

A decomposition technique that redistributes a neural network's prediction score backwards through layers using a conservation property to assign relevance to individual input variables.

• Ensures total relevance is preserved across layers
• Produces heatmaps directly on raw IQ or spectrogram inputs
• Well-suited for debugging false positives in signal detection networks

A visualization technique that uses gradients flowing into the final convolutional layer to produce a coarse localization map highlighting important input regions.

• Generates class-discriminative saliency maps without architectural changes
• Widely adapted for spectrogram-based RF classification
• Reveals whether a model focuses on the signal of interest or background noise artifacts

A causal explanation method that identifies the minimal change to an input required to alter a model's prediction to a predefined alternative outcome.

• Answers: 'What would need to change for this signal to be classified as QAM16 instead of QPSK?'
• Generates actionable diagnostic insights for RF system debugging
• Helps establish decision boundaries in high-dimensional signal space

A subfield that reverse-engineers neural network internals into human-understandable algorithms, treating the model as a scientific object of study.

• Goes beyond feature attribution to identify circuits and polysemantic neurons
• Applied to RF models to discover if networks learn known signal processing operations (e.g., matched filtering, Fourier transforms)
• Critical for mission assurance in defense and spectrum management applications