Feature Visualization

The foundational technique for generating class model visualizations by performing gradient ascent in input space. Starting from random noise, the input is iteratively optimized to maximize the activation of a target neuron or class output.

• Objective: Find input x* that maximizes neuron activation f(x) subject to regularization
• Process: Iteratively compute gradient of target neuron w.r.t. input and update input pixels
• Result: Reveals the canonical prototype a neuron has learned to detect
• Key insight: The resulting image shows what the network 'wants to see', not necessarily what exists in training data

Unconstrained optimization produces adversarial noise rather than interpretable features. Transformation robustness applies small random jitters, rotations, and scales during optimization to force the visualization to be invariant to these transforms.

• Jitter: Randomly shift the image by a few pixels each iteration
• Rotation: Apply small random rotations (±5-10 degrees)
• Scaling: Randomly scale the image slightly up or down
• Effect: Produces naturalistic, coherent patterns instead of high-frequency noise
• Mechanism: The neuron must learn features robust to these transformations, matching how real visual concepts behave

Without intervention, multiple optimization runs for the same neuron converge to similar patterns. Diversity regularization penalizes similarity between visualizations to reveal the full range of features a neuron detects.

• Cosine similarity penalty: Discourage activations that are too similar in feature space
• Result: Uncovers polysemantic neurons that respond to multiple distinct concepts
• Application: Essential for understanding neurons in later layers that combine multiple features
• Revelation: Many neurons are not single-concept detectors but respond to diverse, sometimes unrelated patterns

Unregularized optimization produces high-frequency adversarial patterns that exploit network vulnerabilities rather than revealing learned features. Frequency penalization suppresses these artifacts.

• Total Variation (TV) loss: Penalizes differences between adjacent pixels, encouraging smooth regions
• Gaussian blur: Apply slight blurring between optimization steps
• Fourier space regularization: Directly penalize high-frequency components
• Bilateral filtering: Preserve edges while smoothing textures
• Purpose: Separates genuine learned features from gradient artifacts and noise

Extends feature visualization to understand how individual convolutional channels contribute to class decisions. By combining activation maximization with attribution methods, each channel's visual role becomes explicit.

• Channel-wise optimization: Synthesize inputs that maximize specific filter activations
• Attribution weighting: Weight channels by their importance to a class prediction
• Composite visualization: Combine channel visualizations proportionally to their contribution
• Application: Debugging failure modes by identifying which channels drive incorrect predictions

A critical distinction in feature visualization: dataset examples show real training images that activate a neuron, while prototypes show the idealized synthetic pattern. Comparing both reveals the gap between what the network encounters and what it learns.

• Dataset examples: Real images from training data that maximally activate the neuron
• Prototypes: Synthetic images from activation maximization
• Divergence analysis: Large gaps indicate the neuron has learned an artifact or shortcut rather than the intended concept
• Diagnostic value: Identifies spurious correlations and dataset biases

The foundational optimization process that directly synthesizes an input pattern to maximally excite a target neuron or class output. By performing gradient ascent in the input space, it generates a preferred stimulus, often starting from random noise. This technique is the engine behind feature visualization, revealing the archetypal patterns a network has learned to detect. Regularization is critical to prevent the generation of high-frequency, adversarial noise that is unrecognizable to humans.

A heatmap highlighting the input features that most influence a prediction for a single, specific input. Unlike feature visualization, which generates a generic ideal stimulus, a saliency map answers 'which pixels in this image caused this classification?'. It is typically computed as the gradient of the output with respect to the input, providing a local, instance-specific explanation. Key variants include Gradient × Input and SmoothGrad.

An axiomatic attribution method that computes feature importance by accumulating gradients along a straight-line path from a neutral baseline (e.g., a black image) to the actual input. It satisfies the Completeness axiom, meaning the sum of attributions equals the difference in the model's output. This provides a more robust and theoretically grounded saliency map than raw gradients, mitigating the gradient saturation problem where strong features receive near-zero gradients.

A technique for producing coarse, class-discriminative localization maps. It uses the gradients of a target concept flowing into the final convolutional layer to weight the activation maps, highlighting the image regions most relevant to the prediction. Unlike pixel-space saliency maps, Grad-CAM provides a structural, object-level explanation, making it highly effective for debugging CNN-based image classifiers and visual question-answering models.

A creative visualization technique that enhances the patterns detected by a specific layer in a network. Instead of maximizing a single neuron, DeepDream modifies an input image to amplify the features that a layer already 'sees', creating a feedback loop. This process iteratively applies gradient ascent to the layer's total activation, producing surreal, hallucinogenic imagery that reveals the network's learned feature hierarchy in a visually striking way.

A set of constraints applied during the optimization process to produce human-interpretable visualizations. Without regularization, activation maximization generates high-frequency noise that is mathematically optimal but visually meaningless. Common techniques include total variation (penalizing pixel discontinuities), Gaussian blurring (removing fine-grained noise), and transformation robustness (enforcing stability to small jitters or rotations), steering the optimization toward natural-looking patterns.