Saliency Maps

The simplest form of saliency map computes the gradient of the target class score S_c with respect to the input image I using a single backward pass. For an input with dimensions W × H × C, the saliency map M is defined as:

M = |∂S_c / ∂I|

• The absolute value ensures both positive and negative influences are visualized
• For RGB images, the maximum absolute gradient across color channels is typically taken
• This method satisfies the sensitivity axiom: a feature receiving a zero attribution must not affect the prediction
• Computationally efficient, requiring only one forward and one backward pass

Raw saliency maps often appear noisy and visually fragmented due to shattered gradients—rapid local fluctuations in the gradient signal caused by the piecewise-linear nature of ReLU networks. Key issues include:

• Gradient saturation: When the prediction score is near 1.0, gradients in saturated regions approach zero, falsely indicating low importance
• High-frequency noise: Small adversarial perturbations in the input can dramatically alter the gradient landscape without changing the prediction
• Lack of spatial coherence: Individual pixel gradients do not capture the importance of contiguous regions or higher-level features

These limitations motivated the development of refined methods like SmoothGrad and Integrated Gradients.

Saliency maps are inherently class-discriminative: they highlight only the input features that contribute to the specific predicted class, not all salient objects in the image. This distinguishes them from class-agnostic methods.

• For an image containing both a cat and a dog, the saliency map for class "cat" highlights feline features while suppressing canine regions
• This property is critical for debugging misclassifications: the map reveals which spurious features the model latched onto
• Contrast with activation-based methods like CAM, which may highlight all discriminative regions regardless of the target class
• The gradient is computed with respect to the logit (pre-softmax) score, not the probability, to avoid gradient saturation from the softmax function

A critical theoretical limitation: vanilla saliency maps violate the implementation invariance axiom. Two functionally identical networks with different parameterizations can produce different attribution maps for the same input.

• A network f(x) and a reparameterized version f'(x) that compute identical outputs for all inputs may yield different gradients at specific points
• This occurs because the gradient operator is sensitive to the specific computational path, even when the function is mathematically equivalent
• Integrated Gradients was explicitly designed to satisfy implementation invariance by accumulating gradients along a path
• This failure mode is particularly problematic when comparing attributions across different model architectures or training runs

Saliency maps are closely related to but distinct from DeconvNet and Guided Backpropagation visualizations. The key difference lies in how gradients are handled at ReLU nonlinearities during the backward pass:

• Vanilla Saliency: Standard backpropagation through ReLU, passing gradients only where the forward activation was positive
• DeconvNet: Uses a deconvolutional network that applies ReLU to the gradient signal itself, zeroing out negative gradients regardless of forward activations
• Guided Backpropagation: Combines both conditions—only backpropagates positive gradients at locations where the forward activation was also positive

Guided Backpropagation produces sharper visualizations but is not class-discriminative and can reconstruct patterns even from randomly initialized networks.

The faithfulness of saliency maps is evaluated using perturbation-based metrics that measure how model predictions change when important pixels are modified:

• Deletion Metric: Remove pixels in descending order of importance and measure the area under the probability curve—a steep drop indicates a faithful map
• Insertion Metric: Incrementally reveal pixels from most to least important into a blurred baseline—a rapid rise indicates quality
• Randomization Sanity Check: Randomizing model weights should produce entirely different saliency maps; if the map remains similar, the method is independent of model parameters and invalid
• Cascade Randomization: Randomizing weights layer by layer from top to bottom should cause progressive degradation of the saliency map quality

A class of attribution techniques designed to satisfy specific mathematical guarantees that vanilla gradients violate. The completeness axiom ensures the sum of feature attributions equals the difference from a baseline.

• Integrated Gradients: Accumulates gradients along a path from baseline to input
• DeepLIFT: Compares activations to a reference state using difference-from-reference rules
• Expected Gradients: Averages attributions over multiple background samples

These methods are preferred for high-stakes audit scenarios requiring theoretical rigor.

Techniques specifically designed for convolutional neural networks that produce coarse localization heatmaps highlighting discriminative image regions.

• Grad-CAM: Uses gradients flowing into the final convolutional layer
• Score-CAM: Eliminates gradient dependency by weighting activation maps with confidence scores
• Grad-CAM++: Improves localization for multiple instances of the same class

These methods answer where a model is looking, not just which pixels matter.

Model-agnostic methods that measure feature importance by systematically modifying or occluding parts of the input and observing the change in prediction.

• Occlusion Sensitivity: Slides a gray patch across the image and records probability drops
• RISE: Averages thousands of random binary masks weighted by model confidence
• Extremal Perturbation: Optimizes for the smallest smooth mask that preserves a prediction

These black-box techniques work on any model but are computationally intensive.

Quantitative protocols for measuring the faithfulness and reliability of explanation maps. A visually pleasing heatmap is not necessarily a correct one.

• Deletion Metric: Removes most-important pixels and measures prediction decay
• Insertion Metric: Adds pixels to a blurred baseline and measures probability increase
• ROAR: Retrains the model after removing important features to test if attributions are causal

Rigorous evaluation prevents reliance on plausible-looking but misleading explanations.

A deeper paradigm that moves beyond input attribution to reverse-engineer the internal circuits and algorithms learned by the network's weights.

• Network Dissection: Aligns hidden units with human-labeled semantic concepts
• Causal Scrubbing: Tests hypothesized circuits by resampling activations
• Activation Maximization: Synthesizes inputs that maximally excite specific neurons

This field treats the model as a scientific object of study rather than a black box.