Saliency Map

A saliency map is fundamentally a visualization of the gradient of the model's output with respect to the input. For a classification score $S_c$ for class $c$ and an input image $I$, the saliency map $M$ is computed as $M = \frac{\partial S_c}{\partial I}$. This first-order Taylor approximation answers: If I change a single pixel slightly, how much does the model's confidence in the target class change?

• High gradient magnitude = high sensitivity = visually salient region
• Near-zero gradient = the model is locally insensitive to that feature
• The map is class-specific; a different target class produces a different saliency map

Saliency maps require only a single forward and backward pass through the network, making them one of the fastest attribution methods available. This efficiency enables real-time or near-real-time explanation generation during inference.

• No baseline inputs, multiple integrations, or perturbation sampling required
• Computationally $O(1)$ forward-backward passes, unlike Integrated Gradients ($O(n)$ steps)
• Ideal for latency-sensitive debugging in production pipelines
• The trade-off: simplicity comes at the cost of visual noise and gradient saturation artifacts

A critical failure mode occurs when a feature strongly activates the correct class, pushing the output probability near 1.0. At this saturation point, the gradient flattens to near-zero, making the most important features appear invisible on the map.

• The model is confident, so small input perturbations produce negligible output change
• The saliency map incorrectly suggests the feature is irrelevant
• This motivates advanced methods like Integrated Gradients and DeepLIFT, which address saturation by using reference baselines or summation over paths
• Also related to the shattered gradient phenomenon, where gradients resemble white noise in deep networks

Raw saliency maps often appear as noisy, high-frequency speckle patterns rather than coherent regions. This occurs because deep networks learn highly non-linear decision boundaries, causing the gradient to fluctuate rapidly across adjacent pixels.

• SmoothGrad addresses this by averaging gradients from multiple noisy copies of the input
• VarGrad quantifies the uncertainty of the saliency map by computing gradient variance
• Noise is not necessarily an error—it reflects the true local sensitivity of a highly non-linear function
• For human consumption, post-processing with Gaussian smoothing or thresholding is common

Saliency maps are class-discriminative: they highlight what makes the input belong to a specific class, not what makes the input recognizable in general. This distinguishes them from autoencoder-style reconstructions.

• A saliency map for 'cat' highlights whiskers and ears, not the entire animal silhouette
• The map answers 'why this class?' not 'what is in the image?'
• Contrast with activation maximization, which synthesizes an idealized prototype for a class
• This property makes saliency maps useful for debugging misclassifications: you see what the model latched onto, right or wrong

Saliency maps can be computed for any differentiable model, from simple CNNs to large transformer architectures. However, the quality and interpretability of the resulting map depend heavily on the smoothness of the model's loss landscape.

• Works with any architecture supporting automatic differentiation (PyTorch, TensorFlow, JAX)
• Applies to image pixels, word embeddings, tabular features, and audio spectrograms
• Key limitation: saliency maps explain only a local, infinitesimal neighborhood of the input—they do not capture global feature importance or interactions
• For a more complete attribution, pair with methods that satisfy the Completeness Axiom, such as Integrated Gradients or DeepLIFT