Gradient Saturation

Gradient saturation occurs when a feature's value pushes the network's output into a flat region of the activation function. For a feature that strongly activates the correct class, the local gradient becomes near-zero, causing the attribution method to incorrectly assign it an importance score of zero. This violates the Sensitivity-n axiom, as the model is functionally dependent on the feature, yet the gradient fails to reflect this.

The phenomenon is rooted in the saturating regions of activation functions like sigmoid or tanh, and in the final softmax layer. When a logit $z_c$ for the target class is very high, the softmax gradient $\frac{\partial y_c}{\partial z_c} = y_c(1 - y_c)$ approaches zero. Consequently, any input feature that contributed to that high logit receives a vanishing gradient, making Gradient × Input and raw saliency maps blind to the most predictive evidence.

Visual saliency maps generated from raw gradients often fail to highlight the primary object in an image, instead focusing on irrelevant background pixels. This is because the pixels belonging to the object drive the correct class logit into saturation, causing their gradients to vanish. Techniques like Integrated Gradients and DeepLIFT were specifically designed to overcome this failure mode by considering the path from a baseline rather than a single point estimate.

Gradient saturation is distinct from shattered gradients. Saturation implies a smooth, near-zero gradient in flat loss regions, while shattered gradients resemble high-frequency white noise with large magnitudes but no coherent structure. Saturation is a problem of scale, whereas shattering is a problem of noise. Both render raw gradients useless for interpretability, but they require different mitigation strategies.

Several attribution methods are designed to bypass saturation:

• Integrated Gradients: Accumulates gradients along a path from a non-saturating baseline (e.g., a black image) to the input, capturing the feature's effect before saturation occurs.
• DeepLIFT: Uses a reference activation and difference-from-reference rules to propagate relevance scores, avoiding the zero-gradient issue.
• Expected Gradients: Averages gradients over a background distribution, smoothing out saturation artifacts.
• FullGrad: Incorporates bias gradients to capture contributions that bypass saturated input pathways.

To detect gradient saturation in your model:

• Check gradient magnitudes: If the L2 norm of the input gradient is near zero for high-confidence predictions, saturation is likely.
• Compare with baselines: Run Integrated Gradients with multiple baselines. If the attributions differ dramatically from raw gradients, saturation is confirmed.
• Analyze logit distributions: A highly peaked softmax output (e.g., probability > 0.99) is a strong indicator that the pre-softmax logits are in a saturating regime.