DeepLIFT (Deep Learning Important FeaTures)

DeepLIFT computes contribution scores by measuring the difference between a neuron's activation for the actual input and its activation for a neutral reference input (e.g., a black image or zero embedding vector). This delta-from-reference approach allows the method to propagate importance signals even when gradients are zero, a critical failure point for standard backpropagation-based saliency methods. The choice of reference is crucial: it represents 'absence' of signal and must be carefully selected per domain.

DeepLIFT satisfies the summation-to-delta axiom, which guarantees that the sum of all feature contributions equals the difference between the model's output for the actual input and its output for the reference input. This provides a complete accounting of the prediction: no importance is lost or created during backpropagation. This property is mathematically equivalent to Shapley values under specific multiplier choices, connecting DeepLIFT to game-theoretic interpretability frameworks.

DeepLIFT defines two primary multiplier rules for distributing contributions across layers:

• Rescale Rule: Assigns contributions proportionally to activation deltas, suitable for layers with non-negative outputs like ReLU.
• RevealCancel Rule: Handles both positive and negative contributions separately, treating them as evidence for and against the output. This prevents cancellation artifacts where opposing signals silently neutralize each other.

These rules are applied per-layer, allowing customized propagation strategies for different network components.

A key advantage of DeepLIFT over Integrated Gradients and raw gradient methods is its ability to assign non-zero importance to features that drive neurons into saturation. In a ReLU network, once a neuron outputs zero, its gradient is zero, masking upstream feature influence. DeepLIFT bypasses this by using the difference-from-reference, allowing it to trace contributions through flat gradient regions. This makes it particularly effective for deep networks with many dead or saturated units.

DeepLIFT is implemented as a modified backpropagation pass that replaces standard gradient calculations with custom multiplier operations. For each operation in the forward graph, a corresponding backward function computes how much each input contributed to the output delta. This modular design allows DeepLIFT to be integrated into existing deep learning frameworks by registering custom gradient-like hooks, making it compatible with complex architectures including LSTMs, Transformers, and convolutional networks.

While both methods address gradient saturation, they differ fundamentally:

• DeepLIFT: Uses a single reference point and finite differences, providing faster computation but requiring careful reference selection.
• Integrated Gradients: Integrates gradients along a continuous path from baseline to input, satisfying additional axioms like implementation invariance but at higher computational cost.

DeepLIFT's discrete approach can be viewed as a Riemann approximation of Integrated Gradients, trading axiomatic guarantees for practical speed.

An axiomatic method that computes feature importance by accumulating gradients along a straight-line path from a baseline input to the actual input. Like DeepLIFT, it satisfies the completeness property, ensuring the sum of attributions equals the difference from the reference. However, DeepLIFT's use of discrete differences rather than integrated gradients can provide a more efficient approximation, especially in deep networks where the gradient path integral is computationally expensive to approximate accurately.

A technique that redistributes the prediction score backwards using conservation-preserving propagation rules. DeepLIFT and LRP share deep conceptual roots; specific LRP rules like Alpha-Beta can be seen as equivalent to DeepLIFT's RevealCancel rule. Both methods were designed to handle the saturation problem that plagues simple gradient-based explanations, where a neuron's gradient may be near zero even when it strongly influences the output.

A game-theoretic framework that unifies additive feature attribution methods. DeepLIFT is mathematically equivalent to a SHAP value estimator when using a single reference value, specifically approximating Shapley values with a background distribution collapsed to a single point. The SHAP framework generalizes this by averaging over multiple background samples, providing a more robust but computationally heavier explanation than a single-reference DeepLIFT run.

A simple baseline attribution method that multiplies the input feature value by its gradient. This method fails profoundly on the saturation problem: when a function plateaus, the gradient becomes zero, incorrectly assigning zero importance to a critical feature. DeepLIFT was explicitly designed to overcome this failure mode by using a reference difference and a multiplier based on discrete slopes, providing a non-zero signal even in saturated regimes.

The most basic visualization of feature importance, computed as the raw gradient of the output with respect to the input. While computationally cheap, saliency maps suffer from gradient shattering and the saturation problem, often producing visually noisy and uninterpretable results. DeepLIFT provides a significant qualitative improvement by backpropagating discrete contribution scores rather than continuous gradients, yielding sharper and more semantically meaningful attributions.

A class of methods that explain a prediction by measuring the effect of deviating from a chosen reference or baseline input. DeepLIFT is a canonical example, requiring the user to define a neutral input (e.g., a black image or zero embedding vector). The choice of reference is critical: it defines the 'absence' of a feature and directly shapes the explanation. This contrasts with perturbation-based methods that measure the effect of removing a feature entirely.