DeepLIFT

DeepLIFT computes contributions by comparing a neuron's activation during a forward pass to its activation when the network is given a reference input (often a zero vector or a blurred image). The difference-from-reference is then propagated backward using a linear composition rule.

• Reference input: A neutral baseline representing 'no signal'
• Difference-from-reference: Δx = x - x⁰
• Summation-to-delta: The sum of all neuron contributions equals the difference in output from the reference output

A critical failure mode of pure gradient methods like Gradient × Input occurs when a feature strongly activates the correct class, pushing the output to a flat region of the activation function (e.g., sigmoid or softmax). The local gradient becomes near-zero, incorrectly suggesting the feature is unimportant.

DeepLIFT avoids this by using finite differences rather than infinitesimal gradients. It considers the actual change in output caused by the presence of a feature relative to its absence in the reference, providing a non-zero signal even in saturated regimes.

DeepLIFT defines a multiplier m for each neuron, representing the ratio of the difference in its output to the difference in its input: m = Δy / Δx. Contributions are propagated backward using a modified chain rule that multiplies these multipliers.

• Linear Rule: For linear transformations, the multiplier is the weight matrix
• Rescale Rule: For non-linear activations (ReLU, sigmoid), the multiplier is Δy / Δx
• RevealCancel Rule: Separates positive and negative contributions to handle opposing influences

DeepLIFT offers two primary rules for handling non-linear activations, which determine how positive and negative contributions interact:

Rescale Rule:

• Simpler and computationally cheaper
• Assumes the ratio of positive to negative contributions is preserved
• Multiplier = Δy / Δx

RevealCancel Rule:

• Treats positive and negative contributions separately
• Reveals cases where positive and negative inputs cancel each other out
• More faithful to the actual computation but computationally heavier
• Preferred when opposing influences are expected

DeepLIFT satisfies several key axioms from the axiomatic attribution framework, making it a theoretically grounded method:

• Completeness (Summation-to-Delta): The sum of all feature attributions equals the difference between the model's output for the actual input and the reference input
• Sensitivity-n: If a feature is mathematically independent of the output, it receives zero attribution
• Implementation Invariance: Two functionally equivalent networks produce identical attributions

Note that DeepLIFT does not satisfy symmetry in the same way SHAP does, as it depends on the choice of reference input.

DeepLIFT and Integrated Gradients are closely related. Integrated Gradients computes attributions by integrating gradients along a straight-line path from a baseline to the input. DeepLIFT can be viewed as a Riemann approximation of this integral using a single step.

• DeepLIFT uses a single reference point; Integrated Gradients uses a continuous path
• Both satisfy the completeness axiom
• DeepLIFT is computationally faster but may be less precise for highly non-linear paths
• Expected Gradients generalizes both by averaging over a distribution of baselines

The foundational concept in DeepLIFT. A reference activation is the neuron's output when the network is given a neutral or baseline input (e.g., an all-black image or a zero embedding vector). DeepLIFT explains the difference between the target output and the reference output by propagating activation differences backward through the network. The choice of reference is critical: a poor reference can lead to misleading attributions where important features receive zero contribution scores.

An axiom that DeepLIFT satisfies, equivalent to the Completeness Axiom in other frameworks. It states that the sum of all contribution scores assigned to the input features must equal the difference between the model's output for the actual input and its output for the reference input. This conservation property ensures that the prediction difference is fully decomposed and accounted for, with no attribution leaking or being created during backpropagation.

Two distinct multiplier calculation strategies for assigning contribution scores in DeepLIFT:

• Rescale Rule: Assumes a linear relationship, distributing contribution in proportion to the ratio of positive and negative activation differences. Simple but can be inaccurate for highly non-linear functions.
• RevealCancel Rule: Treats positive and negative contributions as separate streams that can cancel each other out. It attributes the net effect by revealing hidden cancellations, providing a more faithful decomposition for operations like ReLU where negative inputs are suppressed.

A closely related axiomatic method that also satisfies completeness. While DeepLIFT uses a single discrete reference point and finite differences, Integrated Gradients accumulates gradients along a continuous straight-line path from a baseline to the input. DeepLIFT can be viewed as a Riemann approximation of Integrated Gradients using a single step. Both methods address the gradient saturation problem where standard gradients fail to highlight features that have saturated their contribution to the output.

A decomposition method that shares DeepLIFT's core philosophy of redistributing relevance scores backward through the network using conservation rules. LRP defines specific propagation rules (e.g., LRP-αβ, LRP-ε) for different layer types. DeepLIFT's RevealCancel rule is conceptually similar to LRP-αβ with α=1, β=0. Both methods produce explanations by design rather than by numerical approximation, making them computationally efficient alternatives to path integration methods.

A critical flaw in standard gradient-based attribution that DeepLIFT was designed to solve. When a feature strongly activates the correct output class, the loss function flattens, causing the gradient to approach zero. A naive Gradient×Input attribution would incorrectly assign near-zero importance to the very feature driving the prediction. DeepLIFT bypasses this by using a reference activation where the feature is absent, computing the finite difference in output rather than relying on the instantaneous, saturated gradient.