Deep Taylor Decomposition

The decomposition is performed at a root point in the input space where the function value f(x₀) = 0. This root is found by searching along the gradient direction from the input until the decision boundary is crossed. The choice of root point is critical—it determines the path along which the Taylor expansion is computed and directly influences the quality of the resulting relevance maps. For ReLU networks, root points are often located on the piecewise linear boundary where the neuron activation state changes.

Deep Taylor Decomposition provides the theoretical foundation for Layer-wise Relevance Propagation (LRP). The LRP-αβ rules commonly used in practice can be derived as special cases of Deep Taylor expansions with specific choices of root points. This connection elevates LRP from a heuristic propagation scheme to a mathematically justified decomposition method grounded in functional analysis and Taylor's theorem.

The decomposition satisfies the completeness axiom by construction: the sum of all relevance scores assigned to input features exactly equals the model's output value for the target class. This conservation is maintained layer by layer during backpropagation, ensuring that no relevance is artificially created or destroyed. This property is essential for auditing applications where total attribution must be fully accounted for.

For deep networks with non-linear activation functions, the method employs local Taylor approximations at each layer. The relevance received by a neuron is redistributed to its inputs by expanding the neuron's output function around a root point in its input space. Different propagation rules emerge from different choices of root points:

• w²-rule: Root point chosen on the line between the input and origin
• z-rule: Root point at the origin itself
• αβ-rule: Weighted combination of positive and negative contributions

Unlike standard gradient-based methods that can suffer from gradient saturation and shattered gradients, Deep Taylor Decomposition explicitly addresses the non-linearity of deep networks. While Gradient × Input provides only a first-order approximation without conservation guarantees, Deep Taylor Decomposition enforces exact decomposition by design. It also avoids the noise amplification issues of Integrated Gradients by operating directly on the function value rather than accumulating gradients along arbitrary paths.

The broader family of techniques to which Deep Taylor Decomposition belongs. LRP operates on a conservation principle, redistributing the model's output score backwards through the network layers. Each neuron receives a relevance score from higher layers and redistributes it to lower layers based on neuron activations and weights. Deep Taylor Decomposition provides a theoretical justification for specific LRP propagation rules by treating them as Taylor expansions at root points.

A core concept in Deep Taylor Decomposition where a reference point is identified in the input space at which the neuron's function evaluates to zero. The decomposition then performs a first-order Taylor expansion of the function at this root point. The choice of root point is critical:

• Nearest root point: Minimizes higher-order Taylor residuals
• Constrained root points: Respect the input domain (e.g., pixel values bounded by [0,1])
• Multiple root points: Used when a single root point is insufficient

A fundamental mathematical property that Deep Taylor Decomposition satisfies by construction. The axiom states that the sum of all feature attributions must exactly equal the model's output difference from a baseline. This ensures no relevance is created or destroyed during propagation. Formally: Σᵢ Rᵢ = f(x) - f(x₀), where Rᵢ is the relevance of feature i, f(x) is the model output, and f(x₀) is the baseline output. This conservation of relevance makes the method auditable.

A closely related attribution method that also uses a reference-based decomposition strategy. DeepLIFT compares neuron activations to a reference activation and assigns contribution scores based on the difference-from-reference. While Deep Taylor Decomposition uses Taylor expansion at root points, DeepLIFT uses a finite-difference approach with a summation-to-delta property that is analogous to the completeness axiom. Both methods address the gradient saturation problem that plagues simple gradient-based methods.

A critical failure mode of simple gradient-based attribution that Deep Taylor Decomposition is designed to overcome. In deep networks with saturating nonlinearities (like sigmoid or softmax), the gradient of the output with respect to input features can become near-zero for features that strongly activate the correct class. This creates a paradox where the most influential features appear unimportant. Deep Taylor Decomposition avoids this by using function value redistribution rather than local gradient information alone.

The specific mathematical formulas that govern how relevance is redistributed at each layer. Deep Taylor Decomposition derives different rules for different layer types:

• z⁺-rule: For layers with positive-only activations (ReLU)
• αβ-rule: A parameterized rule that controls the ratio of positive to negative relevance
• w²-rule: For layers where weights are squared to eliminate sign cancellation Each rule corresponds to a different Taylor expansion root point choice.