Layer-wise Relevance Propagation (LRP) for Sequences

The foundational mathematical framework for LRP that decomposes the relevance at each neuron using a first-order Taylor expansion around a root point. For sequences, DTD is applied to the unrolled recurrent computational graph, treating each time step as a distinct neuron in a deep feedforward structure. This allows the propagation rules to respect the temporal dependencies encoded in the hidden states.

• Root point selection: Chosen near the neuron's input to minimize the Taylor remainder
• Unrolled graph: An RNN with 100 time steps is treated as a 100-layer network
• Relevance conservation: Ensures the sum of relevance across all time steps equals the final prediction score

The most common LRP rule for sequences, which separates the weighted activations into positive (excitatory) and negative (inhibitory) contributions. The parameters alpha and beta control how much relevance flows through positive versus negative connections, with the constraint that alpha - beta = 1 to maintain conservation.

• Alpha=1, Beta=0: Propagates only positive contributions, producing heatmaps of supporting evidence
• Alpha=2, Beta=1: Emphasizes positive evidence while still accounting for inhibitory signals
• Application: Applied to the input-to-hidden and hidden-to-hidden weight matrices of LSTMs and GRUs

A specialized propagation scheme that respects the gating mechanisms of Long Short-Term Memory networks. Standard LRP rules fail on LSTMs because they treat the cell state update as a simple linear operation, ignoring the multiplicative interactions of the forget, input, and output gates.

• Forget gate: Relevance is scaled by the forget gate activation to model the erasure of information
• Input gate: Relevance is modulated by the input gate to reflect the writing of new memory
• Cell state: The additive nature of the cell state update naturally preserves relevance across time steps
• Signal handling: Prevents relevance from being assigned to time steps where the gates are closed

Adapts the conservation principle to the self-attention mechanism by treating attention weights as relevance redistribution matrices. The relevance of each output token is propagated backward through the Value, Key, and Query projections, with the softmax attention scores determining how much relevance each input token receives.

• Attention head decomposition: Relevance is computed independently per head and then aggregated
• Residual connections: Relevance is split proportionally between the residual branch and the attention/MLP sublayers
• LayerNorm handling: The normalization statistics are treated as constants during the backward pass to maintain conservation

A practical approach that applies different propagation rules at different layers of the sequence model to optimize the trade-off between explanation fidelity and noise. Lower layers close to the input typically use rules that suppress noise, while upper layers use rules that preserve maximal detail.

• LRP-epsilon: Used in lower layers to absorb weak or contradictory relevance (epsilon typically 0.001 to 0.1)
• LRP-gamma: Applied in upper layers to favor positive contributions (gamma typically 0.1 to 1.0)
• Flat rule: Used in the output layer to distribute relevance uniformly across all inputs
• Benefit: Produces cleaner temporal heatmaps with less spurious noise at irrelevant time steps

The final output of LRP for sequences is a relevance heatmap that assigns a scalar score to each time step and feature dimension. This heatmap can be visualized as a stacked bar chart over time or as a colored overlay on the original time-series plot, enabling direct identification of the temporal intervals that drove the prediction.

• Feature-wise aggregation: Relevance can be summed across features to show per-time-step importance
• Temporal aggregation: Relevance can be summed across time to show per-feature importance
• Contrastive analysis: Heatmaps from different classes can be subtracted to reveal discriminative time steps
• Use case: Identifying the exact moment a fault precursor appeared in a predictive maintenance model

An attribution method that computes importance scores for time steps by comparing the activation of each neuron to a reference activation. It satisfies the summation-to-delta property for recurrent networks, meaning the sum of all feature attributions exactly equals the difference between the model's output and the reference output. This property makes it a close conceptual relative to LRP, as both decompose the output score completely without approximation error.

A diagnostic process that visualizes and quantifies the activations of forget, input, and output gates in an LSTM network. By tracking gate states across time steps, engineers can explain how information is stored, updated, or discarded. This provides a mechanistic complement to LRP: while LRP assigns relevance scores to inputs, gate analysis reveals the internal memory dynamics that caused those relevance patterns.

A quantitative evaluation score that measures how accurately a temporal explanation reflects the true reasoning process of the underlying model. It tests correlation between the explanation and model behavior under perturbation. Key properties include:

• Completeness: Does the explanation sum to the output?
• Continuity: Do similar inputs yield similar explanations?
• Implementation invariance: Do functionally equivalent networks produce identical attributions?

A robustness evaluation method that introduces small, controlled noise or distortions to a time series to analyze the stability and continuity of a model's explanations. If an LRP heatmap changes dramatically with imperceptible input changes, the explanation lacks reliability. This technique is essential for validating that relevance scores reflect genuine model reasoning rather than brittle artifacts of the network's geometry.

A perturbation-based method that systematically removes or masks individual time steps from a sequence to measure the resulting change in the model's output. By comparing the prediction drop when each step is ablated, engineers create a ground-truth importance ranking. This serves as a baseline for evaluating LRP: a faithful relevance decomposition should correlate strongly with ablation-based importance measures.

The application of algorithms like PCMCI or VAR-LiNGAM to time-series data to infer a causal graph. While LRP identifies which time steps the model uses, causal discovery determines which time steps actually drive the outcome in the real world. Combining both techniques allows engineers to audit whether a model's relevance assignments align with genuine causal relationships or merely exploit spurious correlations.