Attention Flow

Attention Flow tracks how information from a specific input token or time step is combined and transformed as it passes through successive layers. It accounts for the residual connections in Transformer architectures, which allow information to bypass attention layers. The formal mechanism often involves computing attention rollout—multiplying attention weight matrices across layers while adding the identity matrix to represent the skip connection. This reveals that even if a token is not directly attended to in a deep layer, its influence can persist through the residual stream.

In each attention head, the raw attention weights define a stochastic matrix that mixes value vectors from different positions. Attention Flow aggregates these mixing proportions across heads and layers to quantify how much of a final representation originates from each input position. This is distinct from raw attention weights, which only show direct interactions. Flow analysis reveals the effective receptive field of the model, which often extends far beyond the immediately attended tokens due to the compounding effect of multiple layers.

For time-series Transformers, Attention Flow produces a saliency map over the input sequence, highlighting which historical time steps most influenced a forecast. This is particularly valuable in finance and IoT analytics, where a model might predict a demand spike based on a subtle pattern from 50 time steps ago. The flow values can be normalized to sum to 1 across the input, creating a probability distribution of influence that is directly interpretable by human operators auditing the model's reasoning.

By disaggregating Attention Flow per attention head, analysts can identify functional specialization within the model. Some heads may consistently flow information from immediate predecessors (local pattern detectors), while others may route information from distant, semantically relevant positions (long-range dependency heads). In temporal models, this often reveals heads dedicated to seasonality detection, trend extraction, or anomaly propagation, providing mechanistic insights into how the model decomposes the forecasting problem.

Attention Flow explanations face a critical evaluation challenge. Faithfulness measures whether the explanation accurately reflects the model's true computational process. Plausibility measures whether the explanation aligns with human intuition. Raw attention weights are often plausible but not faithful, as they can be manipulated without changing predictions. Attention Flow, especially when combined with gradient information (Attention-Flow Grad-CAM), improves faithfulness by accounting for the magnitude of value vectors, not just attention probabilities. However, no attention-based method is perfectly faithful, and results should be validated with perturbation tests.

Unlike perturbation-based methods such as Time-Step Ablation or Temporal Occlusion Analysis, which require multiple forward passes, Attention Flow is computed in a single backward-style pass through the attention matrices. For a model with L layers and H heads, the rollout computation is O(L * H * T²) where T is the sequence length. This makes it significantly faster than occlusion (O(T * forward_pass)) for long sequences. The method requires no hyperparameter tuning and produces deterministic results, making it suitable for real-time explainability dashboards in production forecasting systems.