Attention Mechanism

The core function of attention is to compute a context vector as a weighted sum of input features, where the weights are dynamically generated based on the input itself. Unlike static convolutional filters, attention adapts its focus to the specific meteorological context. For example, during a frontal passage, the model can learn to assign higher weight to wind speed and pressure tendency while suppressing irrelevant temperature signals.

The standard mechanism computes compatibility between a query vector (what we are looking for) and key vectors (descriptors of the input) using a dot product, scaled by the square root of the dimension to prevent vanishing gradients in the softmax. This operation is highly parallelizable on GPU hardware, enabling efficient training on high-resolution Numerical Weather Prediction (NWP) grids.

Rather than learning a single attention function, multi-head attention projects queries, keys, and values into multiple lower-dimensional subspaces. This allows the model to simultaneously attend to different types of meteorological relationships:

• Temporal head: Focuses on diurnal cycle patterns
• Spatial head: Correlates output with upwind wind farm production
• Feature head: Isolates the impact of Global Horizontal Irradiance (GHI) versus cloud cover

Attention is inherently permutation-invariant; it has no built-in notion of sequence order. To model time-series data, sinusoidal positional encodings or learned embeddings must be added to the input vectors. This injects critical temporal context, allowing the mechanism to distinguish between a morning irradiance ramp and an evening decline, preserving the sequential nature of the forecast horizon.

Two distinct operational modes define the flow of information:

• Self-Attention: Queries, keys, and values all originate from the same sequence (e.g., historical GHI). This captures intra-sequence dependencies like the autocorrelation of wind speed.
• Cross-Attention: Queries originate from the decoder (target forecast), while keys and values come from the encoder (NWP inputs). This aligns the predicted output sequence with the driving meteorological features.

To enforce causality in time-series forecasting, a look-ahead mask is applied to the attention scores before the softmax. This sets all weights corresponding to future time steps to negative infinity, ensuring the prediction at time t depends only on inputs from time t and earlier. This prevents information leakage and maintains the integrity of the forecast evaluation.

The foundational computational kernel. It computes the dot product of a query vector with all key vectors, scales the result by the inverse square root of the dimension (1/√d_k) to prevent vanishing gradients in the softmax, and uses the resulting weights to aggregate value vectors.

• Formula: Attention(Q,K,V) = softmax(QK^T / √d_k) V
• Scaling Factor: Prevents logits from entering regions of extremely small softmax gradients.
• Complexity: Quadratic in sequence length, O(L²).

A mechanism that projects queries, keys, and values into multiple lower-dimensional subspaces (h heads) in parallel. This allows the model to attend to information from different representation subspaces at different positions simultaneously, rather than averaging signals into a single context vector.

• Parallelism: Each head learns a distinct relational pattern (e.g., temporal adjacency, periodicity).
• Concatenation: Head outputs are concatenated and linearly projected back to the model dimension.
• Benefit: Captures diverse meteorological feature interactions in a single layer.

A configuration where the query, key, and value vectors are all derived from the same input sequence. This enables every position in a time series to attend to every other position, directly capturing long-range dependencies without the sequential bottleneck of recurrent networks.

• Intra-Sequence: Computes relationships within a single sequence of weather features.
• Global Receptive Field: A single layer connects all time steps, unlike CNNs which require stacking for a wide field.
• Application: Identifying that a wind ramp at t-60 is the primary driver of a forecast at t.

A configuration where queries are derived from one sequence (e.g., target power output) and keys/values from another (e.g., numerical weather prediction features). This fuses information between two distinct modalities or sequences, acting as a learned, context-dependent filter.

• Decoder Core: The primary mechanism in Transformer decoders for conditioning generation on encoded inputs.
• Feature Fusion: Aligns NWP forecast grids with specific turbine location coordinates.
• Differentiation: Distinct from self-attention because the key and query source sequences differ.

An attention modification that prevents positions from attending to subsequent positions in a sequence by adding a mask of negative infinity values before the softmax. This preserves the autoregressive property required for real-time forecasting, ensuring predictions for time t depend only on inputs from t-1 and earlier.

• Implementation: An upper-triangular matrix of -∞ is added to the attention logits.
• Temporal Integrity: Prevents information leakage from the future during training.
• Use Case: Essential for generating operational day-ahead forecasts sequentially.

A class of approximations that reduce the quadratic O(L²) memory complexity of full attention by restricting each query to attend to only a subset of keys. This enables processing very long meteorological sequences or high-resolution satellite imagery patches.

• Patterns: Includes sliding window, dilated sliding window, and global+local combinations.
• Efficiency: Reduces complexity to O(L√L) or O(L log L).
• Trade-off: Sacrifices the global receptive field for computational tractability on long sequences.