Long Short-Term Memory (LSTM)

The core innovation of LSTM is the Constant Error Carousel, a linear self-connected unit that maintains a constant error flow across time steps. Unlike standard RNNs where gradients decay exponentially, the CEC allows the network to bridge time lags exceeding 1000 discrete steps. This is critical for capturing the diurnal solar cycle or multi-day weather patterns in irradiance forecasting without signal degradation.

The forget gate is a sigmoid-activated layer that outputs a value between 0 and 1 for each number in the cell state, determining what information to discard. A value of 1 means 'completely retain,' while 0 means 'completely forget.' In wind power forecasting, this gate learns to reset accumulated wind speed trends when a frontal system passes, preventing stale meteorological context from corrupting the prediction.

New information is added to the cell state through a two-step process:

• The input gate (sigmoid layer) decides which values to update.
• A tanh layer creates a vector of new candidate values that could be added to the state. This dual-path design allows the network to selectively incorporate sudden irradiance ramp events caused by cloud edges while ignoring sensor noise.

The output gate controls what parts of the cell state are exposed to the next layer and the subsequent time step. It applies a sigmoid filter to the current input and previous hidden state, then multiplies it with a tanh-squashed version of the updated cell state. This ensures that only task-relevant memory—such as the current clear sky index trend—is passed forward for the final power prediction.

While standard LSTMs process sequences in forward chronological order, Bidirectional LSTMs run two independent recurrent layers: one forward and one backward through time. This allows the model to condition its prediction on both past and future context within a training window. For day-ahead solar forecasting, this captures the symmetrical build-up and dissipation of cumulus cloud fields around the target hour.

A structural variant where the gates are allowed to inspect the actual cell state directly, rather than relying solely on the hidden state. This gives the forget, input, and output gates direct access to the CEC's memory. In practice, this helps the network learn precise timing for resetting accumulated wind energy estimates when the turbine's rated power curve saturates.

The core innovation enabling LSTMs to learn long-range dependencies without vanishing gradients. Three gates regulate information flow:

• Forget Gate: Decides what information to discard from the cell state.
• Input Gate: Decides which new information to store in the cell state.
• Output Gate: Decides what to output based on the cell state. This structure allows the network to preserve gradients over hundreds of time steps, making it ideal for capturing diurnal solar patterns and multi-day weather fronts.

The training algorithm used to update LSTM weights by unrolling the network over the temporal sequence. Key characteristics:

• The network is unrolled into a deep feed-forward network where layers represent time steps.
• Errors are propagated backward through all time steps to calculate weight gradients.
• Truncated BPTT is often used to limit the unroll length, managing memory constraints while still capturing relevant meteorological dependencies for day-ahead forecasts.

A practical comparison with the Gated Recurrent Unit, a popular simplification:

• LSTM: Uses three gates (forget, input, output) and maintains a separate cell state and hidden state. More expressive but more parameters.
• GRU: Combines the forget and input gates into a single update gate, and merges the cell and hidden states. Computationally faster. For renewable forecasting, LSTMs often outperform GRUs on long sequences with complex seasonal patterns, while GRUs may suffice for shorter intraday ramp rate prediction.

An encoder-decoder framework where LSTMs process an input sequence (e.g., historical NWP data) and generate an output sequence (e.g., future power output).

• The Encoder LSTM compresses the input history into a fixed-length context vector.
• The Decoder LSTM unrolls this vector to produce multi-step predictions.
• Attention mechanisms are often added to allow the decoder to focus on specific input time steps, significantly improving probabilistic power forecasts for horizons beyond 6 hours.

The critical failure mode of standard RNNs that LSTMs were designed to solve. In deep or long sequences:

• Gradients shrink exponentially as they are backpropagated through time.
• Early layers learn extremely slowly, preventing the network from connecting distant events. LSTMs mitigate this via the Constant Error Carousel (CEC)—the additive cell state update that allows gradient flow across many time steps without multiplication by recurrent weights. This is essential for linking morning cloud cover to afternoon solar ramp events.

An architecture that processes the input sequence in both forward and backward directions, combining two hidden states. This provides the output layer with full context from both past and future time steps.

• Particularly useful for site calibration tasks where the entire historical time series is available.
• In forecasting, BiLSTMs are typically used in the encoder of a Seq2Seq model to create a richer context vector from the available meteorological history before the decoder generates the prediction.