Long Short-Term Memory

The core innovation of LSTM is the cell state (C_t), a linear highway running through the entire chain. Unlike standard RNNs where gradients decay exponentially, the cell state allows uninterrupted gradient flow across arbitrary time lags. This Constant Error Carousel (CEC) ensures that a signal from t=1 can directly influence the output at t=100 without being multiplied by a recurrent weight matrix at every step, effectively solving the vanishing gradient problem that plagued earlier architectures.

LSTM controls information flow through three multiplicative gates, each a sigmoid neural net layer outputting values between 0 and 1:

• Forget Gate (f_t): Decides what to discard from the cell state. It examines h_{t-1} and x_t, outputting a number near 0 to 'forget' or near 1 to 'keep' each value.
• Input Gate (i_t): Determines which new information to store. A sigmoid layer decides which values to update, while a tanh layer creates a vector of new candidate values (C̃_t).
• Output Gate (o_t): Controls what parts of the cell state are exposed to the hidden state and output. The cell state is pushed through tanh and multiplied by the output gate's sigmoid output.

LSTM maintains two distinct state vectors, unlike the single hidden state of vanilla RNNs:

• Cell State (C_t): The long-term memory. It changes slowly through linear interactions, preserving information across long sequences. Gradients flow through it additively, not multiplicatively.
• Hidden State (h_t): The short-term working memory. It is a filtered, non-linear transformation of the cell state, representing the information relevant for the immediate next step.

This separation allows the network to maintain a stable long-term context while dynamically adjusting its short-term output based on the current input.

A common LSTM variant introduces peephole connections that allow the gate layers to directly inspect the cell state. In the standard architecture, gates base their decisions only on h_{t-1} and x_t. With peepholes, the forget gate also receives C_{t-1} as input, and the input and output gates receive C_t. This gives the network precise timing capabilities—for example, the forget gate can learn to reset the cell state exactly when a sequence boundary is reached, even if the hidden state hasn't yet encoded that information.

The LSTM's design ensures that the gradient of the error with respect to the cell state does not vanish. During backpropagation through time (BPTT), the recurrent edge weight on the cell state is effectively 1.0 (the identity function). This means the error signal is multiplied by the forget gate's output at each step, not by a weight matrix. As long as the forget gate remains near 1, the gradient propagates intact. This allows LSTMs to learn dependencies spanning 1,000+ time steps, compared to the 10-20 step limit of vanilla RNNs.

For demand forecasting, LSTMs are typically deployed in an encoder-decoder configuration. The encoder LSTM ingests the historical demand sequence and condenses it into a final cell state vector. The decoder LSTM is initialized with this context and generates the multi-step forecast autoregressively. This architecture naturally handles:

• Variable-length input sequences (e.g., 90 days of history)
• Multi-horizon output (e.g., 30-day forecast)
• Exogenous features like promotions, holidays, and pricing

Modern implementations often replace the decoder with a linear output layer that directly maps the final hidden state to all forecast horizons simultaneously.

The fundamental challenge that LSTMs were designed to solve. In standard RNNs, gradients shrink exponentially as they propagate backward through time, preventing the network from learning dependencies beyond ~10 time steps. LSTMs mitigate this through a Constant Error Carousel (CEC) — the cell state pathway where gradients flow unchanged via linear operations. This architectural choice enables learning across hundreds of time steps, critical for capturing quarterly demand cycles or annual seasonality in supply chain data.

LSTMs control information flow through three specialized gates, each a learned neural network layer with sigmoid activation:

• Forget Gate: Decides what to discard from the cell state (e.g., obsolete promotional effects)
• Input Gate: Determines which new information to store (e.g., a sudden demand spike signal)
• Output Gate: Controls what parts of the cell state inform the hidden state output

These gates give LSTMs selective memory, allowing them to retain multi-month supplier lead time patterns while ignoring irrelevant daily noise.

The training algorithm that unrolls the LSTM across its entire sequence length before computing gradients. For a 52-week demand history, BPTT unrolls 52 time steps, applying the chain rule through every gate activation. The truncated BPTT variant limits unrolling to a fixed window (e.g., 20 steps) to balance memory constraints with long-range learning. This is computationally intensive but essential for capturing the full temporal structure of intermittent demand patterns.

An encoder-decoder architecture where one LSTM encodes the input history into a fixed-length context vector, and a second LSTM decodes it into multi-step forecasts. This design enables:

• Multi-horizon output: Generate 12-week forecasts in one pass
• Teacher forcing: Use ground truth during training to stabilize convergence
• Dynamic context: The decoder attends to relevant encoder states at each output step

Widely used in DeepAR and other probabilistic forecasting models for supply chain applications.

An LSTM variant where the gates are allowed to inspect the actual cell state when making decisions, not just the hidden state and current input. This gives the forget and input gates direct visibility into what is currently stored. In demand forecasting, peephole connections help the model learn precise timing — for example, recognizing exactly when a seasonal peak has been fully captured in the cell state and can be safely forgotten.

An architecture that processes the input sequence in both forward and reverse directions using two separate LSTM layers whose outputs are concatenated. While primarily used in NLP, BiLSTMs have supply chain applications in demand pattern classification and anomaly detection, where understanding both preceding and following context around a potential disruption event improves accuracy. The reverse pass captures patterns that only become apparent retrospectively.